Skip to contentSkip to conversation

io_uring: hold uring_lock to complete faild polled io in io_wq_submit_work()

PATCH In review Xiaoguang Wang 3 participants 8 messages Dec 14, 2020 – Dec 23, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 io_uring: hold uring_lock to complete faild polled io in io_wq_submit_work()

io_iopoll_complete() does not hold completion_lock to complete polled io, so in io_wq_submit_work(), we can not call io_req_complete() directly, to complete polled io, otherwise there maybe concurrent access to cqring, defer_list, etc, which is not safe. Commit dad1b1242fd5 ("io_uring: always let io_iopoll_complete() complete polled io") has fixed this issue, but Pavel reported that IOPOLL apart from rw can do buf reg/unreg requests( IORING_OP_PROVIDE_BUFFERS or IORING_OP_REMOVE_BUFFERS), so the fix is not good.

Given that io_iopoll_complete() is always called under uring_lock, so here for polled io, we can also get uring_lock to fix this issue.

Fixes dad1b1242fd5 ("io_uring: always let io_iopoll_complete() complete polled io")
---
 fs/io_uring.c | 25 +++++++++++++++----------
 1 file changed, 15 insertions(+), 10 deletions(-)
fs/io_uring.c +15 -10
index f53356ced5ab..eab3d2b7d232 100644
@@ -6354,19 +6354,24 @@ static struct io_wq_work *io_wq_submit_work(struct io_wq_work *work)
 	}
 
 	if (ret) {
+		bool iopoll_enabled = req->ctx->flags & IORING_SETUP_IOPOLL;
+
 		/*
-		 * io_iopoll_complete() does not hold completion_lock to complete
-		 * polled io, so here for polled io, just mark it done and still let
-		 * io_iopoll_complete() complete it.
+		 * io_iopoll_complete() does not hold completion_lock to complete polled
+		 * io, so here for polled io, we can not call io_req_complete() directly,
+		 * otherwise there maybe concurrent access to cqring, defer_list, etc,
+		 * which is not safe. Given that io_iopoll_complete() is always called
+		 * under uring_lock, so here for polled io, we also get uring_lock to
+		 * complete it.
 		 */
-		if (req->ctx->flags & IORING_SETUP_IOPOLL) {
-			struct kiocb *kiocb = &req->rw.kiocb;
+		if (iopoll_enabled)
+			mutex_lock(&req->ctx->uring_lock);
 
-			kiocb_done(kiocb, ret, NULL);
-		} else {
-			req_set_fail_links(req);
-			io_req_complete(req, ret);
-		}
+		req_set_fail_links(req);
+		io_req_complete(req, ret);
+
+		if (iopoll_enabled)
+			mutex_unlock(&req->ctx->uring_lock);
 	}
 
 	return io_steal_work(req);
Signature · 1 line
2.17.2
#1 lore raw
Quoted 11 lines · Xiaoguang Wang

io_iopoll_complete() does not hold completion_lock to complete polled io, so in io_wq_submit_work(), we can not call io_req_complete() directly, to complete polled io, otherwise there maybe concurrent access to cqring, defer_list, etc, which is not safe. Commit dad1b1242fd5 ("io_uring: always let io_iopoll_complete() complete polled io") has fixed this issue, but Pavel reported that IOPOLL apart from rw can do buf reg/unreg requests( IORING_OP_PROVIDE_BUFFERS or IORING_OP_REMOVE_BUFFERS), so the fix is not good.

Given that io_iopoll_complete() is always called under uring_lock, so here for polled io, we can also get uring_lock to fix this issue.

One thing I don't like is that io_wq_submit_work() won't be able to publish an event while someone polling io_uring_enter(ENTER_GETEVENTS), that's because both take the lock. The problem is when the poller waits for an event that is currently in io-wq (i.e. io_wq_submit_work()). The polling loop will eventually exit, so that's not a deadlock, but latency,etc. would be huge.

Quoted 47 lines
Fixes dad1b1242fd5 ("io_uring: always let io_iopoll_complete() complete polled io")
---
 fs/io_uring.c | 25 +++++++++++++++----------
 1 file changed, 15 insertions(+), 10 deletions(-)
fs/io_uring.c +15 -10
index f53356ced5ab..eab3d2b7d232 100644
@@ -6354,19 +6354,24 @@ static struct io_wq_work *io_wq_submit_work(struct io_wq_work *work)
 	}
 
 	if (ret) {
+		bool iopoll_enabled = req->ctx->flags & IORING_SETUP_IOPOLL;
+
 		/*
-		 * io_iopoll_complete() does not hold completion_lock to complete
-		 * polled io, so here for polled io, just mark it done and still let
-		 * io_iopoll_complete() complete it.
+		 * io_iopoll_complete() does not hold completion_lock to complete polled
+		 * io, so here for polled io, we can not call io_req_complete() directly,
+		 * otherwise there maybe concurrent access to cqring, defer_list, etc,
+		 * which is not safe. Given that io_iopoll_complete() is always called
+		 * under uring_lock, so here for polled io, we also get uring_lock to
+		 * complete it.
 		 */
-		if (req->ctx->flags & IORING_SETUP_IOPOLL) {
-			struct kiocb *kiocb = &req->rw.kiocb;
+		if (iopoll_enabled)
+			mutex_lock(&req->ctx->uring_lock);
 
-			kiocb_done(kiocb, ret, NULL);
-		} else {
-			req_set_fail_links(req);
-			io_req_complete(req, ret);
-		}
+		req_set_fail_links(req);
+		io_req_complete(req, ret);
+
+		if (iopoll_enabled)
+			mutex_unlock(&req->ctx->uring_lock);
 	}
 
 	return io_steal_work(req);
Signature · 1 line
Pavel Begunkov
#2 lore raw

hi,

Quoted 13 lines · Xiaoguang Wang
Quoted 11 lines

io_iopoll_complete() does not hold completion_lock to complete polled io, so in io_wq_submit_work(), we can not call io_req_complete() directly, to complete polled io, otherwise there maybe concurrent access to cqring, defer_list, etc, which is not safe. Commit dad1b1242fd5 ("io_uring: always let io_iopoll_complete() complete polled io") has fixed this issue, but Pavel reported that IOPOLL apart from rw can do buf reg/unreg requests( IORING_OP_PROVIDE_BUFFERS or IORING_OP_REMOVE_BUFFERS), so the fix is not good.

Given that io_iopoll_complete() is always called under uring_lock, so here for polled io, we can also get uring_lock to fix this issue.

Quoted 6 lines

One thing I don't like is that io_wq_submit_work() won't be able to publish an event while someone polling io_uring_enter(ENTER_GETEVENTS), that's because both take the lock. The problem is when the poller waits for an event that is currently in io-wq (i.e. io_wq_submit_work()). The polling loop will eventually exit, so that's not a deadlock, but latency,etc. would be huge.

In this patch, we just hold uring_lock for polled io in error path, so I think normally it maybe not an issue, and seems that the critical section is not that big, so it also may not result in huge latecy.
I also noticed that current codes also hold uring_lock in io_wq_submit_work() call chain:
==> io_wq_submit_work()
====> io_issue_sqe()
======> io_provide_buffers()
========> io_ring_submit_lock(ctx, !force_nonblock);

Regards,
Xiaoguang Wang

Quoted 49 lines
Quoted 48 lines
Fixes dad1b1242fd5 ("io_uring: always let io_iopoll_complete() complete polled io")
---
  fs/io_uring.c | 25 +++++++++++++++----------
  1 file changed, 15 insertions(+), 10 deletions(-)
fs/io_uring.c +15 -10
index f53356ced5ab..eab3d2b7d232 100644
@@ -6354,19 +6354,24 @@ static struct io_wq_work *io_wq_submit_work(struct io_wq_work *work)
  	}
 
  	if (ret) {
+		bool iopoll_enabled = req->ctx->flags & IORING_SETUP_IOPOLL;
+
  		/*
-		 * io_iopoll_complete() does not hold completion_lock to complete
-		 * polled io, so here for polled io, just mark it done and still let
-		 * io_iopoll_complete() complete it.
+		 * io_iopoll_complete() does not hold completion_lock to complete polled
+		 * io, so here for polled io, we can not call io_req_complete() directly,
+		 * otherwise there maybe concurrent access to cqring, defer_list, etc,
+		 * which is not safe. Given that io_iopoll_complete() is always called
+		 * under uring_lock, so here for polled io, we also get uring_lock to
+		 * complete it.
  		 */
-		if (req->ctx->flags & IORING_SETUP_IOPOLL) {
-			struct kiocb *kiocb = &req->rw.kiocb;
+		if (iopoll_enabled)
+			mutex_lock(&req->ctx->uring_lock);
 
-			kiocb_done(kiocb, ret, NULL);
-		} else {
-			req_set_fail_links(req);
-			io_req_complete(req, ret);
-		}
+		req_set_fail_links(req);
+		io_req_complete(req, ret);
+
+		if (iopoll_enabled)
+			mutex_unlock(&req->ctx->uring_lock);
  	}
 
  	return io_steal_work(req);
#3 lore raw
Quoted 21 lines · Xiaoguang Wang

hi,

Quoted 13 lines · Xiaoguang Wang
Quoted 11 lines

io_iopoll_complete() does not hold completion_lock to complete polled io, so in io_wq_submit_work(), we can not call io_req_complete() directly, to complete polled io, otherwise there maybe concurrent access to cqring, defer_list, etc, which is not safe. Commit dad1b1242fd5 ("io_uring: always let io_iopoll_complete() complete polled io") has fixed this issue, but Pavel reported that IOPOLL apart from rw can do buf reg/unreg requests( IORING_OP_PROVIDE_BUFFERS or IORING_OP_REMOVE_BUFFERS), so the fix is not good.

Given that io_iopoll_complete() is always called under uring_lock, so here for polled io, we can also get uring_lock to fix this issue.

Quoted 6 lines

One thing I don't like is that io_wq_submit_work() won't be able to publish an event while someone polling io_uring_enter(ENTER_GETEVENTS), that's because both take the lock. The problem is when the poller waits for an event that is currently in io-wq (i.e. io_wq_submit_work()). The polling loop will eventually exit, so that's not a deadlock, but latency,etc. would be huge.

Quoted 3 lines

In this patch, we just hold uring_lock for polled io in error path, so I think normally it maybe not an issue, and seems that the critical section is not that big, so it also may not result in huge latecy.

To give a bit of a context, it breaks out of the io_iopoll_check()'s loop with need_resched(). Anyway, fair enough, probably should be dealt separately.

Quoted 6 lines

I also noticed that current codes also hold uring_lock in io_wq_submit_work() call chain:
==> io_wq_submit_work()
====> io_issue_sqe()
======> io_provide_buffers()
========> io_ring_submit_lock(ctx, !force_nonblock);

Yep, that's not good either, though I care more about rw.

Quoted 49 lines

Regards,
Xiaoguang Wang

Quoted 46 lines
Quoted 45 lines
Fixes dad1b1242fd5 ("io_uring: always let io_iopoll_complete() complete polled io")

---
  fs/io_uring.c | 25 +++++++++++++++----------

  1 file changed, 15 insertions(+), 10 deletions(-)
fs/io_uring.c +15 -9
index f53356ced5ab..eab3d2b7d232 100644
@@ -6354,19 +6354,24 @@ static struct io_wq_work *io_wq_submit_work(struct io_wq_work *work)
       }
         if (ret) {
+        bool iopoll_enabled = req->ctx->flags & IORING_SETUP_IOPOLL;
+
           /*
-         * io_iopoll_complete() does not hold completion_lock to complete
-         * polled io, so here for polled io, just mark it done and still let
-         * io_iopoll_complete() complete it.
+         * io_iopoll_complete() does not hold completion_lock to complete polled
+         * io, so here for polled io, we can not call io_req_complete() directly,
+         * otherwise there maybe concurrent access to cqring, defer_list, etc,
+         * which is not safe. Given that io_iopoll_complete() is always called
+         * under uring_lock, so here for polled io, we also get uring_lock to
+         * complete it.
            */
-        if (req->ctx->flags & IORING_SETUP_IOPOLL) {
-            struct kiocb *kiocb = &req->rw.kiocb;
+        if (iopoll_enabled)
+            mutex_lock(&req->ctx->uring_lock);
   -            kiocb_done(kiocb, ret, NULL);
-        } else {
-            req_set_fail_links(req);
-            io_req_complete(req, ret);
-        }
+        req_set_fail_links(req);
+        io_req_complete(req, ret);
+
+        if (iopoll_enabled)
+            mutex_unlock(&req->ctx->uring_lock);
       }
         return io_steal_work(req);
 
Signature · 1 line
Pavel Begunkov
Reviewed-by on v1 1/1
#4 lore raw
In reply to Xiaoguang Wang
Quoted 11 lines · Xiaoguang Wang

io_iopoll_complete() does not hold completion_lock to complete polled io, so in io_wq_submit_work(), we can not call io_req_complete() directly, to complete polled io, otherwise there maybe concurrent access to cqring, defer_list, etc, which is not safe. Commit dad1b1242fd5 ("io_uring: always let io_iopoll_complete() complete polled io") has fixed this issue, but Pavel reported that IOPOLL apart from rw can do buf reg/unreg requests( IORING_OP_PROVIDE_BUFFERS or IORING_OP_REMOVE_BUFFERS), so the fix is not good.

Given that io_iopoll_complete() is always called under uring_lock, so here for polled io, we can also get uring_lock to fix this issue.

This returns it to the state it was before fixing + mutex locking for IOPOLL, and it's much better than having it half-broken as it is now.

Quoted 47 lines
Fixes dad1b1242fd5 ("io_uring: always let io_iopoll_complete() complete polled io")
---
 fs/io_uring.c | 25 +++++++++++++++----------
 1 file changed, 15 insertions(+), 10 deletions(-)
fs/io_uring.c +15 -10
index f53356ced5ab..eab3d2b7d232 100644
@@ -6354,19 +6354,24 @@ static struct io_wq_work *io_wq_submit_work(struct io_wq_work *work)
 	}
 
 	if (ret) {
+		bool iopoll_enabled = req->ctx->flags & IORING_SETUP_IOPOLL;
+
 		/*
-		 * io_iopoll_complete() does not hold completion_lock to complete
-		 * polled io, so here for polled io, just mark it done and still let
-		 * io_iopoll_complete() complete it.
+		 * io_iopoll_complete() does not hold completion_lock to complete polled
+		 * io, so here for polled io, we can not call io_req_complete() directly,
+		 * otherwise there maybe concurrent access to cqring, defer_list, etc,
+		 * which is not safe. Given that io_iopoll_complete() is always called
+		 * under uring_lock, so here for polled io, we also get uring_lock to
+		 * complete it.
 		 */
-		if (req->ctx->flags & IORING_SETUP_IOPOLL) {
-			struct kiocb *kiocb = &req->rw.kiocb;
+		if (iopoll_enabled)
+			mutex_lock(&req->ctx->uring_lock);
 
-			kiocb_done(kiocb, ret, NULL);
-		} else {
-			req_set_fail_links(req);
-			io_req_complete(req, ret);
-		}
+		req_set_fail_links(req);
+		io_req_complete(req, ret);
+
+		if (iopoll_enabled)
+			mutex_unlock(&req->ctx->uring_lock);
 	}
 
 	return io_steal_work(req);
Signature · 1 line
Pavel Begunkov
#5 lore raw
Quoted 10 lines · Pavel Begunkov
Quoted 9 lines · Xiaoguang Wang

io_iopoll_complete() does not hold completion_lock to complete polled io, so in io_wq_submit_work(), we can not call io_req_complete() directly, to complete polled io, otherwise there maybe concurrent access to cqring, defer_list, etc, which is not safe. Commit dad1b1242fd5 ("io_uring: always let io_iopoll_complete() complete polled io") has fixed this issue, but Pavel reported that IOPOLL apart from rw can do buf reg/unreg requests( IORING_OP_PROVIDE_BUFFERS or IORING_OP_REMOVE_BUFFERS), so the fix is not good.

Quoted 5 lines
Quoted 2 lines

Given that io_iopoll_complete() is always called under uring_lock, so here for polled io, we can also get uring_lock to fix this issue.

This returns it to the state it was before fixing + mutex locking for IOPOLL, and it's much better than having it half-broken as it is now.

btw, comments are over 80, but that's minor.

Quoted 52 lines
Quoted 47 lines
Fixes dad1b1242fd5 ("io_uring: always let io_iopoll_complete() complete polled io")
---
 fs/io_uring.c | 25 +++++++++++++++----------
 1 file changed, 15 insertions(+), 10 deletions(-)
fs/io_uring.c +15 -10
index f53356ced5ab..eab3d2b7d232 100644
@@ -6354,19 +6354,24 @@ static struct io_wq_work *io_wq_submit_work(struct io_wq_work *work)
 	}
 
 	if (ret) {
+		bool iopoll_enabled = req->ctx->flags & IORING_SETUP_IOPOLL;
+
 		/*
-		 * io_iopoll_complete() does not hold completion_lock to complete
-		 * polled io, so here for polled io, just mark it done and still let
-		 * io_iopoll_complete() complete it.
+		 * io_iopoll_complete() does not hold completion_lock to complete polled
+		 * io, so here for polled io, we can not call io_req_complete() directly,
+		 * otherwise there maybe concurrent access to cqring, defer_list, etc,
+		 * which is not safe. Given that io_iopoll_complete() is always called
+		 * under uring_lock, so here for polled io, we also get uring_lock to
+		 * complete it.
 		 */
-		if (req->ctx->flags & IORING_SETUP_IOPOLL) {
-			struct kiocb *kiocb = &req->rw.kiocb;
+		if (iopoll_enabled)
+			mutex_lock(&req->ctx->uring_lock);
 
-			kiocb_done(kiocb, ret, NULL);
-		} else {
-			req_set_fail_links(req);
-			io_req_complete(req, ret);
-		}
+		req_set_fail_links(req);
+		io_req_complete(req, ret);
+
+		if (iopoll_enabled)
+			mutex_unlock(&req->ctx->uring_lock);
 	}
 
 	return io_steal_work(req);
Signature · 1 line
Pavel Begunkov
maintainer Reviewed-by Pavel Begunkov
#6 lore raw
Quoted 11 lines · Pavel Begunkov
Quoted 10 lines · Pavel Begunkov
Quoted 9 lines · Xiaoguang Wang

io_iopoll_complete() does not hold completion_lock to complete polled io, so in io_wq_submit_work(), we can not call io_req_complete() directly, to complete polled io, otherwise there maybe concurrent access to cqring, defer_list, etc, which is not safe. Commit dad1b1242fd5 ("io_uring: always let io_iopoll_complete() complete polled io") has fixed this issue, but Pavel reported that IOPOLL apart from rw can do buf reg/unreg requests( IORING_OP_PROVIDE_BUFFERS or IORING_OP_REMOVE_BUFFERS), so the fix is not good.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Given that io_iopoll_complete() is always called under uring_lock, so here for polled io, we can also get uring_lock to fix this issue.

This returns it to the state it was before fixing + mutex locking for IOPOLL, and it's much better than having it half-broken as it is now.

btw, comments are over 80, but that's minor.

I fixed that up, but I don't particularly like how 'req' is used after calling complete. How about the below variant - same as before, just using the ctx instead to determine if we need to lock it or not.

commit 253b60e7d8adcb980be91f77e64968a58d836b5e

Author: Xiaoguang Wang <[email protected]>
Date:   Mon Dec 14 23:49:41 2020 +0800

io_uring: hold uring_lock while completing failed polled io in io_wq_submit_work()

io_iopoll_complete() does not hold completion_lock to complete polled io, so in io_wq_submit_work(), we can not call io_req_complete() directly, to complete polled io, otherwise there maybe concurrent access to cqring, defer_list, etc, which is not safe. Commit dad1b1242fd5 ("io_uring: always let io_iopoll_complete() complete polled io") has fixed this issue, but Pavel reported that IOPOLL apart from rw can do buf reg/unreg requests( IORING_OP_PROVIDE_BUFFERS or IORING_OP_REMOVE_BUFFERS), so the fix is not good.

Given that io_iopoll_complete() is always called under uring_lock, so here for polled io, we can also get uring_lock to fix this issue.

Fixes dad1b1242fd5 ("io_uring: always let io_iopoll_complete() complete polled io")
Cc <[email protected]> # 5.5+
[axboe: don't deref 'req' after completing it']
fs/io_uring.c +19 -10
index b27f61e3e0d6..0a8cf3fad955 100644
@@ -6332,19 +6332,28 @@ static struct io_wq_work *io_wq_submit_work(struct io_wq_work *work)
 	}
 
 	if (ret) {
+		struct io_ring_ctx *lock_ctx = NULL;
+
+		if (req->ctx->flags & IORING_SETUP_IOPOLL)
+			lock_ctx = req->ctx;
+
 		/*
-		 * io_iopoll_complete() does not hold completion_lock to complete
-		 * polled io, so here for polled io, just mark it done and still let
-		 * io_iopoll_complete() complete it.
+		 * io_iopoll_complete() does not hold completion_lock to
+		 * complete polled io, so here for polled io, we can not call
+		 * io_req_complete() directly, otherwise there maybe concurrent
+		 * access to cqring, defer_list, etc, which is not safe. Given
+		 * that io_iopoll_complete() is always called under uring_lock,
+		 * so here for polled io, we also get uring_lock to complete
+		 * it.
 		 */
-		if (req->ctx->flags & IORING_SETUP_IOPOLL) {
-			struct kiocb *kiocb = &req->rw.kiocb;
+		if (lock_ctx)
+			mutex_lock(&lock_ctx->uring_lock);
 
-			kiocb_done(kiocb, ret, NULL);
-		} else {
-			req_set_fail_links(req);
-			io_req_complete(req, ret);
-		}
+		req_set_fail_links(req);
+		io_req_complete(req, ret);
+
+		if (lock_ctx)
+			mutex_unlock(&lock_ctx->uring_lock);
 	}
 
 	return io_steal_work(req);
Signature · 1 line
Jens Axboe
#7 lore raw

hi,

Quoted 15 lines · Pavel Begunkov
Quoted 14 lines · Pavel Begunkov
Quoted 13 lines · Xiaoguang Wang
Quoted 11 lines

io_iopoll_complete() does not hold completion_lock to complete polled io, so in io_wq_submit_work(), we can not call io_req_complete() directly, to complete polled io, otherwise there maybe concurrent access to cqring, defer_list, etc, which is not safe. Commit dad1b1242fd5 ("io_uring: always let io_iopoll_complete() complete polled io") has fixed this issue, but Pavel reported that IOPOLL apart from rw can do buf reg/unreg requests( IORING_OP_PROVIDE_BUFFERS or IORING_OP_REMOVE_BUFFERS), so the fix is not good.

Given that io_iopoll_complete() is always called under uring_lock, so here for polled io, we can also get uring_lock to fix this issue.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

This returns it to the state it was before fixing + mutex locking for IOPOLL, and it's much better than having it half-broken as it is now.

btw, comments are over 80, but that's minor.

I fixed that up, but I don't particularly like how 'req' is used after calling complete. How about the below variant - same as before, just using the ctx instead to determine if we need to lock it or not.

It looks better, thanks.

Regards,
Xiaoguang Wang

Quoted 71 lines

commit 253b60e7d8adcb980be91f77e64968a58d836b5e

Author: Xiaoguang Wang <[email protected]>
Date:   Mon Dec 14 23:49:41 2020 +0800

io_uring: hold uring_lock while completing failed polled io in io_wq_submit_work()

io_iopoll_complete() does not hold completion_lock to complete polled io, so in io_wq_submit_work(), we can not call io_req_complete() directly, to complete polled io, otherwise there maybe concurrent access to cqring, defer_list, etc, which is not safe. Commit dad1b1242fd5 ("io_uring: always let io_iopoll_complete() complete polled io") has fixed this issue, but Pavel reported that IOPOLL apart from rw can do buf reg/unreg requests( IORING_OP_PROVIDE_BUFFERS or IORING_OP_REMOVE_BUFFERS), so the fix is not good.

Given that io_iopoll_complete() is always called under uring_lock, so here for polled io, we can also get uring_lock to fix this issue.

Fixes dad1b1242fd5 ("io_uring: always let io_iopoll_complete() complete polled io")
Cc <[email protected]> # 5.5+
[axboe: don't deref 'req' after completing it']
fs/io_uring.c +19 -10
index b27f61e3e0d6..0a8cf3fad955 100644
@@ -6332,19 +6332,28 @@ static struct io_wq_work *io_wq_submit_work(struct io_wq_work *work)
  	}
 
  	if (ret) {
+		struct io_ring_ctx *lock_ctx = NULL;
+
+		if (req->ctx->flags & IORING_SETUP_IOPOLL)
+			lock_ctx = req->ctx;
+
  		/*
-		 * io_iopoll_complete() does not hold completion_lock to complete
-		 * polled io, so here for polled io, just mark it done and still let
-		 * io_iopoll_complete() complete it.
+		 * io_iopoll_complete() does not hold completion_lock to
+		 * complete polled io, so here for polled io, we can not call
+		 * io_req_complete() directly, otherwise there maybe concurrent
+		 * access to cqring, defer_list, etc, which is not safe. Given
+		 * that io_iopoll_complete() is always called under uring_lock,
+		 * so here for polled io, we also get uring_lock to complete
+		 * it.
  		 */
-		if (req->ctx->flags & IORING_SETUP_IOPOLL) {
-			struct kiocb *kiocb = &req->rw.kiocb;
+		if (lock_ctx)
+			mutex_lock(&lock_ctx->uring_lock);
 
-			kiocb_done(kiocb, ret, NULL);
-		} else {
-			req_set_fail_links(req);
-			io_req_complete(req, ret);
-		}
+		req_set_fail_links(req);
+		io_req_complete(req, ret);
+
+		if (lock_ctx)
+			mutex_unlock(&lock_ctx->uring_lock);
  	}
 
  	return io_steal_work(req);