Skip to contentSkip to conversation

fixes of poll

PATCH In review Hao Xu 2 participants 6 messages Sep 27, 2021 – Sep 29, 2021

Subsystems (2)
lore
Share & tools
lore

Hao Xu

author
#0 lore raw

cover fixes of poll

Two fixes to address poll issues.

Hao Xu (2):
io_uring: fix concurrent poll interruption
io_uring: fix tw list mess-up by adding tw while it's already in tw
list

 fs/io_uring.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
Signature · 1 line
2.24.4
author
#1 lore raw

PATCH 1/2 io_uring: fix concurrent poll interruption

There may be concurrent poll interruptions:

async_wake()
->del wait entry
->add to tw list

                       async_wake()
                       ->del wait entry
                       ->add to tw list

This mess up the tw list, let's avoid it by adding a if check before delete wait entry:
async_wake()
->if empty(wait entry)
return
->del wait entry
->add to tw list

                       async_wake()
                       ->if empty(wait entry)
                           return <------------will return here
                       ->del wait entry
                       ->add to tw list
Fixes 88e41cf928a6 ("io_uring: add multishot mode for IORING_OP_POLL_ADD")
---
 fs/io_uring.c | 2 ++
 1 file changed, 2 insertions(+)
fs/io_uring.c +2 -0
index 8317c360f7a4..d0b358b9b589 100644
@@ -5245,6 +5245,8 @@ static int __io_async_wake(struct io_kiocb *req, struct io_poll_iocb *poll,
 
 	trace_io_uring_task_add(req->ctx, req->opcode, req->user_data, mask);
 
+	if (list_empty(&poll->wait.entry))
+		return 0;
 	list_del_init(&poll->wait.entry);
 
 	req->result = mask;
Signature · 1 line
2.24.4
author
#2 lore raw

Re: [PATCH 1/2] io_uring: fix concurrent poll interruption

Quoted 16 lines · Hao Xu

There may be concurrent poll interruptions:

async_wake()
->del wait entry
->add to tw list

                        async_wake()
                        ->del wait entry
                        ->add to tw list

This mess up the tw list, let's avoid it by adding a if check before delete wait entry:
async_wake()
->if empty(wait entry)
return
->del wait entry
->add to tw list

Quoted 5 lines
                        async_wake()
                        ->if empty(wait entry)
                            return <------------will return here
                        ->del wait entry
                        ->add to tw list

I now think the issue may not exist, since async_wake() is protected by poll->head->lock, so the wait entry has already been deleted when the second time event happens and try to visit the wait list to wakeup it.

Quoted 21 lines
Fixes 88e41cf928a6 ("io_uring: add multishot mode for IORING_OP_POLL_ADD")
---
  fs/io_uring.c | 2 ++
  1 file changed, 2 insertions(+)
fs/io_uring.c +2 -0
index 8317c360f7a4..d0b358b9b589 100644
@@ -5245,6 +5245,8 @@ static int __io_async_wake(struct io_kiocb *req, struct io_poll_iocb *poll,
 
  	trace_io_uring_task_add(req->ctx, req->opcode, req->user_data, mask);
 
+	if (list_empty(&poll->wait.entry))
+		return 0;
  	list_del_init(&poll->wait.entry);
 
  	req->result = mask;
author
#3 lore raw

PATCH 2/2 io_uring: fix tw list mess-up by adding tw while it's already in tw list

In reply to Hao Xu

For multishot mode, there may be cases like:
io_poll_task_func()
-> add_wait_queue()

                            async_wake()
                            ->io_req_task_work_add()
                            this one mess up the running task_work list
                            since req->io_task_work.node is in use.

similar situation for req->io_task_work.fallback_node.
Fix it by set node->next = NULL before we run the tw, so that when we add req back to the wait queue in middle of tw running, we can safely re-add it to the tw list.

Fixes 7cbf1722d5fc ("io_uring: provide FIFO ordering for task_work")
---
 fs/io_uring.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
fs/io_uring.c +8 -3
index d0b358b9b589..f667d6286438 100644
@@ -1250,13 +1250,17 @@ static void io_fallback_req_func(struct work_struct *work)
 	struct io_ring_ctx *ctx = container_of(work, struct io_ring_ctx,
 						fallback_work.work);
 	struct llist_node *node = llist_del_all(&ctx->fallback_llist);
-	struct io_kiocb *req, *tmp;
+	struct io_kiocb *req;
 	bool locked = false;
 
 	percpu_ref_get(&ctx->refs);
-	llist_for_each_entry_safe(req, tmp, node, io_task_work.fallback_node)
+	req = llist_entry(node, struct io_kiocb, io_task_work.fallback_node);
+	while (member_address_is_nonnull(req, io_task_work.fallback_node)) {
+		node = req->io_task_work.fallback_node.next;
+		req->io_task_work.fallback_node.next = NULL;
 		req->io_task_work.func(req, &locked);
-
+		req = llist_entry(node, struct io_kiocb, io_task_work.fallback_node);
+	}
 	if (locked) {
 		io_submit_flush_completions(ctx);
 		mutex_unlock(&ctx->uring_lock);
@@ -2156,6 +2160,7 @@ static void tctx_task_work(struct callback_head *cb)
 				locked = mutex_trylock(&ctx->uring_lock);
 				percpu_ref_get(&ctx->refs);
 			}
+			node->next = NULL;
 			req->io_task_work.func(req, &locked);
 			node = next;
 		} while (node);
Signature · 1 line
2.24.4
#4 lore raw

Re: [PATCH 2/2] io_uring: fix tw list mess-up by adding tw while it's already in tw list

Quoted 7 lines · Hao Xu

For multishot mode, there may be cases like:
io_poll_task_func()
-> add_wait_queue()

                            async_wake()
                            ->io_req_task_work_add()
                            this one mess up the running task_work list
                            since req->io_task_work.node is in use.

By the time req->io_task_work.func() is called, node->next is undefined and free to use. io_req_task_work_add() will override it without looking at a prior value and that's fine, as the calling code doesn't touch it after the callback.

Quoted 4 lines

similar situation for req->io_task_work.fallback_node.
Fix it by set node->next = NULL before we run the tw, so that when we add req back to the wait queue in middle of tw running, we can safely re-add it to the tw list.

I might be missing what is the problem you're trying to fix. Does the above makes sense? It doesn't sound like node->next=NULL can solve anything.

Quoted 40 lines
Fixes 7cbf1722d5fc ("io_uring: provide FIFO ordering for task_work")
---
 fs/io_uring.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
fs/io_uring.c +8 -3
index d0b358b9b589..f667d6286438 100644
@@ -1250,13 +1250,17 @@ static void io_fallback_req_func(struct work_struct *work)
 	struct io_ring_ctx *ctx = container_of(work, struct io_ring_ctx,
 						fallback_work.work);
 	struct llist_node *node = llist_del_all(&ctx->fallback_llist);
-	struct io_kiocb *req, *tmp;
+	struct io_kiocb *req;
 	bool locked = false;
 
 	percpu_ref_get(&ctx->refs);
-	llist_for_each_entry_safe(req, tmp, node, io_task_work.fallback_node)
+	req = llist_entry(node, struct io_kiocb, io_task_work.fallback_node);
+	while (member_address_is_nonnull(req, io_task_work.fallback_node)) {
+		node = req->io_task_work.fallback_node.next;
+		req->io_task_work.fallback_node.next = NULL;
 		req->io_task_work.func(req, &locked);
-
+		req = llist_entry(node, struct io_kiocb, io_task_work.fallback_node);
+	}
 	if (locked) {
 		io_submit_flush_completions(ctx);
 		mutex_unlock(&ctx->uring_lock);
@@ -2156,6 +2160,7 @@ static void tctx_task_work(struct callback_head *cb)
 				locked = mutex_trylock(&ctx->uring_lock);
 				percpu_ref_get(&ctx->refs);
 			}
+			node->next = NULL;
 			req->io_task_work.func(req, &locked);
 			node = next;
 		} while (node);
Signature · 1 line
Pavel Begunkov
author
#5 lore raw

Re: [PATCH 2/2] io_uring: fix tw list mess-up by adding tw while it's already in tw list

Quoted 9 lines · Pavel Begunkov
Quoted 7 lines · Hao Xu

For multishot mode, there may be cases like:
io_poll_task_func()
-> add_wait_queue()

                             async_wake()
                             ->io_req_task_work_add()
                             this one mess up the running task_work list
                             since req->io_task_work.node is in use.
Quoted 4 lines

By the time req->io_task_work.func() is called, node->next is undefined and free to use. io_req_task_work_add() will override it without looking at a prior value and that's fine, as the calling code doesn't touch it after the callback.

I misunderstood the code, since node->next will be reset to NULL in wq_list_add_tail(), so no problem here.

Quoted 51 lines
Quoted 5 lines

similar situation for req->io_task_work.fallback_node.
Fix it by set node->next = NULL before we run the tw, so that when we add req back to the wait queue in middle of tw running, we can safely re-add it to the tw list.

I might be missing what is the problem you're trying to fix. Does the above makes sense? It doesn't sound like node->next=NULL can solve anything.

Quoted 40 lines
Fixes 7cbf1722d5fc ("io_uring: provide FIFO ordering for task_work")
---
  fs/io_uring.c | 11 ++++++++---
  1 file changed, 8 insertions(+), 3 deletions(-)
fs/io_uring.c +8 -3
index d0b358b9b589..f667d6286438 100644
@@ -1250,13 +1250,17 @@ static void io_fallback_req_func(struct work_struct *work)
  	struct io_ring_ctx *ctx = container_of(work, struct io_ring_ctx,
  						fallback_work.work);
  	struct llist_node *node = llist_del_all(&ctx->fallback_llist);
-	struct io_kiocb *req, *tmp;
+	struct io_kiocb *req;
  	bool locked = false;
 
  	percpu_ref_get(&ctx->refs);
-	llist_for_each_entry_safe(req, tmp, node, io_task_work.fallback_node)
+	req = llist_entry(node, struct io_kiocb, io_task_work.fallback_node);
+	while (member_address_is_nonnull(req, io_task_work.fallback_node)) {
+		node = req->io_task_work.fallback_node.next;
+		req->io_task_work.fallback_node.next = NULL;
  		req->io_task_work.func(req, &locked);
-
+		req = llist_entry(node, struct io_kiocb, io_task_work.fallback_node);
+	}
  	if (locked) {
  		io_submit_flush_completions(ctx);
  		mutex_unlock(&ctx->uring_lock);
@@ -2156,6 +2160,7 @@ static void tctx_task_work(struct callback_head *cb)
  				locked = mutex_trylock(&ctx->uring_lock);
  				percpu_ref_get(&ctx->refs);
  			}
+			node->next = NULL;
  			req->io_task_work.func(req, &locked);
  			node = next;
  		} while (node);