Skip to contentSkip to conversation

poll fixes

PATCH In review Hao Xu 2 participants 5 messages Sep 22, 2021

Subsystems (2)
lore
Share & tools
lore

Hao Xu

author
#0 lore raw

cover poll fixes

Three poll fixes.

Hao Xu (3):
io_uring: fix race between poll completion and cancel_hash insertion
io_uring: fix lacking of EPOLLONESHOT
io_uring: fix potential req refcount underflow

 fs/io_uring.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
Signature · 1 line
2.24.4
author
#1 lore raw

PATCH 1/3 io_uring: fix race between poll completion and cancel_hash insertion

If poll arming and poll completion runs parallelly, there maybe races. For instance, run io_poll_add in iowq and io_poll_task_func in original context, then:
iowq original context
io_poll_add

    vfs_poll
     (interruption happens
      tw queued to original
      context)                              io_poll_task_func
                                              generate cqe
                                              del from cancel_hash[]
    if !poll.done
      insert to cancel_hash[]

The entry left in cancel_hash[], similar case for fast poll. Fix it by set poll.done = true when del from cancel_hash[].

Fixes 5082620fb2ca ("io_uring: terminate multishot poll for CQ ring overflow")
---
 fs/io_uring.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
fs/io_uring.c +3 -3
index 91e4c89abf78..4b0a40ad28b0 100644
@@ -5340,10 +5340,8 @@ static bool __io_poll_complete(struct io_kiocb *req, __poll_t mask)
 	}
 	if (req->poll.events & EPOLLONESHOT)
 		flags = 0;
-	if (!io_cqring_fill_event(ctx, req->user_data, error, flags)) {
-		req->poll.done = true;
+	if (!io_cqring_fill_event(ctx, req->user_data, error, flags))
 		flags = 0;
-	}
 	if (flags & IORING_CQE_F_MORE)
 		ctx->cq_extra++;
 
@@ -5374,6 +5372,7 @@ static void io_poll_task_func(struct io_kiocb *req, bool *locked)
 		if (done) {
 			io_poll_remove_double(req);
 			hash_del(&req->hash_node);
+			req->poll.done = true;
 		} else {
 			req->result = 0;
 			add_wait_queue(req->poll.head, &req->poll.wait);
@@ -5511,6 +5510,7 @@ static void io_async_task_func(struct io_kiocb *req, bool *locked)
 
 	hash_del(&req->hash_node);
 	io_poll_remove_double(req);
+	apoll->poll.done = true;
 	spin_unlock(&ctx->completion_lock);
 
 	if (!READ_ONCE(apoll->poll.canceled))
Signature · 1 line
2.24.4
author
#2 lore raw

PATCH 2/3 io_uring: fix lacking of EPOLLONESHOT

In reply to Hao Xu

We should set EPOLLONESHOT if cqring_fill_event() returns false since io_poll_add() decides to put req or not by it.

Fixes 5082620fb2ca ("io_uring: terminate multishot poll for CQ ring overflow")
---
 fs/io_uring.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
fs/io_uring.c +3 -1
index 4b0a40ad28b0..b7c9fcce2de2 100644
@@ -5340,8 +5340,10 @@ static bool __io_poll_complete(struct io_kiocb *req, __poll_t mask)
 	}
 	if (req->poll.events & EPOLLONESHOT)
 		flags = 0;
-	if (!io_cqring_fill_event(ctx, req->user_data, error, flags))
+	if (!io_cqring_fill_event(ctx, req->user_data, error, flags)) {
+		req->poll.events |= EPOLLONESHOT;
 		flags = 0;
+	}
 	if (flags & IORING_CQE_F_MORE)
 		ctx->cq_extra++;
 
Signature · 1 line
2.24.4
author
#3 lore raw

PATCH 3/3 io_uring: fix potential req refcount underflow

In reply to Hao Xu

For multishot mode, there may be cases like:
iowq original context
io_poll_add
_arm_poll()
mask = vfs_poll() is not 0
if mask
(2) io_poll_complete()
compl_unlock
(interruption happens
tw queued to original
context)
io_poll_task_func() compl_lock
(3) done = io_poll_complete() is true compl_unlock
put req ref
(1) if (poll->flags & EPOLLONESHOT)
put req ref

EPOLLONESHOT flag in (1) may be from (2) or (3), so there are multiple combinations that can cause ref underfow.
Let's address it by:
- check the return value in (2) as done
- change (1) to if (done)
in this way, we only do ref put in (1) if 'oneshot flag' is from
(2)
- do poll.done check in io_poll_task_func(), so that we won't put ref for the second time.

---
 fs/io_uring.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
fs/io_uring.c +7 -2
index b7c9fcce2de2..b008edccad46 100644
@@ -5370,6 +5370,10 @@ static void io_poll_task_func(struct io_kiocb *req, bool *locked)
 	} else {
 		bool done;
 
+		if (req->poll.done) {
+			spin_unlock(&ctx->completion_lock);
+			return;
+		}
 		done = __io_poll_complete(req, req->result);
 		if (done) {
 			io_poll_remove_double(req);
@@ -5833,6 +5837,7 @@ static int io_poll_add(struct io_kiocb *req, unsigned int issue_flags)
 	struct io_ring_ctx *ctx = req->ctx;
 	struct io_poll_table ipt;
 	__poll_t mask;
+	bool done;
 
 	ipt.pt._qproc = io_poll_queue_proc;
 
@@ -5841,13 +5846,13 @@ static int io_poll_add(struct io_kiocb *req, unsigned int issue_flags)
 
 	if (mask) { /* no async, we'd stolen it */
 		ipt.error = 0;
-		io_poll_complete(req, mask);
+		done = io_poll_complete(req, mask);
 	}
 	spin_unlock(&ctx->completion_lock);
 
 	if (mask) {
 		io_cqring_ev_posted(ctx);
-		if (poll->events & EPOLLONESHOT)
+		if (done)
 			io_put_req(req);
 	}
 	return ipt.error;
Signature · 1 line
2.24.4
maintainer
#4 lore raw
In reply to Hao Xu
Quoted 9 lines · Hao Xu

Three poll fixes.

Hao Xu (3):
io_uring: fix race between poll completion and cancel_hash insertion
io_uring: fix lacking of EPOLLONESHOT
io_uring: fix potential req refcount underflow

 fs/io_uring.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

These look good to me, nice fixes! I'll run this through some testing and get them applied for 5.15. Thanks.

Signature · 1 line
Jens Axboe