Skip to contentSkip to conversation

improvements for poll requests

RFC Superseded Xiaoguang Wang 4 participants 11 messages Sep 22, 2021 – Sep 24, 2021

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover improvements for poll requests

This patchset tries to improve echo_server model based on io_uring's IORING_POLL_ADD_MULTI feature.

Xiaoguang Wang (3):
io_uring: reduce frequent add_wait_queue() overhead for multi-shot
poll request
io_uring: don't get completion_lock in io_poll_rewait()
io_uring: try to batch poll request completion

 fs/io_uring.c | 116 ++++++++++++++++++++++++++++++++++++++++++++++------------
 1 file changed, 92 insertions(+), 24 deletions(-)
Signature · 1 line
2.14.4.44.g2045bb6
#1 lore raw

RFC 1/3 io_uring: reduce frequent add_wait_queue() overhead for multi-shot poll request

Run echo_server to evaluate io_uring's multi-shot poll performance, perf shows that add_wait_queue() has obvious overhead. Intruduce a new state 'active' in io_poll_iocb to indicate whether io_poll_wake() should queue a task_work. This new state will be set to true initially, be set to false when starting to queue a task work, and be set to true again when a poll cqe has been committed. One concern is that this method may lost waken-up event, but seems it's ok.

io_poll_wake io_poll_task_func

t1                       |
t2                       |    WRITE_ONCE(req->poll.active, true);
t3                       |
t4                       |    io_commit_cqring(ctx);
t5                       |
t6                       |

If waken-up events happens before or at t4, it's ok, user app will always see a cqe. If waken-up events happens after t4 and IIUC, io_poll_wake() will see the new req->poll.active value by using READ_ONCE().

With this patch, a pure echo_server(1000 connections, packet is 16 bytes) shows about 1.6% reqs improvement.

---
 fs/io_uring.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
fs/io_uring.c +16 -4
index 1294b1ef4acb..ca4464a75c7b 100644
@@ -487,6 +487,7 @@ struct io_poll_iocb {
 	__poll_t			events;
 	bool				done;
 	bool				canceled;
+	bool				active;
 	struct wait_queue_entry		wait;
 };
 
@@ -5025,8 +5026,6 @@ static int __io_async_wake(struct io_kiocb *req, struct io_poll_iocb *poll,
 
 	trace_io_uring_task_add(req->ctx, req->opcode, req->user_data, mask);
 
-	list_del_init(&poll->wait.entry);
-
 	req->result = mask;
 	req->io_task_work.func = func;
 
@@ -5057,7 +5056,10 @@ static bool io_poll_rewait(struct io_kiocb *req, struct io_poll_iocb *poll)
 
 	spin_lock(&ctx->completion_lock);
 	if (!req->result && !READ_ONCE(poll->canceled)) {
-		add_wait_queue(poll->head, &poll->wait);
+		if (req->opcode == IORING_OP_POLL_ADD)
+			WRITE_ONCE(req->poll.active, true);
+		else
+			add_wait_queue(poll->head, &poll->wait);
 		return true;
 	}
 
@@ -5133,6 +5135,9 @@ static inline bool io_poll_complete(struct io_kiocb *req, __poll_t mask)
 	return done;
 }
 
+static bool __io_poll_remove_one(struct io_kiocb *req,
+				 struct io_poll_iocb *poll, bool do_cancel);
+
 static void io_poll_task_func(struct io_kiocb *req, bool *locked)
 {
 	struct io_ring_ctx *ctx = req->ctx;
@@ -5146,10 +5151,11 @@ static void io_poll_task_func(struct io_kiocb *req, bool *locked)
 		done = __io_poll_complete(req, req->result);
 		if (done) {
 			io_poll_remove_double(req);
+			__io_poll_remove_one(req, io_poll_get_single(req), true);
 			hash_del(&req->hash_node);
 		} else {
 			req->result = 0;
-			add_wait_queue(req->poll.head, &req->poll.wait);
+			WRITE_ONCE(req->poll.active, true);
 		}
 		io_commit_cqring(ctx);
 		spin_unlock(&ctx->completion_lock);
@@ -5204,6 +5210,7 @@ static void io_init_poll_iocb(struct io_poll_iocb *poll, __poll_t events,
 	poll->head = NULL;
 	poll->done = false;
 	poll->canceled = false;
+	poll->active = true;
 #define IO_POLL_UNMASK	(EPOLLERR|EPOLLHUP|EPOLLNVAL|EPOLLRDHUP)
 	/* mask in events that we always want/need */
 	poll->events = events | IO_POLL_UNMASK;
@@ -5301,6 +5308,7 @@ static int io_async_wake(struct wait_queue_entry *wait, unsigned mode, int sync,
 	trace_io_uring_poll_wake(req->ctx, req->opcode, req->user_data,
 					key_to_poll(key));
 
+	list_del_init(&poll->wait.entry);
 	return __io_async_wake(req, poll, key_to_poll(key), io_async_task_func);
 }
 
@@ -5569,6 +5577,10 @@ static int io_poll_wake(struct wait_queue_entry *wait, unsigned mode, int sync,
 	struct io_kiocb *req = wait->private;
 	struct io_poll_iocb *poll = &req->poll;
 
+	if (!READ_ONCE(poll->active))
+		return 0;
+
+	WRITE_ONCE(poll->active, false);
 	return __io_async_wake(req, poll, key_to_poll(key), io_poll_task_func);
 }
 
Signature · 1 line
2.14.4.44.g2045bb6
#2 lore raw

Re: [RFC 1/3] io_uring: reduce frequent add_wait_queue() overhead for multi-shot poll request

Quoted 72 lines · Xiaoguang Wang

Run echo_server to evaluate io_uring's multi-shot poll performance, perf shows that add_wait_queue() has obvious overhead. Intruduce a new state 'active' in io_poll_iocb to indicate whether io_poll_wake() should queue a task_work. This new state will be set to true initially, be set to false when starting to queue a task work, and be set to true again when a poll cqe has been committed. One concern is that this method may lost waken-up event, but seems it's ok.

io_poll_wake io_poll_task_func

t1                       |
t2                       |    WRITE_ONCE(req->poll.active, true);
t3                       |
t4                       |    io_commit_cqring(ctx);
t5                       |
t6                       |

If waken-up events happens before or at t4, it's ok, user app will always see a cqe. If waken-up events happens after t4 and IIUC, io_poll_wake() will see the new req->poll.active value by using READ_ONCE().

With this patch, a pure echo_server(1000 connections, packet is 16 bytes) shows about 1.6% reqs improvement.

---
  fs/io_uring.c | 20 ++++++++++++++++----
  1 file changed, 16 insertions(+), 4 deletions(-)
fs/io_uring.c +8 -3
index 1294b1ef4acb..ca4464a75c7b 100644
@@ -487,6 +487,7 @@ struct io_poll_iocb {
  	__poll_t			events;
  	bool				done;
  	bool				canceled;
+	bool				active;
  	struct wait_queue_entry		wait;
  };
 
@@ -5025,8 +5026,6 @@ static int __io_async_wake(struct io_kiocb *req, struct io_poll_iocb *poll,
 
  	trace_io_uring_task_add(req->ctx, req->opcode, req->user_data, mask);
 
-	list_del_init(&poll->wait.entry);
-
  	req->result = mask;
  	req->io_task_work.func = func;
 
@@ -5057,7 +5056,10 @@ static bool io_poll_rewait(struct io_kiocb *req, struct io_poll_iocb *poll)
 
  	spin_lock(&ctx->completion_lock);
  	if (!req->result && !READ_ONCE(poll->canceled)) {
-		add_wait_queue(poll->head, &poll->wait);
+		if (req->opcode == IORING_OP_POLL_ADD)
+			WRITE_ONCE(req->poll.active, true);
+		else
+			add_wait_queue(poll->head, &poll->wait);
  		return true;
  	}
 
@@ -5133,6 +5135,9 @@ static inline bool io_poll_complete(struct io_kiocb *req, __poll_t mask)
  	return done;
  }
 
+static bool __io_poll_remove_one(struct io_kiocb *req,
+				 struct io_poll_iocb *poll, bool do_cancel);
+
  static void io_poll_task_func(struct io_kiocb *req, bool *locked)
  {
  	struct io_ring_ctx *ctx = req->ctx;

fs/io_uring.c static void io_poll_task_func(struct io_kiocb *req, bool *locked) · line 5154

Quoted 4 lines
fs/io_uring.c +1 -0
@@ -5146,10 +5151,11 @@ static void io_poll_task_func(struct io_kiocb *req, bool *locked)
  		done = __io_poll_complete(req, req->result);
  		if (done) {
  			io_poll_remove_double(req);
+			__io_poll_remove_one(req, io_poll_get_single(req), true);

This may cause race problems, like there may be multiple cancelled cqes considerring io_poll_add() parallelled. hash_del is redundant either. __io_poll_remove_one may not be the best choice here, and since we now don't del wait entry inbetween, code in _arm_poll should probably be tweaked as well(not very sure, will dive into it tomorrow).

Regards,
Hao

Quoted 36 lines
fs/io_uring.c +7 -1
  			hash_del(&req->hash_node);
  		} else {
  			req->result = 0;
-			add_wait_queue(req->poll.head, &req->poll.wait);
+			WRITE_ONCE(req->poll.active, true);
  		}
  		io_commit_cqring(ctx);
  		spin_unlock(&ctx->completion_lock);
@@ -5204,6 +5210,7 @@ static void io_init_poll_iocb(struct io_poll_iocb *poll, __poll_t events,
  	poll->head = NULL;
  	poll->done = false;
  	poll->canceled = false;
+	poll->active = true;
  #define IO_POLL_UNMASK	(EPOLLERR|EPOLLHUP|EPOLLNVAL|EPOLLRDHUP)
  	/* mask in events that we always want/need */
  	poll->events = events | IO_POLL_UNMASK;
@@ -5301,6 +5308,7 @@ static int io_async_wake(struct wait_queue_entry *wait, unsigned mode, int sync,
  	trace_io_uring_poll_wake(req->ctx, req->opcode, req->user_data,
  					key_to_poll(key));
 
+	list_del_init(&poll->wait.entry);
  	return __io_async_wake(req, poll, key_to_poll(key), io_async_task_func);
  }
 
@@ -5569,6 +5577,10 @@ static int io_poll_wake(struct wait_queue_entry *wait, unsigned mode, int sync,
  	struct io_kiocb *req = wait->private;
  	struct io_poll_iocb *poll = &req->poll;
 
+	if (!READ_ONCE(poll->active))
+		return 0;
+
+	WRITE_ONCE(poll->active, false);
  	return __io_async_wake(req, poll, key_to_poll(key), io_poll_task_func);
  }
 
#3 lore raw

RFC 2/3 io_uring: don't get completion_lock in io_poll_rewait()

In reply to Xiaoguang Wang

In current implementation, if there are not available events, io_poll_rewait() just gets completion_lock, and unlocks it in io_poll_task_func() or io_async_task_func(), which isn't necessary.

Change this logic to let io_poll_task_func() or io_async_task_func() get the completion_lock lock, this is also a preparation for later patch, which will batch poll request completion.

---
 fs/io_uring.c | 50 ++++++++++++++++++++++----------------------------
 1 file changed, 22 insertions(+), 28 deletions(-)
fs/io_uring.c +22 -28
index ca4464a75c7b..6fdfb688cf91 100644
@@ -5040,10 +5040,7 @@ static int __io_async_wake(struct io_kiocb *req, struct io_poll_iocb *poll,
 }
 
 static bool io_poll_rewait(struct io_kiocb *req, struct io_poll_iocb *poll)
-	__acquires(&req->ctx->completion_lock)
 {
-	struct io_ring_ctx *ctx = req->ctx;
-
 	/* req->task == current here, checking PF_EXITING is safe */
 	if (unlikely(req->task->flags & PF_EXITING))
 		WRITE_ONCE(poll->canceled, true);
@@ -5054,7 +5051,6 @@ static bool io_poll_rewait(struct io_kiocb *req, struct io_poll_iocb *poll)
 		req->result = vfs_poll(req->file, &pt) & poll->events;
 	}
 
-	spin_lock(&ctx->completion_lock);
 	if (!req->result && !READ_ONCE(poll->canceled)) {
 		if (req->opcode == IORING_OP_POLL_ADD)
 			WRITE_ONCE(req->poll.active, true);
@@ -5142,30 +5138,29 @@ static void io_poll_task_func(struct io_kiocb *req, bool *locked)
 {
 	struct io_ring_ctx *ctx = req->ctx;
 	struct io_kiocb *nxt;
+	bool done;
 
-	if (io_poll_rewait(req, &req->poll)) {
-		spin_unlock(&ctx->completion_lock);
-	} else {
-		bool done;
+	if (io_poll_rewait(req, &req->poll))
+		return;
 
-		done = __io_poll_complete(req, req->result);
-		if (done) {
-			io_poll_remove_double(req);
-			__io_poll_remove_one(req, io_poll_get_single(req), true);
-			hash_del(&req->hash_node);
-		} else {
-			req->result = 0;
-			WRITE_ONCE(req->poll.active, true);
-		}
-		io_commit_cqring(ctx);
-		spin_unlock(&ctx->completion_lock);
-		io_cqring_ev_posted(ctx);
+	spin_lock(&ctx->completion_lock);
+	done = __io_poll_complete(req, req->result);
+	if (done) {
+		io_poll_remove_double(req);
+		__io_poll_remove_one(req, io_poll_get_single(req), true);
+		hash_del(&req->hash_node);
+	} else {
+		req->result = 0;
+		WRITE_ONCE(req->poll.active, true);
+	}
+	io_commit_cqring(ctx);
+	spin_unlock(&ctx->completion_lock);
+	io_cqring_ev_posted(ctx);
 
-		if (done) {
-			nxt = io_put_req_find_next(req);
-			if (nxt)
-				io_req_task_submit(nxt, locked);
-		}
+	if (done) {
+		nxt = io_put_req_find_next(req);
+		if (nxt)
+			io_req_task_submit(nxt, locked);
 	}
 }
 
@@ -5284,11 +5279,10 @@ static void io_async_task_func(struct io_kiocb *req, bool *locked)
 
 	trace_io_uring_task_run(req->ctx, req, req->opcode, req->user_data);
 
-	if (io_poll_rewait(req, &apoll->poll)) {
-		spin_unlock(&ctx->completion_lock);
+	if (io_poll_rewait(req, &apoll->poll))
 		return;
-	}
 
+	spin_lock(&ctx->completion_lock);
 	hash_del(&req->hash_node);
 	io_poll_remove_double(req);
 	spin_unlock(&ctx->completion_lock);
Signature · 1 line
2.14.4.44.g2045bb6
#4 lore raw

RFC 3/3 io_uring: try to batch poll request completion

In reply to Xiaoguang Wang

For an echo-server based on io_uring's IORING_POLL_ADD_MULTI feature, only poll request are completed in task work, normal read/write requests are issued when user app sees cqes on corresponding poll requests, and they will mostly read/write data successfully, which don't need task work. So at least for echo-server model, batching poll request completion properly will give benefits.

Currently don't find any appropriate place to store batched poll requests, put them in struct io_submit_state temporarily, which I think it'll need rework in future.

---
 fs/io_uring.c | 64 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 63 insertions(+), 1 deletion(-)
fs/io_uring.c +63 -1
index 6fdfb688cf91..14118388bfc6 100644
@@ -321,6 +321,11 @@ struct io_submit_state {
 	 */
 	struct io_kiocb		*compl_reqs[IO_COMPL_BATCH];
 	unsigned int		compl_nr;
+
+	struct io_kiocb		*poll_compl_reqs[IO_COMPL_BATCH];
+	bool			poll_req_status[IO_COMPL_BATCH];
+	unsigned int		poll_compl_nr;
+
 	/* inline/task_work completion list, under ->uring_lock */
 	struct list_head	free_list;
 
@@ -2093,6 +2098,8 @@ static void ctx_flush_and_put(struct io_ring_ctx *ctx, bool *locked)
 	percpu_ref_put(&ctx->refs);
 }
 
+static void io_poll_flush_completions(struct io_ring_ctx *ctx, bool *locked);
+
 static void tctx_task_work(struct callback_head *cb)
 {
 	bool locked = false;
@@ -2103,8 +2110,11 @@ static void tctx_task_work(struct callback_head *cb)
 	while (1) {
 		struct io_wq_work_node *node;
 
-		if (!tctx->task_list.first && locked && ctx->submit_state.compl_nr)
+		if (!tctx->task_list.first && locked && (ctx->submit_state.compl_nr ||
+		    ctx->submit_state.poll_compl_nr)) {
 			io_submit_flush_completions(ctx);
+			io_poll_flush_completions(ctx, &locked);
+		}
 
 		spin_lock_irq(&tctx->task_lock);
 		node = tctx->task_list.first;
@@ -5134,6 +5144,49 @@ static inline bool io_poll_complete(struct io_kiocb *req, __poll_t mask)
 static bool __io_poll_remove_one(struct io_kiocb *req,
 				 struct io_poll_iocb *poll, bool do_cancel);
 
+static void io_poll_flush_completions(struct io_ring_ctx *ctx, bool *locked)
+	__must_hold(&ctx->uring_lock)
+{
+	struct io_submit_state *state = &ctx->submit_state;
+	struct io_kiocb *req, *nxt;
+	int i, nr = state->poll_compl_nr;
+	bool done, skip_done = true;
+
+	spin_lock(&ctx->completion_lock);
+	for (i = 0; i < nr; i++) {
+		req = state->poll_compl_reqs[i];
+		done = __io_poll_complete(req, req->result);
+		if (done) {
+			io_poll_remove_double(req);
+			__io_poll_remove_one(req, io_poll_get_single(req), true);
+			hash_del(&req->hash_node);
+			state->poll_req_status[i] = true;
+			if (skip_done)
+				skip_done = false;
+		} else {
+			req->result = 0;
+			state->poll_req_status[i] = false;
+			WRITE_ONCE(req->poll.active, true);
+		}
+	}
+	io_commit_cqring(ctx);
+	spin_unlock(&ctx->completion_lock);
+	io_cqring_ev_posted(ctx);
+	state->poll_compl_nr = 0;
+
+	if (skip_done)
+		return;
+
+	for (i = 0; i < nr; i++) {
+		if (state->poll_req_status[i]) {
+			req = state->poll_compl_reqs[i];
+			nxt = io_put_req_find_next(req);
+			if (nxt)
+				io_req_task_submit(nxt, locked);
+		}
+	}
+}
+
 static void io_poll_task_func(struct io_kiocb *req, bool *locked)
 {
 	struct io_ring_ctx *ctx = req->ctx;
@@ -5143,6 +5196,15 @@ static void io_poll_task_func(struct io_kiocb *req, bool *locked)
 	if (io_poll_rewait(req, &req->poll))
 		return;
 
+	if (*locked) {
+		struct io_submit_state *state = &ctx->submit_state;
+
+		state->poll_compl_reqs[state->poll_compl_nr++] = req;
+		if (state->poll_compl_nr == ARRAY_SIZE(state->poll_compl_reqs))
+			io_poll_flush_completions(ctx, locked);
+		return;
+	}
+
 	spin_lock(&ctx->completion_lock);
 	done = __io_poll_complete(req, req->result);
 	if (done) {
Signature · 1 line
2.14.4.44.g2045bb6
#5 lore raw

Re: [RFC 3/3] io_uring: try to batch poll request completion

Quoted 42 lines · Xiaoguang Wang

For an echo-server based on io_uring's IORING_POLL_ADD_MULTI feature, only poll request are completed in task work, normal read/write requests are issued when user app sees cqes on corresponding poll requests, and they will mostly read/write data successfully, which don't need task work. So at least for echo-server model, batching poll request completion properly will give benefits.

Currently don't find any appropriate place to store batched poll requests, put them in struct io_submit_state temporarily, which I think it'll need rework in future.

---
 fs/io_uring.c | 64 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 63 insertions(+), 1 deletion(-)
fs/io_uring.c +7 -0
index 6fdfb688cf91..14118388bfc6 100644
@@ -321,6 +321,11 @@ struct io_submit_state {
 	 */
 	struct io_kiocb		*compl_reqs[IO_COMPL_BATCH];
 	unsigned int		compl_nr;
+
+	struct io_kiocb		*poll_compl_reqs[IO_COMPL_BATCH];
+	bool			poll_req_status[IO_COMPL_BATCH];
+	unsigned int		poll_compl_nr;
+
 	/* inline/task_work completion list, under ->uring_lock */
 	struct list_head	free_list;
 
@@ -2093,6 +2098,8 @@ static void ctx_flush_and_put(struct io_ring_ctx *ctx, bool *locked)
 	percpu_ref_put(&ctx->refs);
 }
 
+static void io_poll_flush_completions(struct io_ring_ctx *ctx, bool *locked);
+
 static void tctx_task_work(struct callback_head *cb)
 {
 	bool locked = false;

fs/io_uring.c static void tctx_task_work(struct callback_head *cb) · line 2114

Quoted 6 lines
fs/io_uring.c +2 -1
@@ -2103,8 +2110,11 @@ static void tctx_task_work(struct callback_head *cb)
 	while (1) {
 		struct io_wq_work_node *node;
 
-		if (!tctx->task_list.first && locked && ctx->submit_state.compl_nr)
+		if (!tctx->task_list.first && locked && (ctx->submit_state.compl_nr ||
+		    ctx->submit_state.poll_compl_nr)) {

io_submit_flush_completions() shouldn't be called if there are no requests... And the check is already inside for-next, will be

if (... && locked) {
	io_submit_flush_completions();
	if (poll_compl_nr)
		io_poll_flush_completions();
}
Quoted 12 lines
fs/io_uring.c +4 -0
 			io_submit_flush_completions(ctx);
+			io_poll_flush_completions(ctx, &locked);
+		}
 
 		spin_lock_irq(&tctx->task_lock);
 		node = tctx->task_list.first;
@@ -5134,6 +5144,49 @@ static inline bool io_poll_complete(struct io_kiocb *req, __poll_t mask)
 static bool __io_poll_remove_one(struct io_kiocb *req,
 				 struct io_poll_iocb *poll, bool do_cancel);
 
+static void io_poll_flush_completions(struct io_ring_ctx *ctx, bool *locked)
+	__must_hold(&ctx->uring_lock)

fs/io_uring.c static inline bool io_poll_complete(struct io_kiocb *req, __poll_t mask) · line 5158

Quoted 10 lines
fs/io_uring.c +10 -0
+{
+	struct io_submit_state *state = &ctx->submit_state;
+	struct io_kiocb *req, *nxt;
+	int i, nr = state->poll_compl_nr;
+	bool done, skip_done = true;
+
+	spin_lock(&ctx->completion_lock);
+	for (i = 0; i < nr; i++) {
+		req = state->poll_compl_reqs[i];
+		done = __io_poll_complete(req, req->result);

I believe we first need to fix all the poll problems and lay out something more intuitive than the current implementation, or it'd be pure hell to do afterwards.

Can be a nice addition, curious about numbers as well.

Signature · 1 line
Pavel Begunkov
#6 lore raw

Re: [RFC 3/3] io_uring: try to batch poll request completion

hi,

Quoted 52 lines · Xiaoguang Wang
Quoted 42 lines

For an echo-server based on io_uring's IORING_POLL_ADD_MULTI feature, only poll request are completed in task work, normal read/write requests are issued when user app sees cqes on corresponding poll requests, and they will mostly read/write data successfully, which don't need task work. So at least for echo-server model, batching poll request completion properly will give benefits.

Currently don't find any appropriate place to store batched poll requests, put them in struct io_submit_state temporarily, which I think it'll need rework in future.

---
  fs/io_uring.c | 64 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
  1 file changed, 63 insertions(+), 1 deletion(-)
fs/io_uring.c +7 -0
index 6fdfb688cf91..14118388bfc6 100644
@@ -321,6 +321,11 @@ struct io_submit_state {
  	 */
  	struct io_kiocb		*compl_reqs[IO_COMPL_BATCH];
  	unsigned int		compl_nr;
+
+	struct io_kiocb		*poll_compl_reqs[IO_COMPL_BATCH];
+	bool			poll_req_status[IO_COMPL_BATCH];
+	unsigned int		poll_compl_nr;
+
  	/* inline/task_work completion list, under ->uring_lock */
  	struct list_head	free_list;
 
@@ -2093,6 +2098,8 @@ static void ctx_flush_and_put(struct io_ring_ctx *ctx, bool *locked)
  	percpu_ref_put(&ctx->refs);
  }
 
+static void io_poll_flush_completions(struct io_ring_ctx *ctx, bool *locked);
+
  static void tctx_task_work(struct callback_head *cb)
  {
  	bool locked = false;

fs/io_uring.c static void tctx_task_work(struct callback_head *cb) · line 2114

Quoted 6 lines
fs/io_uring.c +2 -1
@@ -2103,8 +2110,11 @@ static void tctx_task_work(struct callback_head *cb)
  	while (1) {
  		struct io_wq_work_node *node;
 
-		if (!tctx->task_list.first && locked && ctx->submit_state.compl_nr)
+		if (!tctx->task_list.first && locked && (ctx->submit_state.compl_nr ||
+		    ctx->submit_state.poll_compl_nr)) {

io_submit_flush_completions() shouldn't be called if there are no requests... And the check is already inside for-next, will be

Quoted 4 lines
if (... && locked) {
	io_submit_flush_completions();
	if (poll_compl_nr)
		io_poll_flush_completions();

OK, thanks for pointing this, and I have dropped the poll request completion batching patch, since

it shows performance fluctuations, hard to say whether it's useful.

Regards,

Xiaoguang Wang

Quoted 30 lines

}

Quoted 12 lines
fs/io_uring.c +4 -0
  			io_submit_flush_completions(ctx);
+			io_poll_flush_completions(ctx, &locked);
+		}
 
  		spin_lock_irq(&tctx->task_lock);
  		node = tctx->task_list.first;
@@ -5134,6 +5144,49 @@ static inline bool io_poll_complete(struct io_kiocb *req, __poll_t mask)
  static bool __io_poll_remove_one(struct io_kiocb *req,
  				 struct io_poll_iocb *poll, bool do_cancel);
 
+static void io_poll_flush_completions(struct io_ring_ctx *ctx, bool *locked)
+	__must_hold(&ctx->uring_lock)

fs/io_uring.c static inline bool io_poll_complete(struct io_kiocb *req, __poll_t mask) · line 5158

Quoted 10 lines
fs/io_uring.c +10 -0
+{
+	struct io_submit_state *state = &ctx->submit_state;
+	struct io_kiocb *req, *nxt;
+	int i, nr = state->poll_compl_nr;
+	bool done, skip_done = true;
+
+	spin_lock(&ctx->completion_lock);
+	for (i = 0; i < nr; i++) {
+		req = state->poll_compl_reqs[i];
+		done = __io_poll_complete(req, req->result);

I believe we first need to fix all the poll problems and lay out something more intuitive than the current implementation, or it'd be pure hell to do afterwards.

Can be a nice addition, curious about numbers as well.

#7 lore raw

Re: [RFC 3/3] io_uring: try to batch poll request completion

In reply to Xiaoguang Wang
Quoted 7 lines · Xiaoguang Wang

For an echo-server based on io_uring's IORING_POLL_ADD_MULTI feature, only poll request are completed in task work, normal read/write requests are issued when user app sees cqes on corresponding poll requests, and they will mostly read/write data successfully, which don't need task work. So at least for echo-server model, batching poll request completion properly will give benefits.

Quoted 5 lines

Currently don't find any appropriate place to store batched poll requests, put them in struct io_submit_state temporarily, which I think it'll need rework in future.

We may need flush completion when io_submit_end as well, there may be situations where pure poll reqs are sparse. For instance, users may just use pure poll to do accept, and fast poll for read/write, send/recv, latency for pure poll reqs may amplify.

#8 lore raw

Re: [RFC 3/3] io_uring: try to batch poll request completion

Quoted 8 lines · Hao Xu
Quoted 7 lines · Xiaoguang Wang

For an echo-server based on io_uring's IORING_POLL_ADD_MULTI feature, only poll request are completed in task work, normal read/write requests are issued when user app sees cqes on corresponding poll requests, and they will mostly read/write data successfully, which don't need task work. So at least for echo-server model, batching poll request completion properly will give benefits.

Quoted 8 lines
Quoted 5 lines

Currently don't find any appropriate place to store batched poll requests, put them in struct io_submit_state temporarily, which I think it'll need rework in future.

We may need flush completion when io_submit_end as well, there may be situations where pure poll reqs are sparse. For instance, users may just use pure poll to do accept, and fast poll for read/write,

I mean in persistent connection model.

Quoted 1 line

send/recv, latency for pure poll reqs may amplify.

#9 lore raw

Re: [RFC 3/3] io_uring: try to batch poll request completion

Quoted 13 lines · Hao Xu
Quoted 12 lines · Hao Xu
Quoted 11 lines · Xiaoguang Wang

For an echo-server based on io_uring's IORING_POLL_ADD_MULTI feature, only poll request are completed in task work, normal read/write requests are issued when user app sees cqes on corresponding poll requests, and they will mostly read/write data successfully, which don't need task work. So at least for echo-server model, batching poll request completion properly will give benefits.

Currently don't find any appropriate place to store batched poll requests, put them in struct io_submit_state temporarily, which I think it'll need rework in future.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

We may need flush completion when io_submit_end as well, there may be situations where pure poll reqs are sparse. For instance, users may just use pure poll to do accept, and fast poll for read/write,

I mean in persistent connection model.

Quoted 1 line

send/recv, latency for pure poll reqs may amplify.

I actually think flush them in io_subimit_state_end doesn't work as well since it is multishot req..

maintainer
#10 lore raw
In reply to Xiaoguang Wang
Quoted 2 lines · Xiaoguang Wang

This patchset tries to improve echo_server model based on io_uring's IORING_POLL_ADD_MULTI feature.

Nifty, I'll take a look. Can you put the echo server using multishot somewhere so others can test it too? Maybe it already is, but would be nice to detail exactly what was run.

Signature · 1 line
Jens Axboe