Skip to contentSkip to conversation

io_uring: only wake up sq thread while current task is in io worker context

PATCH Superseded Xiaoguang Wang 2 participants 2 messages Nov 12, 2020

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 io_uring: only wake up sq thread while current task is in io worker context

When IORING_SETUP_SQPOLL is enabled, io_uring will always handle sqes in sq thread task context, so in io_iopoll_req_issued(), if we're not in io worker context, we don't need to check whether should wake up sq thread. io_iopoll_req_issued() calls wq_has_sleeper(), which has smp_mb() memory barrier, perf shows obvious overhead:
Samples: 481K of event 'cycles', Event count (approx.): 299807382878

  Overhead  Comma  Shared Object     Symbol
     3.69%  :9630  [kernel.vmlinux]  [k] io_issue_sqe

With this patch, perf shows:
Samples: 482K of event 'cycles', Event count (approx.): 299929547283

  Overhead  Comma  Shared Object     Symbol
     0.70%  :4015  [kernel.vmlinux]  [k] io_issue_sqe

It shows some obvious improvements.

---
 fs/io_uring.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
fs/io_uring.c +3 -3
index f594c72de777..c7e035433c59 100644
@@ -2732,7 +2732,7 @@ static void io_complete_rw_iopoll(struct kiocb *kiocb, long res, long res2)
  * find it from a io_iopoll_getevents() thread before the issuer is done
  * accessing the kiocb cookie.
  */
-static void io_iopoll_req_issued(struct io_kiocb *req)
+static void io_iopoll_req_issued(struct io_kiocb *req, bool in_async)
 {
 	struct io_ring_ctx *ctx = req->ctx;
 
@@ -2761,7 +2761,7 @@ static void io_iopoll_req_issued(struct io_kiocb *req)
 	else
 		list_add_tail(&req->inflight_entry, &ctx->iopoll_list);
 
-	if ((ctx->flags & IORING_SETUP_SQPOLL) &&
+	if (in_async && (ctx->flags & IORING_SETUP_SQPOLL) &&
 	    wq_has_sleeper(&ctx->sq_data->wait))
 		wake_up(&ctx->sq_data->wait);
 }
@@ -6245,7 +6245,7 @@ static int io_issue_sqe(struct io_kiocb *req, bool force_nonblock,
 		if (in_async)
 			mutex_lock(&ctx->uring_lock);
 
-		io_iopoll_req_issued(req);
+		io_iopoll_req_issued(req, in_async);
 
 		if (in_async)
 			mutex_unlock(&ctx->uring_lock);
Signature · 1 line
2.17.2
maintainer
#1 lore raw
Quoted 9 lines · Xiaoguang Wang

When IORING_SETUP_SQPOLL is enabled, io_uring will always handle sqes in sq thread task context, so in io_iopoll_req_issued(), if we're not in io worker context, we don't need to check whether should wake up sq thread. io_iopoll_req_issued() calls wq_has_sleeper(), which has smp_mb() memory barrier, perf shows obvious overhead:
Samples: 481K of event 'cycles', Event count (approx.): 299807382878

  Overhead  Comma  Shared Object     Symbol
     3.69%  :9630  [kernel.vmlinux]  [k] io_issue_sqe
Quoted 6 lines

With this patch, perf shows:
Samples: 482K of event 'cycles', Event count (approx.): 299929547283

  Overhead  Comma  Shared Object     Symbol
     0.70%  :4015  [kernel.vmlinux]  [k] io_issue_sqe

It shows some obvious improvements.

Looks good to me, but:

fs/io_uring.c static void io_iopoll_req_issued(struct io_kiocb *req) · line 2767

Quoted 9 lines
fs/io_uring.c +1 -1
@@ -2761,7 +2761,7 @@ static void io_iopoll_req_issued(struct io_kiocb *req)
 	else
 		list_add_tail(&req->inflight_entry, &ctx->iopoll_list);
 
-	if ((ctx->flags & IORING_SETUP_SQPOLL) &&
+	if (in_async && (ctx->flags & IORING_SETUP_SQPOLL) &&
 	    wq_has_sleeper(&ctx->sq_data->wait))
 		wake_up(&ctx->sq_data->wait);
 }

This really needs a comment as to why we don't have to check and wake from this path.

Signature · 1 line
Jens Axboe