io_uring: only wake up sq thread while current task is in io worker context
PATCH 1/1 io_uring: only wake up sq thread while current task is in io worker context
When IORING_SETUP_SQPOLL is enabled, io_uring will always handle sqes in sq thread task context, so in io_iopoll_req_issued(), if we're not in io worker context, we don't need to check whether should wake up sq thread. io_iopoll_req_issued() calls wq_has_sleeper(), which has smp_mb() memory barrier, perf shows obvious overhead:
Samples: 481K of event 'cycles', Event count (approx.): 299807382878
Overhead Comma Shared Object Symbol
3.69% :9630 [kernel.vmlinux] [k] io_issue_sqeWith this patch, perf shows:
Samples: 482K of event 'cycles', Event count (approx.): 299929547283
Overhead Comma Shared Object Symbol
0.70% :4015 [kernel.vmlinux] [k] io_issue_sqeIt shows some obvious improvements.
--- fs/io_uring.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
fs/io_uring.c +3 -3
@@ -2732,7 +2732,7 @@ static void io_complete_rw_iopoll(struct kiocb *kiocb, long res, long res2)* find it from a io_iopoll_getevents() thread before the issuer is done * accessing the kiocb cookie. */ -static void io_iopoll_req_issued (struct io_kiocb *req) +static void io_iopoll_req_issued (struct io_kiocb *req,bool in_async) {struct io_ring_ctx *ctx = req->ctx;
@@ -2761,7 +2761,7 @@ static void io_iopoll_req_issued(struct io_kiocb *req)else list_add_tail (&req->inflight_entry, &ctx->iopoll_list); -if ((ctx->flags &IORING_SETUP_SQPOLL ) && +if (in_async && (ctx->flags &IORING_SETUP_SQPOLL ) &&wq_has_sleeper (&ctx->sq_data->wait))wake_up (&ctx->sq_data->wait); }
@@ -6245,7 +6245,7 @@ static int io_issue_sqe(struct io_kiocb *req, bool force_nonblock,if (in_async)mutex_lock (&ctx->uring_lock); -io_iopoll_req_issued (req); +io_iopoll_req_issued (req, in_async);if (in_async)mutex_unlock (&ctx->uring_lock);
Signature · 1 line
2.17.2
Jens Axboe
maintainerQuoted 9 lines · Xiaoguang Wang
When
IORING_SETUP_SQPOLLis enabled,io_uringwill always handle sqes in sq thread task context, so inio_iopoll_req_issued(), if we're not in io worker context, we don't need to check whether should wake up sq thread.io_iopoll_req_issued()callswq_has_sleeper(), which hassmp_mb()memory barrier, perf shows obvious overhead:
Samples: 481K of event 'cycles', Event count (approx.): 299807382878Overhead Comma Shared Object Symbol 3.69% :9630 [kernel.vmlinux] [k] io_issue_sqe
Quoted 6 lines
With this patch, perf shows:
Samples: 482K of event 'cycles', Event count (approx.): 299929547283Overhead Comma Shared Object Symbol 0.70% :4015 [kernel.vmlinux] [k] io_issue_sqeIt shows some obvious improvements.
Looks good to me, but:
fs/io_uring.c static void io_iopoll_req_issued(struct io_kiocb *req) · line 2767
Quoted 9 lines
fs/io_uring.c +1
-1@@ -2761,7 +2761,7 @@ static void io_iopoll_req_issued(struct io_kiocb *req)else list_add_tail (&req->inflight_entry, &ctx->iopoll_list); -if ((ctx->flags &IORING_SETUP_SQPOLL ) && +if (in_async && (ctx->flags &IORING_SETUP_SQPOLL ) &&wq_has_sleeper (&ctx->sq_data->wait))wake_up (&ctx->sq_data->wait); }
This really needs a comment as to why we don't have to check and wake from this path.
Signature · 1 line
Jens Axboe