io_uring: io_uring_enter(2) don't poll while SETUP_IOPOLL|SETUP_SQPOLL enabled
PATCH 1/1 io_uring: io_uring_enter(2) don't poll while SETUP_IOPOLL|SETUP_SQPOLL enabled
When SETUP_IOPOLL and SETUP_SQPOLL are both enabled, applications don't need to do io completion events polling again, they can rely on io_sq_thread to do polling work, which can reduce cpu usage and uring_lock contention.
I modify fio io_uring engine codes a bit to evaluate the performance:
static int fio_ioring_getevents(struct thread_data *td, unsigned int min,
continue;
}- if (!o->sqpoll_thread) {
+ if (o->sqpoll_thread && o->hipri) {
r = io_uring_enter(ld, 0, actual_min,
IORING_ENTER_GETEVENTS);
if (r < 0) {and use "fio -name=fiotest -filename=/dev/nvme0n1 -iodepth=$depth -thread -rw=read -ioengine=io_uring -hipri=1 -sqthread_poll=1 -direct=1 -bs=4k -size=10G -numjobs=1 -time_based -runtime=120"
original codes
--------------------------------------------------------------------
iodepth | 4 | 8 | 16 | 32 | 64 bw | 1133MB/s | 1519MB/s | 2090MB/s | 2710MB/s | 3012MB/s fio cpu usage | 100% | 100% | 100% | 100% | 100%
--------------------------------------------------------------------
with patch
--------------------------------------------------------------------
iodepth | 4 | 8 | 16 | 32 | 64 bw | 1196MB/s | 1721MB/s | 2351MB/s | 2977MB/s | 3357MB/s fio cpu usage | 63.8% | 74.4%% | 81.1% | 83.7% | 82.4%
--------------------------------------------------------------------
bw improve | 5.5% | 13.2% | 12.3% | 9.8% | 11.5%
--------------------------------------------------------------------
From above test results, we can see that bw has above 5.5%~13% improvement, and fio process's cpu usage also drops much. Note this won't improve io_sq_thread's cpu usage when SETUP_IOPOLL|SETUP_SQPOLL are both enabled, in this case, io_sq_thread always has 100% cpu usage. I think this patch will be friendly to applications which will often use io_uring_wait_cqe() or similar from liburing.
--- fs/io_uring.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
fs/io_uring.c +10 -1
@@ -1574,6 +1574,8 @@ static void io_iopoll_complete(struct io_ring_ctx *ctx, unsigned int *nr_events, }io_commit_cqring (ctx); +if (ctx->flags &IORING_SETUP_SQPOLL ) +io_cqring_ev_posted (ctx);io_free_req_many (ctx, &rb); }
@@ -6637,7 +6639,14 @@ static unsigned long io_uring_nommu_get_unmapped_area(struct file *file, min_complete =min (min_complete, ctx->cq_entries); -if (ctx->flags &IORING_SETUP_IOPOLL ) { +/* +* When SETUP_IOPOLL and SETUP_SQPOLL are both enabled, user +* space applications don't need to do io completion events +* polling again, they can rely on io_sq_thread to do polling +* work, which can reduce cpu usage and uring_lock contention. +*/ +if (ctx->flags &IORING_SETUP_IOPOLL && + !(ctx->flags &IORING_SETUP_SQPOLL )) { ret =io_iopoll_check (ctx, &nr_events, min_complete); }else { ret =io_cqring_wait (ctx, min_complete, sig, sigsz);
Signature · 1 line
1.8.3.1
Jens Axboe
maintainerQuoted 35 lines · Xiaoguang Wang
When
SETUP_IOPOLLandSETUP_SQPOLLare both enabled, applications don't need to do io completion events polling again, they can rely onio_sq_threadto do polling work, which can reduce cpu usage anduring_lockcontention.I modify fio
io_uringengine codes a bit to evaluate the performance:
static int fio_ioring_getevents(structthread_data*td, unsigned int min,continue; }(quoted diff) +1
-1- if (!o->sqpoll_thread) { + if (o->sqpoll_thread && o->hipri) { r = io_uring_enter(ld, 0, actual_min, IORING_ENTER_GETEVENTS); if (r < 0) {and use "fio -name=fiotest -filename=/dev/nvme0n1 -iodepth=$depth -thread
(quoted diff) +0
-2-rw=read -ioengine=io_uring -hipri=1 -sqthread_poll=1 -direct=1 -bs=4k -size=10G -numjobs=1 -time_based -runtime=120"original codes
--------------------------------------------------------------------iodepth | 4 | 8 | 16 | 32 | 64 bw | 1133MB/s | 1519MB/s | 2090MB/s | 2710MB/s | 3012MB/s fio cpu usage | 100% | 100% | 100% | 100% | 100%--------------------------------------------------------------------
with patch
--------------------------------------------------------------------iodepth | 4 | 8 | 16 | 32 | 64 bw | 1196MB/s | 1721MB/s | 2351MB/s | 2977MB/s | 3357MB/s fio cpu usage | 63.8% | 74.4%% | 81.1% | 83.7% | 82.4%--------------------------------------------------------------------
bw improve | 5.5% | 13.2% | 12.3% | 9.8% | 11.5%
--------------------------------------------------------------------
Quoted 6 lines
From above test results, we can see that bw has above 5.5%~13% improvement, and fio process's cpu usage also drops much. Note this won't improve io_sq_thread's cpu usage when SETUP_IOPOLL|SETUP_SQPOLL are both enabled, in this case,
io_sq_threadalways has 100% cpu usage. I think this patch will be friendly to applications which will often useio_uring_wait_cqe()or similar from liburing.
I think this looks reasonable, and true to the spirit of how polling should work when SQPOLL is used. I'll apply this for 5.7, thanks.
Signature · 1 line
Jens Axboe