io_uring: Use io_schedule* in cqring wait
PATCH 1/1 io_uring: Use io_schedule* in cqring wait
I observed poor performance of io_uring compared to synchronous IO. That turns out to be caused by deeper CPU idle states entered with io_uring, due to io_uring using plain schedule(), whereas synchronous IO uses io_schedule().
The losses due to this are substantial. On my cascade lake workstation, t/io_uring from the fio repository e.g. yields regressions between 20% and 40% with the following command:
./t/io_uring -r 5 -X0 -d 1 -s 1 -c 1 -p 0 -S$use_sync -R 0 /mnt/t2/fio/write.0.0
This is repeatable with different filesystems, using raw block devices and using different block devices.
Use io_schedule_prepare() / io_schedule_finish() in io_cqring_wait_schedule() to address the difference.
After that using io_uring is on par or surpassing synchronous IO (using registered files etc makes it reliably win, but arguably is a less fair comparison).
There are other calls to schedule() in io_uring/, but none immediately jump out to be similarly situated, so I did not touch them. Similarly, it's possible that mutex_lock_io() should be used, but it's not clear if there are cases where that matters.
--- io_uring/io_uring.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
io_uring/io_uring.c +14 -2
@@ -2575,6 +2575,9 @@ int io_run_task_work_sig(struct io_ring_ctx *ctx)static inline int io_cqring_wait_schedule (struct io_ring_ctx *ctx,struct io_wait_queue *iowq) { +int ret; +int token; +if (unlikely (READ_ONCE (ctx->check_cq)))return 1 ;if (unlikely (!llist_empty (&ctx->work_llist)))
@@ -2585,11 +2588,20 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx,return -EINTR ;if (unlikely (io_should_wake (iowq)))return 0 ; + +/* +* Use io_schedule_prepare/finish, so cpufreq can take into account +* that the task is waiting for IO - turns out to be important for low +* QD IO. +*/ + token =io_schedule_prepare (); + ret =0 ;if (iowq->timeout ==KTIME_MAX )schedule ();else if (!schedule_hrtimeout (&iowq->timeout,HRTIMER_MODE_ABS )) -return -ETIME ; -return 0 ; + ret = -ETIME ; +io_schedule_finish (token); +return ret; }/*
Signature · 1 line
2.38.0
Jens Axboe
maintainerQuoted 16 lines · Andres Freund
I observed poor performance of
io_uringcompared to synchronous IO. That turns out to be caused by deeper CPU idle states entered withio_uring, due toio_uringusing plainschedule(), whereas synchronous IO usesio_schedule().The losses due to this are substantial. On my cascade lake workstation, t/io_uring from the fio repository e.g. yields regressions between 20% and 40% with the following command:
./t/io_uring -r 5 -X0 -d 1 -s 1 -c 1 -p 0 -S$use_sync -R 0/mnt/t2/fio/write.0.0This is repeatable with different filesystems, using raw block devices and using different block devices.
Use
io_schedule_prepare()/io_schedule_finish()inio_cqring_wait_schedule()to address the difference.
Quoted 8 lines
After that using
io_uringis on par or surpassing synchronous IO (using registered files etc makes it reliably win, but arguably is a less fair comparison).There are other calls to
schedule()inio_uring/, but none immediately jump out to be similarly situated, so I did not touch them. Similarly, it's possible thatmutex_lock_io()should be used, but it's not clear if there are cases where that matters.
This looks good to me, and I also separately tested a similar patch and it showed good results for me even with a heavily performance oriented setup:
pread2 io_uring io_uring w/io_sched
QD1 185K 170K 186K
QD2 NA 304K 327K
QD4 NA 630K 640K
QD8 NA 891K 892K
I'll add this, with just one small minor cosmetic edit:
io_uring/io_uring.c int io_run_task_work_sig(struct io_ring_ctx *ctx) · line 2579
Quoted 6 lines
io_uring/io_uring.c +2
-0@@ -2575,6 +2575,9 @@ int io_run_task_work_sig(struct io_ring_ctx *ctx)static inline int io_cqring_wait_schedule (struct io_ring_ctx *ctx,struct io_wait_queue *iowq) { +int ret; +int token;
Should just be a single line.
And I'll mark this for stable as well. Thanks!
Signature · 1 line
Jens Axboe
Quoted 11 lines · Andres Freund
I observed poor performance of
io_uringcompared to synchronous IO. That turns out to be caused by deeper CPU idle states entered withio_uring, due toio_uringusing plainschedule(), whereas synchronous IO usesio_schedule().The losses due to this are substantial. On my cascade lake workstation, t/io_uring from the fio repository e.g. yields regressions between 20% and 40% with the following command:
./t/io_uring -r 5 -X0 -d 1 -s 1 -c 1 -p 0 -S$use_sync -R 0/mnt/t2/fio/write.0.0[...]
Applied, thanks!
[1/1] io_uring: Use io_schedule* in cqring wait
(no commit info)
Best regards,
Signature · 1 line
Jens Axboe