Improve async iomap DIO performance
Subsystems (4)
Jens Axboe
maintainerv2 cover Improve async iomap DIO performance
Hi,
iomap always punts async dio write completions to a workqueue, which has a cost in terms of efficiency (now you need an unrelated worker to process it) and latency (now you're bouncing a completion through an async worker, which is a classic slowdown scenario).
This patchset intends to improve that situation. For polled IO, we always have a task reaping completions. Those do, by definition, not need to be punted through a workqueue. This is patch 1, and it adds an IOMAP_DIO_INLINE_COMP flag that tells the completion side that we can handle this dio completion without punting to a workqueue, if we're called from the appropriate (task) context. This is good for up to an 11% improvement in my testing. Details in that patch commit message.
For IRQ driven IO, it's a bit more tricky. The iomap dio completion will happen in hard/soft irq context, and we need a saner context to process these completions. IOCB_DIO_DEFER is added, which can be set in a struct kiocb->ki_flags by the issuer. If the completion side of the iocb handling understands this flag, it can choose to set a kiocb->dio_complete() handler and just call ki_complete from IRQ context. The issuer must then ensure that this callback is processed from a task. io_uring punts IRQ completions to task_work already, so it's trivial wire it up to run more of the completion before posting a CQE. Patches 2 and 3 add the necessary flag and io_uring support, and patches 4 and 5 add iomap support for it. This is good for up to a 37% improvement in throughput/latency for low queue depth IO, patch 5 has the details.
This work came about when Andres tested low queue depth dio writes for postgres and compared it to doing sync dio writes, showing that the async processing slows us down a lot.
fs/iomap/direct-io.c | 44 +++++++++++++++++++++++++++++++++++++------- include/linux/fs.h | 30 ++++++++++++++++++++++++++++-- io_uring/rw.c | 24 ++++++++++++++++++++---- 3 files changed, 85 insertions(+), 13 deletions(-)
Can also be found in a git branch here:
https://git.kernel.dk/cgit/linux/log/?h=xfs-async-dio.2
Changelog:
- Rewrite patch 1 to add an explicit flag to manage when dio completions can be done inline. This drops any write related checks. We set this flag by default for both reads and writes, and clear it for the latter if we need zero out or O_DSYNC handling.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH 1/1 io_uring: Use io_schedule* in cqring wait
From: Andres Freund <[email protected]>
I observed poor performance of io_uring compared to synchronous IO. That turns out to be caused by deeper CPU idle states entered with io_uring, due to io_uring using plain schedule(), whereas synchronous IO uses io_schedule().
The losses due to this are substantial. On my cascade lake workstation, t/io_uring from the fio repository e.g. yields regressions between 20% and 40% with the following command:
./t/io_uring -r 5 -X0 -d 1 -s 1 -c 1 -p 0 -S$use_sync -R 0 /mnt/t2/fio/write.0.0
This is repeatable with different filesystems, using raw block devices and using different block devices.
Use io_schedule_prepare() / io_schedule_finish() in io_cqring_wait_schedule() to address the difference.
After that using io_uring is on par or surpassing synchronous IO (using registered files etc makes it reliably win, but arguably is a less fair comparison).
There are other calls to schedule() in io_uring/, but none immediately jump out to be similarly situated, so I did not touch them. Similarly, it's possible that mutex_lock_io() should be used, but it's not clear if there are cases where that matters.
--- io_uring/io_uring.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-)
io_uring/io_uring.c +13 -2
@@ -2489,6 +2489,8 @@ int io_run_task_work_sig(struct io_ring_ctx *ctx)static inline int io_cqring_wait_schedule (struct io_ring_ctx *ctx,struct io_wait_queue *iowq) { +int token, ret; +if (unlikely (READ_ONCE (ctx->check_cq)))return 1 ;if (unlikely (!llist_empty (&ctx->work_llist)))
@@ -2499,11 +2501,20 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx,return -EINTR ;if (unlikely (io_should_wake (iowq)))return 0 ; + +/* +* Use io_schedule_prepare/finish, so cpufreq can take into account +* that the task is waiting for IO - turns out to be important for low +* QD IO. +*/ + token =io_schedule_prepare (); + ret =0 ;if (iowq->timeout ==KTIME_MAX )schedule ();else if (!schedule_hrtimeout (&iowq->timeout,HRTIMER_MODE_ABS )) -return -ETIME ; -return 0 ; + ret = -ETIME ; +io_schedule_finish (token); +return ret; }/*
Signature · 1 line
2.40.1
Jens Axboe
maintainerRe: [PATCH] io_uring: Use io_schedule* in cqring wait
Quoted 1 line · Jens Axboe
From: Andres Freund <[email protected]>
Ignore this one, leftover patch in the patch directory when sending out this one...
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH v2 1/5 iomap: simplify logic for when a dio can get completed inline
In reply to Jens AxboeCurrently iomap gates this on !IOMAP_DIO_WRITE, but this isn't entirely accurate. Some writes can complete just fine inline. One such example is polled IO, where the completion always happens in task context.
Add IOMAP_DIO_INLINE_COMP which tells the completion side if we can complete this dio inline, or if it needs punting to a workqueue. We set this flag by default for any dio, and turn it off for unwritten extents or blocks that require a sync at completion time.
Gate the inline completion on whether we're in a task or not as well. This will always be true for polled IO, but for IRQ driven IO, the completion context may not allow for inline completions.
Testing a basic QD 1..8 dio random write with polled IO with the following fio job:
fio --name=polled-dio-write --filename=/data1/file --time_based=1 \
--runtime=10 --bs=4096 --rw=randwrite --norandommap --buffered=0 \
--cpus_allowed=4 --ioengine=io_uring --iodepth=$depth --hipri=1
yields:
Stock Patched Diff
=======================================
QD1 180K 201K +11% QD2 356K 394K +10% QD4 608K 650K +7% QD8 827K 831K +0.5%
which shows a nice win, particularly for lower queue depth writes. This is expected, as higher queue depths will be busy polling completions while the offloaded workqueue completions can happen in parallel.
--- fs/iomap/direct-io.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
fs/iomap/direct-io.c +9 -5
@@ -20,6 +20,7 @@* Private flags for iomap_dio, must not overlap with the public ones in * iomap.h: */ +#define IOMAP_DIO_INLINE_COMP (1 << 27)#define IOMAP_DIO_WRITE_FUA (1 << 28)#define IOMAP_DIO_NEED_SYNC (1 << 29)#define IOMAP_DIO_WRITE (1 << 30)
@@ -161,15 +162,15 @@ void iomap_dio_bio_end_io(struct bio *bio)struct task_struct *waiter = dio->submit.waiter;WRITE_ONCE (dio->submit.waiter,NULL );blk_wake_io_task (waiter); - }else if (dio->flags &IOMAP_DIO_WRITE ) { + }else if ((dio->flags &IOMAP_DIO_INLINE_COMP ) &&in_task ()) { +WRITE_ONCE (dio->iocb->private,NULL ); +iomap_dio_complete_work (&dio->aio.work); + }else {struct inode *inode =file_inode (dio->iocb->ki_filp);WRITE_ONCE (dio->iocb->private,NULL );INIT_WORK (&dio->aio.work, iomap_dio_complete_work);queue_work (inode->i_sb->s_dio_done_wq, &dio->aio.work); - }else { -WRITE_ONCE (dio->iocb->private,NULL ); -iomap_dio_complete_work (&dio->aio.work); } }
@@ -244,6 +245,7 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter,if (iomap->type ==IOMAP_UNWRITTEN ) { dio->flags |=IOMAP_DIO_UNWRITTEN ; + dio->flags &= ~IOMAP_DIO_INLINE_COMP ; need_zeroout = true; }
@@ -500,7 +502,8 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, dio->i_size =i_size_read (inode); dio->dops = dops; dio->error =0 ; - dio->flags =0 ; +/* default to inline completion, turned off when not supported */ + dio->flags =IOMAP_DIO_INLINE_COMP ; dio->done_before = done_before; dio->submit.iter = iter;
@@ -535,6 +538,7 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,/* for data sync or sync, we need sync completion processing */ if (iocb_is_dsync (iocb)) { dio->flags |=IOMAP_DIO_NEED_SYNC ; + dio->flags &= ~IOMAP_DIO_INLINE_COMP ;/* * For datasync only writes, we optimistically try
Signature · 1 line
2.40.1
Re: [PATCH 1/5] iomap: simplify logic for when a dio can get completed inline
Quoted 8 lines · Jens Axboe
Currently iomap gates this on !IOMAP_DIO_WRITE, but this isn't entirely accurate. Some writes can complete just fine inline. One such example is polled IO, where the completion always happens in task context.
Add
IOMAP_DIO_INLINE_COMPwhich tells the completion side if we can complete this dio inline, or if it needs punting to a workqueue. We set this flag by default for any dio, and turn it off for unwritten extents or blocks that require a sync at completion time.
Ignoring the O_DSYNC case (I'll get to that at the end), this is still wrong - it misses extending writes that need to change file size at IO completion. For some filesystems, file extension at IO completion has the same constraints as unwritten extent conversion (i.e. requires locking and transactions), but the iomap infrastructure has no idea whether the filesystem performing the IO requires this or not.
i.e. if iomap always punts unwritten extent IO to a workqueue, we also have to punt extending writes to a workqueue. Fundamentally, the iomap code itself cannot make a correct determination of whether IO completion of any specific write IO requires completion in task context.
Only the filesystem knows that,
However, the filesystem knows if the IO is going to need IO completion processing at submission time. It tells iomap that it needs completion processing via the IOMAP_F_DIRTY flag. This allows filesystems to determine what IOs iomap can consider as "writes that don't need filesystem completion processing".
With this flag, iomap can optimise the IO appropriately. We can use REQ_FUA for O_DSYNC writes if IOMAP_F_DIRTY is not set. We can do inline completion if IOMAP_F_DIRTY is not set. But if IOMAP_F_DIRTY is set, the filesystem needs to run it's own completion processing, and so iomap cannot run that write with an inline completion.
Quoted 3 lines
Gate the inline completion on whether we're in a task or not as well. This will always be true for polled IO, but for IRQ driven IO, the completion context may not allow for inline completions.
Again, context does not matter for pure overwrites - we can complete them inline regardless of completion context. The task context only matters when the filesystem needs to do completion work, and we've already established that we are not doing inline completion for polled IO for unwritten, O_DSYNC or extending file writes.
IOWs, we already avoid polled completions for all the situations where IOMAP_F_DIRTY is set by the filesystem to indicate the operation is not a pure overwrite....
Quoted 17 lines
--- fs/iomap/direct-io.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)fs/iomap/direct-io.c +1
-0@@ -20,6 +20,7 @@* Private flags for iomap_dio, must not overlap with the public ones in * iomap.h: */ +#define IOMAP_DIO_INLINE_COMP (1 << 27)#define IOMAP_DIO_WRITE_FUA (1 << 28)#define IOMAP_DIO_NEED_SYNC (1 << 29)#define IOMAP_DIO_WRITE (1 << 30)
fs/iomap/direct-io.c void iomap_dio_bio_end_io(struct bio *bio) · line 165
Quoted 5 lines
fs/iomap/direct-io.c +1
-1@@ -161,15 +162,15 @@ void iomap_dio_bio_end_io(struct bio *bio)struct task_struct *waiter = dio->submit.waiter;WRITE_ONCE (dio->submit.waiter,NULL );blk_wake_io_task (waiter); - }else if (dio->flags &IOMAP_DIO_WRITE ) { + }else if ((dio->flags &IOMAP_DIO_INLINE_COMP ) &&in_task ()) {
Regardless of whether the code is correct or not, this needs a comment explaining what problem the in_task() check is working around...
Quoted 23 lines
fs/iomap/direct-io.c +4
-3+WRITE_ONCE (dio->iocb->private,NULL ); +iomap_dio_complete_work (&dio->aio.work); + }else {struct inode *inode =file_inode (dio->iocb->ki_filp);WRITE_ONCE (dio->iocb->private,NULL );INIT_WORK (&dio->aio.work, iomap_dio_complete_work);queue_work (inode->i_sb->s_dio_done_wq, &dio->aio.work); - }else { -WRITE_ONCE (dio->iocb->private,NULL ); -iomap_dio_complete_work (&dio->aio.work); } }@@ -244,6 +245,7 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter,if (iomap->type ==IOMAP_UNWRITTEN ) { dio->flags |=IOMAP_DIO_UNWRITTEN ; + dio->flags &= ~IOMAP_DIO_INLINE_COMP ; need_zeroout = true; }
fs/iomap/direct-io.c __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, · line 507
Quoted 7 lines
fs/iomap/direct-io.c +2
-1@@ -500,7 +502,8 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, dio->i_size =i_size_read (inode); dio->dops = dops; dio->error =0 ; - dio->flags =0 ; +/* default to inline completion, turned off when not supported */ + dio->flags =IOMAP_DIO_INLINE_COMP ; dio->done_before = done_before;
I think this is poorly coded. If we get the clearing logic wrong (as is the case in this patch) then bad things will happen when we run inline completion in an irq context when the filesystem needs to run a transaction. e.g. file extension.
It looks to me like you hacked around this "default is wrong" case with the "in_task()" check in completion, but given that check is completely undocumented....
fs/iomap/direct-io.c __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, · line 541
Quoted 6 lines
fs/iomap/direct-io.c +1
-0dio->submit.iter = iter;@@ -535,6 +538,7 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,/* for data sync or sync, we need sync completion processing */ if (iocb_is_dsync (iocb)) { dio->flags |=IOMAP_DIO_NEED_SYNC ; + dio->flags &= ~IOMAP_DIO_INLINE_COMP ;
This is looks wrong, too. We set IOMAP_DIO_WRITE_FUA ca couple of lines later, and during bio submission we check if REQ_FUA can be used if IOMAP_F_DIRTY is not set. If all the bios we submit use REQ_FUA, then we clear IOMAP_DIO_NEED_SYNC before we drop the dio submission reference.
For such a REQ_FUA bio chains, we can now safely do inline completion because we don't run generic_write_sync() in IO completion now. The filesystem does not need to perform blocking or IO operations in completion, either, so these IOs can be completed in line like any other pure overwrite DIO....
-Dave.
Signature · 2 lines
Dave Chinner [email protected]
Jens Axboe
maintainerRe: [PATCH 1/5] iomap: simplify logic for when a dio can get completed inline
Quoted 32 lines · Dave Chinner
Quoted 8 lines · Jens Axboe
Currently iomap gates this on !IOMAP_DIO_WRITE, but this isn't entirely accurate. Some writes can complete just fine inline. One such example is polled IO, where the completion always happens in task context.
Add
IOMAP_DIO_INLINE_COMPwhich tells the completion side if we can complete this dio inline, or if it needs punting to a workqueue. We set this flag by default for any dio, and turn it off for unwritten extents or blocks that require a sync at completion time.Ignoring the
O_DSYNCcase (I'll get to that at the end), this is still wrong - it misses extending writes that need to change file size at IO completion. For some filesystems, file extension at IO completion has the same constraints as unwritten extent conversion (i.e. requires locking and transactions), but the iomap infrastructure has no idea whether the filesystem performing the IO requires this or not.i.e. if iomap always punts unwritten extent IO to a workqueue, we also have to punt extending writes to a workqueue. Fundamentally, the iomap code itself cannot make a correct determination of whether IO completion of any specific write IO requires completion in task context.
Only the filesystem knows that,
However, the filesystem knows if the IO is going to need IO completion processing at submission time. It tells iomap that it needs completion processing via the
IOMAP_F_DIRTYflag. This allows filesystems to determine what IOs iomap can consider as "writes that don't need filesystem completion processing".
Quoted 5 lines
With this flag, iomap can optimise the IO appropriately. We can use
REQ_FUAforO_DSYNCwrites ifIOMAP_F_DIRTYis not set. We can do inline completion ifIOMAP_F_DIRTYis not set. But ifIOMAP_F_DIRTYis set, the filesystem needs to run it's own completion processing, and so iomap cannot run that write with an inline completion.
Gotcha, so we need to gate INLINE_COMP on !IOMAP_F_DIRTY as well. I'll make that change.
Quoted 9 lines
Quoted 3 lines
Gate the inline completion on whether we're in a task or not as well. This will always be true for polled IO, but for IRQ driven IO, the completion context may not allow for inline completions.
Again, context does not matter for pure overwrites - we can complete them inline regardless of completion context. The task context only matters when the filesystem needs to do completion work, and we've already established that we are not doing inline completion for polled IO for unwritten,
O_DSYNCor extending file writes.
Right, looks like I was just missing that bit as well, I assumed that the previous case would co er it.
Quoted 3 lines
IOWs, we already avoid polled completions for all the situations where
IOMAP_F_DIRTYis set by the filesystem to indicate the operation is not a pure overwrite....
Yep
Quoted 17 lines
Quoted 17 lines
fs/iomap/direct-io.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)fs/iomap/direct-io.c +1
-0@@ -20,6 +20,7 @@* Private flags for iomap_dio, must not overlap with the public ones in * iomap.h: */ +#define IOMAP_DIO_INLINE_COMP (1 << 27)#define IOMAP_DIO_WRITE_FUA (1 << 28)#define IOMAP_DIO_NEED_SYNC (1 << 29)#define IOMAP_DIO_WRITE (1 << 30)@@ -161,15 +162,15 @@ void iomap_dio_bio_end_io(struct bio *bio)struct task_struct *waiter = dio->submit.waiter;
Quoted 8 lines
fs/iomap/direct-io.c void iomap_dio_bio_end_io(struct bio *bio) · line 165
Quoted 4 lines
fs/iomap/direct-io.c +1
-1WRITE_ONCE (dio->submit.waiter,NULL );blk_wake_io_task (waiter); - }else if (dio->flags &IOMAP_DIO_WRITE ) { + }else if ((dio->flags &IOMAP_DIO_INLINE_COMP ) &&in_task ()) {Regardless of whether the code is correct or not, this needs a comment explaining what problem the
in_task()check is working around...
It's meant to catch cases where we're doing polled IO, but it got cleared/disabled in the block layer. We cannot catch this at submission time, it has to be checked at completion time. There are a few ways we could check for that, one would be in_task(), another would be to check the bio REQ_POLLED flag like v1 did. I don't have a strong preference here, though it did seem like a saner check to use in_task() as generic catch-all for if we're doing this from soft/hard irq processing or not, unexpectedly.
Quoted 22 lines
Quoted 22 lines
fs/iomap/direct-io.c +4
-3+WRITE_ONCE (dio->iocb->private,NULL ); +iomap_dio_complete_work (&dio->aio.work); + }else {struct inode *inode =file_inode (dio->iocb->ki_filp);WRITE_ONCE (dio->iocb->private,NULL );INIT_WORK (&dio->aio.work, iomap_dio_complete_work);queue_work (inode->i_sb->s_dio_done_wq, &dio->aio.work); - }else { -WRITE_ONCE (dio->iocb->private,NULL ); -iomap_dio_complete_work (&dio->aio.work); } }@@ -244,6 +245,7 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter,if (iomap->type ==IOMAP_UNWRITTEN ) { dio->flags |=IOMAP_DIO_UNWRITTEN ; + dio->flags &= ~IOMAP_DIO_INLINE_COMP ; need_zeroout = true; }
Quoted 13 lines
fs/iomap/direct-io.c __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, · line 507
Quoted 8 lines
fs/iomap/direct-io.c +2
-1@@ -500,7 +502,8 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, dio->i_size =i_size_read (inode); dio->dops = dops; dio->error =0 ; - dio->flags =0 ; +/* default to inline completion, turned off when not supported */ + dio->flags =IOMAP_DIO_INLINE_COMP ; dio->done_before = done_before;I think this is poorly coded. If we get the clearing logic wrong (as is the case in this patch) then bad things will happen when we run inline completion in an irq context when the filesystem needs to run a transaction. e.g. file extension.
Agree, it seems a bit fragile. The alternative is doing it the other way around, enabling it for cases that we know it'll work for instead. I'll take a stab at that approach along with the other changes.
Quoted 3 lines
It looks to me like you hacked around this "default is wrong" case with the "
in_task()" check in completion, but given that check is completely undocumented....
It's not a hacky work-around, it's a known case that could go wrong.
Quoted 14 lines
fs/iomap/direct-io.c __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, · line 541
Quoted 7 lines
fs/iomap/direct-io.c +1
-0dio->submit.iter = iter;@@ -535,6 +538,7 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,/* for data sync or sync, we need sync completion processing */ if (iocb_is_dsync (iocb)) { dio->flags |=IOMAP_DIO_NEED_SYNC ; + dio->flags &= ~IOMAP_DIO_INLINE_COMP ;This is looks wrong, too. We set
IOMAP_DIO_WRITE_FUAca couple of lines later, and during bio submission we check ifREQ_FUAcan be used ifIOMAP_F_DIRTYis not set. If all the bios we submit useREQ_FUA, then we clearIOMAP_DIO_NEED_SYNCbefore we drop the dio submission reference.
Quoted 5 lines
For such a
REQ_FUAbio chains, we can now safely do inline completion because we don't rungeneric_write_sync()in IO completion now. The filesystem does not need to perform blocking or IO operations in completion, either, so these IOs can be completed in line like any other pure overwrite DIO....
True, non-extending FUA writes would be fine as well.
Thanks for the review!
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH v2 2/5 fs: add IOCB flags related to passing back dio completions
In reply to Jens AxboeAsync dio completions generally happen from hard/soft IRQ context, which means that users like iomap may need to defer some of the completion handling to a workqueue. This is less efficient than having the original issuer handle it, like we do for sync IO, and it adds latency to the completions.
Add IOCB_DIO_DEFER, which the issuer can set if it is able to safely punt these completions to a safe context. If the dio handler is aware of this flag, assign a callback handler in kiocb->dio_complete and associated data io kiocb->private. The issuer will then call this handler with that data from task context.
No functional changes in this patch.
--- include/linux/fs.h | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-)
include/linux/fs.h +28 -2
@@ -338,6 +338,16 @@ enum rw_hint {#define IOCB_NOIO (1 << 20)/* can use bio alloc cache */ #define IOCB_ALLOC_CACHE (1 << 21) +/* +* IOCB_DIO_DEFER can be set by the iocb owner, to indicate that the +* iocb completion can be passed back to the owner for execution from a safe +* context rather than needing to be punted through a workqueue. If this +* flag is set, the completion handling may set iocb->dio_complete to a +* handler, which the issuer will then call from task context to complete +* the processing of the iocb. iocb->private should then also be set to +* the argument being passed to this handler. +*/ +#define IOCB_DIO_DEFER (1 << 22)/* for use in trace events */ #define TRACE_IOCB_STRINGS \
@@ -351,7 +361,8 @@ enum rw_hint { { IOCB_WRITE, "WRITE" }, \ { IOCB_WAITQ, "WAITQ" }, \ { IOCB_NOIO, "NOIO" }, \ - { IOCB_ALLOC_CACHE, "ALLOC_CACHE" } + { IOCB_ALLOC_CACHE, "ALLOC_CACHE" }, \ + { IOCB_DIO_DEFER, "DIO_DEFER" } struct kiocb {struct file *ki_filp;
@@ -360,7 +371,22 @@ struct kiocb {void *private;int ki_flags;u16 ki_ioprio;/* See linux/ioprio.h */ -struct wait_page_queue *ki_waitq;/* for async buffered IO */ +union { +/* +* Only used for async buffered reads, where it denotes the +* page waitqueue associated with completing the read. Valid +* IFF IOCB_WAITQ is set. +*/ +struct wait_page_queue *ki_waitq; +/* +* Can be used for O_DIRECT IO, where the completion handling +* is punted back to the issuer of the IO. May only be set +* if IOCB_DIO_DEFER is set by the issuer, and the issuer must +* then check for presence of this handler when ki_complete is +* invoked. +*/ +ssize_t (*dio_complete)(void *data); + }; };static inline bool is_sync_kiocb (struct kiocb *kiocb)
Signature · 1 line
2.40.1
Jens Axboe
maintainerPATCH v2 3/5 io_uring/rw: add write support for IOCB_DIO_DEFER
In reply to Jens AxboeIf the filesystem dio handler understands IOCB_DIO_DEFER, we'll get a kiocb->ki_complete() callback with kiocb->dio_complete set. In that case, rather than complete the IO directly through task_work, queue up an intermediate task_work handler that first processes this callback and then immediately completes the request.
For XFS, this avoids a punt through a workqueue, which is a lot less efficient and adds latency to lower queue depth (or sync) O_DIRECT writes.
--- io_uring/rw.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-)
io_uring/rw.c +20 -4
@@ -285,6 +285,14 @@ static inline int io_fixup_rw_res(struct io_kiocb *req, long res)void io_req_rw_complete (struct io_kiocb *req,struct io_tw_state *ts) { + struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); + +if (rw->kiocb.dio_complete) { +long res = rw->kiocb.dio_complete (rw->kiocb.private); + +io_req_set_res (req,io_fixup_rw_res (req, res),0 ); + } +io_req_io_end (req);if (req->flags & (REQ_F_BUFFER_SELECTED |REQ_F_BUFFER_RING )) {
@@ -300,9 +308,11 @@ static void io_complete_rw(struct kiocb *kiocb, long res) struct io_rw *rw = container_of(kiocb, struct io_rw, kiocb);struct io_kiocb *req =cmd_to_io_kiocb (rw); -if (__io_complete_rw_common (req, res)) -return ; -io_req_set_res (req,io_fixup_rw_res (req, res),0 ); +if (!rw->kiocb.dio_complete) { +if (__io_complete_rw_common (req, res)) +return ; +io_req_set_res (req,io_fixup_rw_res (req, res),0 ); + } req->io_task_work.func = io_req_rw_complete;__io_req_task_work_add (req,IOU_F_TWQ_LAZY_WAKE ); }
@@ -914,7 +924,13 @@ int io_write(struct io_kiocb *req, unsigned int issue_flags)__sb_writers_release (file_inode (req->file)->i_sb,SB_FREEZE_WRITE ); } - kiocb->ki_flags |=IOCB_WRITE ; + +/* +* Set IOCB_DIO_DEFER, stating that our handler groks deferring the +* completion to task context. +*/ + kiocb->ki_flags |=IOCB_WRITE |IOCB_DIO_DEFER ; + kiocb->dio_complete =NULL ;if (likely (req->file->f_op->write_iter)) ret2 =call_write_iter (req->file, kiocb, &s->iter);
Signature · 1 line
2.40.1
Jens Axboe
maintainerPATCH v2 4/5 iomap: add local 'iocb' variable in iomap_dio_bio_end_io()
In reply to Jens AxboeWe use this multiple times, add a local variable for the kiocb.
--- fs/iomap/direct-io.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
fs/iomap/direct-io.c +4 -2
@@ -158,6 +158,8 @@ void iomap_dio_bio_end_io(struct bio *bio)iomap_dio_set_error (dio,blk_status_to_errno (bio->bi_status));if (atomic_dec_and_test (&dio->ref)) { +struct kiocb *iocb = dio->iocb; +if (dio->wait_for_completion) {struct task_struct *waiter = dio->submit.waiter;WRITE_ONCE (dio->submit.waiter,NULL );
@@ -166,9 +168,9 @@ void iomap_dio_bio_end_io(struct bio *bio)WRITE_ONCE (dio->iocb->private,NULL );iomap_dio_complete_work (&dio->aio.work); }else { -struct inode *inode =file_inode (dio->iocb->ki_filp); +struct inode *inode =file_inode (iocb->ki_filp); -WRITE_ONCE (dio->iocb->private,NULL ); +WRITE_ONCE (iocb->private,NULL );INIT_WORK (&dio->aio.work, iomap_dio_complete_work);queue_work (inode->i_sb->s_dio_done_wq, &dio->aio.work); }
Signature · 1 line
2.40.1
Jens Axboe
maintainerPATCH v2 5/5 iomap: support IOCB_DIO_DEFER
In reply to Jens AxboeIf IOCB_DIO_DEFER is set, utilize that to set kiocb->dio_complete handler and data for that callback. Rather than punt the completion to a workqueue, we pass back the handler and data to the issuer and will get a callback from a safe task context.
Using the following fio job to randomly dio write 4k blocks at queue depths of 1..16:
fio --name=dio-write --filename=/data1/file --time_based=1 \
--runtime=10 --bs=4096 --rw=randwrite --norandommap --buffered=0 \
--cpus_allowed=4 --ioengine=io_uring --iodepth=16
shows the following results before and after this patch:
Stock Patched Diff
=======================================
QD1 155K 162K + 4.5%
QD2 290K 313K + 7.9%
QD4 533K 597K +12.0%
QD8 604K 827K +36.9%
QD16 615K 845K +37.4%
which shows nice wins all around. If we factored in per-IOP efficiency, the wins look even nicer. This becomes apparent as queue depth rises, as the offloaded workqueue completions runs out of steam.
--- fs/iomap/direct-io.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
fs/iomap/direct-io.c +24 -0
@@ -131,6 +131,11 @@ ssize_t iomap_dio_complete(struct iomap_dio *dio) }EXPORT_SYMBOL_GPL (iomap_dio_complete); +static ssize_t iomap_dio_deferred_complete (void *data) +{ +return iomap_dio_complete (data); +} +static void iomap_dio_complete_work (struct work_struct *work) { struct iomap_dio *dio = container_of(work, struct iomap_dio, aio.work);
@@ -167,6 +172,25 @@ void iomap_dio_bio_end_io(struct bio *bio) }else if ((dio->flags &IOMAP_DIO_INLINE_COMP ) &&in_task ()) {WRITE_ONCE (dio->iocb->private,NULL );iomap_dio_complete_work (&dio->aio.work); + }else if ((dio->flags &IOMAP_DIO_INLINE_COMP ) && + (iocb->ki_flags &IOCB_DIO_DEFER )) { +/* only polled IO cares about private cleared */ + iocb->private = dio; + iocb->dio_complete = iomap_dio_deferred_complete; +/* +* Invoke ->ki_complete() directly. We've assigned +* out dio_complete callback handler, and since the +* issuer set IOCB_DIO_DEFER, we know their +* ki_complete handler will notice ->dio_complete +* being set and will defer calling that handler +* until it can be done from a safe task context. +* +* Note that the 'res' being passed in here is +* not important for this case. The actual completion +* value of the request will be gotten from dio_complete +* when that is run by the issuer. +*/ + iocb->ki_complete (iocb,0 ); }else {struct inode *inode =file_inode (iocb->ki_filp);
Signature · 1 line
2.40.1
Re: [PATCH 5/5] iomap: support IOCB_DIO_DEFER
Quoted 14 lines · Jens Axboe
If
IOCB_DIO_DEFERis set, utilize that to setkiocb->dio_completehandler and data for that callback. Rather than punt the completion to a workqueue, we pass back the handler and data to the issuer and will get a callback from a safe task context.Using the following fio job to randomly dio write 4k blocks at queue depths of 1..16:
fio --name=dio-write --filename=/data1/file --time_based=1 \
--runtime=10 --bs=4096 --rw=randwrite --norandommap --buffered=0 \
--cpus_allowed=4 --ioengine=io_uring --iodepth=16shows the following results before and after this patch:
Quoted 7 lines
Stock Patched Diff
=======================================
QD1 155K 162K + 4.5%
QD2 290K 313K + 7.9%
QD4 533K 597K +12.0%
QD8 604K 827K +36.9%
QD16 615K 845K +37.4%
Nice.
Quoted 41 lines
which shows nice wins all around. If we factored in per-IOP efficiency, the wins look even nicer. This becomes apparent as queue depth rises, as the offloaded workqueue completions runs out of steam.
Signed-off-by Jens Axboe <[email protected]>--- fs/iomap/direct-io.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)fs/iomap/direct-io.c +17
-0@@ -131,6 +131,11 @@ ssize_t iomap_dio_complete(struct iomap_dio *dio) }EXPORT_SYMBOL_GPL (iomap_dio_complete); +static ssize_t iomap_dio_deferred_complete (void *data) +{ +return iomap_dio_complete (data); +} +static void iomap_dio_complete_work (struct work_struct *work) { struct iomap_dio *dio = container_of(work, struct iomap_dio, aio.work);@@ -167,6 +172,25 @@ void iomap_dio_bio_end_io(struct bio *bio) }else if ((dio->flags &IOMAP_DIO_INLINE_COMP ) &&in_task ()) {WRITE_ONCE (dio->iocb->private,NULL );iomap_dio_complete_work (&dio->aio.work); + }else if ((dio->flags &IOMAP_DIO_INLINE_COMP ) && + (iocb->ki_flags &IOCB_DIO_DEFER )) { +/* only polled IO cares about private cleared */ + iocb->private = dio; + iocb->dio_complete = iomap_dio_deferred_complete; +/* +* Invoke ->ki_complete() directly. We've assigned +* out dio_complete callback handler, and since the +* issuer set IOCB_DIO_DEFER, we know their +* ki_complete handler will notice ->dio_complete +* being set and will defer calling that handler +* until it can be done from a safe task context.
fs/iomap/direct-io.c void iomap_dio_bio_end_io(struct bio *bio) · line 196
Quoted 10 lines
fs/iomap/direct-io.c +7
-0+* +* Note that the 'res' being passed in here is +* not important for this case. The actual completion +* value of the request will be gotten from dio_complete +* when that is run by the issuer. +*/ + iocb->ki_complete (iocb,0 ); }else {struct inode *inode =file_inode (iocb->ki_filp);
Hmmm. No problems with the change, but all the special cases is making the completion function a bit of a mess.
Given that all read DIOs use inline completions, we can largely simplify the completion down to just looking at dio->wait_for_completion and IOMAP_DIO_COMPLETE_INLINE, and not caring about what type of IO is being completed at all.
Hence I think that at the end of this series, the completion function should look something like this:
void iomap_dio_bio_end_io(struct bio *bio)
{
struct iomap_dio *dio = bio->bi_private;
struct kiocb *iocb = dio->iocb;
bool should_dirty = (dio->flags & IOMAP_DIO_DIRTY);
ssize_t result = 0;
if (bio->bi_status)
iomap_dio_set_error(dio, blk_status_to_errno(bio->bi_status));
if (!atomic_dec_and_test(&dio->ref))
goto release_bio;
/* Synchronous IO completion. */
if (dio->wait_for_completion) {
struct task_struct *waiter = dio->submit.waiter;
WRITE_ONCE(dio->submit.waiter, NULL);
blk_wake_io_task(waiter);
goto release_bio;
}
/*
* Async DIO completion that requires filesystem level
* completion work gets punted to a work queue to complete
* as the operation may require more IO to be issued to
* finalise filesystem metadata changes or guarantee data
* integrity.
*/
if (!(dio->flags & IOMAP_DIO_COMPLETE_INLINE)) {
struct inode *inode = file_inode(iocb->ki_filp);
WRITE_ONCE(iocb->private, NULL);
INIT_WORK(&dio->aio.work, iomap_dio_complete_work);
queue_work(inode->i_sb->s_dio_done_wq, &dio->aio.work);
goto release_bio;
}
/*
* Inline completion for async DIO.
*
* If the IO submitter is running DIO completions directly
* itself, set up the callback it needs. The value we pass
* to .ki_complete in this case does not matter, the defered
* completion will pull the result from the completion
* callback we provide.
*
* Otherwise, run the dio completion directly, then pass the
* result to the iocb completion function to finish the IO.
*/
if (iocb->ki_flags & IOCB_DEFER_DIO) {
WRITE_ONCE(iocb->private, dio);
iocb->dio_complete = iomap_dio_deferred_complete;
} else {
WRITE_ONCE(dio->iocb->private, NULL);
result = iomap_dio_complete(dio);
}
iocb->ki_complete(iocb, result);release_bio:
if (should_dirty) {
bio_check_pages_dirty(bio);
} else {
bio_release_pages(bio, false);
bio_put(bio);
}
}-Dave.
Signature · 2 lines
Dave Chinner [email protected]
Jens Axboe
maintainerRe: [PATCH 5/5] iomap: support IOCB_DIO_DEFER
Quoted 30 lines · Dave Chinner
Quoted 16 lines
fs/iomap/direct-io.c +12
-0@@ -167,6 +172,25 @@ void iomap_dio_bio_end_io(struct bio *bio) }else if ((dio->flags &IOMAP_DIO_INLINE_COMP ) &&in_task ()) {WRITE_ONCE (dio->iocb->private,NULL );iomap_dio_complete_work (&dio->aio.work); + }else if ((dio->flags &IOMAP_DIO_INLINE_COMP ) && + (iocb->ki_flags &IOCB_DIO_DEFER )) { +/* only polled IO cares about private cleared */ + iocb->private = dio; + iocb->dio_complete = iomap_dio_deferred_complete; +/* +* Invoke ->ki_complete() directly. We've assigned +* out dio_complete callback handler, and since the +* issuer set IOCB_DIO_DEFER, we know their +* ki_complete handler will notice ->dio_complete +* being set and will defer calling that handler +* until it can be done from a safe task context. fs/iomap/direct-io.c void iomap_dio_bio_end_io(struct bio *bio) · line 196
Quoted 10 lines
fs/iomap/direct-io.c +7
-0+* +* Note that the 'res' being passed in here is +* not important for this case. The actual completion +* value of the request will be gotten from dio_complete +* when that is run by the issuer. +*/ + iocb->ki_complete (iocb,0 ); }else {struct inode *inode =file_inode (iocb->ki_filp);Hmmm. No problems with the change, but all the special cases is making the completion function a bit of a mess.
Quoted 7 lines
Given that all read DIOs use inline completions, we can largely simplify the completion down to just looking at
dio->wait_for_completionandIOMAP_DIO_COMPLETE_INLINE, and not caring about what type of IO is being completed at all.Hence I think that at the end of this series, the completion function should look something like this:
I took inspiration from this as I think it's a good idea, and did a few cleanups and introduced things like the above as we go. It's in v3 I just posted.
Signature · 1 line
Jens Axboe