block: skip redundant flush for O_DSYNC direct writes
PATCH 1/1 block: skip redundant flush for O_DSYNC direct writes
For an O_DIRECT | O_DSYNC write to a block device, dio_bio_write_op() adds REQ_FUA to the bio, so the data is durable when the direct-IO returns. blkdev_write_iter() then falls through to generic_write_sync(), which for O_DSYNC turns into an additional REQ_PREFLUSH. That flush is redundant: the FUA write already made the data persistent.
Skip generic_write_sync() when the direct-IO path already provided durability via FUA. The buffered and direct_write_fallback() paths are unchanged.
Measured on a Seagate ST20000NM007D (20 TB, 7200 rpm, fua=1, write_cache=write back), Linux v7.2.0-rc7, single-threaded pwrite() loop opening the raw block device with O_WRONLY | O_DIRECT | O_DSYNC, 4 KiB writes for 60 s:
Sequential 4 KiB writes:
baseline patched
IOPS 119.7 7497.0
avg latency (us) 8357 133
p50 latency (us) 8346 127
p99 latency (us) 8368 395
p99.9 latency (us) 8728 569Random 4 KiB writes (100 GiB span):
baseline patched
IOPS 156.1 666.4
avg latency (us) 6405 1500
p50 latency (us) 6186 1450
p99 latency (us) 16133 2285
p99.9 latency (us) 17250 9916--- block/fops.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
block/fops.c +7 -2
@@ -726,6 +726,7 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from)bool atomic = iocb->ki_flags &IOCB_ATOMIC ;loff_t size =bdev_nr_bytes (bdev);size_t shorted =0 ; +bool dio_fua_done = false;ssize_t ret;if (bdev_read_only (bdev))
@@ -763,9 +764,13 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from)if (iocb->ki_flags &IOCB_DIRECT ) { ret =blkdev_direct_write (iocb, from); -if (ret >=0 &&iov_iter_count (from)) +if (ret >=0 &&iov_iter_count (from)) { ret =direct_write_fallback (iocb, from, ret,blkdev_buffered_write (iocb, from)); + }else if (ret >0 &&iocb_is_dsync (iocb)) { +/* FUA from dio_bio_write_op() already made it durable */ + dio_fua_done = true; + } }else {/* * Take i_rwsem and invalidate_lock to avoid racing with
@@ -777,7 +782,7 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from)inode_unlock_shared (bd_inode); } -if (ret >0 ) +if (ret >0 && !dio_fua_done) ret =generic_write_sync (iocb, ret);iov_iter_reexpand (from,iov_iter_count (from) + shorted);return ret;
Signature · 1 line
2.43.5
Quoted 5 lines
block/fops.c +1
-0+bool dio_fua_done = false;ssize_t ret;if (bdev_read_only (bdev))
block/fops.c static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from) · line 772
Quoted 10 lines
block/fops.c +4
-1@@ -763,9 +764,13 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from)if (iocb->ki_flags &IOCB_DIRECT ) { ret =blkdev_direct_write (iocb, from); -if (ret >=0 &&iov_iter_count (from)) +if (ret >=0 &&iov_iter_count (from)) { ret =direct_write_fallback (iocb, from, ret,blkdev_buffered_write (iocb, from)); + }else if (ret >0 &&iocb_is_dsync (iocb)) { +/* FUA from dio_bio_write_op() already made it durable */ + dio_fua_done = true;
This is kinda the wrong way aroud, it should just have a need_sync variable that is only set for the fallback.
But looking at this, the original logic here actually is very fishy. If we have a driver that does not actually support FUA, emulating it for every write command is a bad idea. So this should be fixed to only do FUA when actually useful, and then return from the non-fallback implementations if that actually happened.
Quoted 7 lines · Christoph Hellwig
Quoted 7 lines
block/fops.c +1
-0+bool dio_fua_done = false;ssize_t ret;if (bdev_read_only (bdev))@@ -763,9 +764,13 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from)
Quoted 12 lines
block/fops.c static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from) · line 772
Quoted 9 lines
block/fops.c +4
-1if (iocb->ki_flags &IOCB_DIRECT ) { ret =blkdev_direct_write (iocb, from); -if (ret >=0 &&iov_iter_count (from)) +if (ret >=0 &&iov_iter_count (from)) { ret =direct_write_fallback (iocb, from, ret,blkdev_buffered_write (iocb, from)); + }else if (ret >0 &&iocb_is_dsync (iocb)) { +/* FUA from dio_bio_write_op() already made it durable */ + dio_fua_done = true;This is kinda the wrong way aroud, it should just have a
need_syncvariable that is only set for the fallback.
Agreed, that reads much cleaner, I'll modify the flag in v2.
Quoted 5 lines
But looking at this, the original logic here actually is very fishy. If we have a driver that does not actually support FUA, emulating it for every write command is a bad idea. So this should be fixed to only do FUA when actually useful, and then return from the non-fallback implementations if that actually happened.
Let me make sure:
blkdev_write_iter() then only skips generic_write_sync() when the direct-IO path actually issued REQ_FUA (i.e. iocb_is_dsync(iocb) && bdev_fua(bdev) && the write completed without falling back to buffered). Is that the direction you had in mind? If yes, I'll send patch v2 later.
Thanks for the review,
Zhenxian
Quoted 11 lines · 马振先
Quoted 5 lines
But looking at this, the original logic here actually is very fishy. If we have a driver that does not actually support FUA, emulating it for every write command is a bad idea. So this should be fixed to only do FUA when actually useful, and then return from the non-fallback implementations if that actually happened.
Let me make sure:
blkdev_write_iter()then only skipsgeneric_write_sync()when the direct-IO path actually issuedREQ_FUA(i.e.iocb_is_dsync(iocb)&&bdev_fua(bdev)&& the write completed without falling back to buffered). Is that the direction you had in mind? If yes, I'll send patch
Yes. If you want you can probably do this as an incremental patch on top of the new version of this patch.
v2 cover block: avoid redundant flushes for O_DSYNC direct writes
In reply to Zhenxian MaThis series removes redundant cache flushes on the O_DIRECT | O_DSYNC write path to a block device, following Christoph's review of v1.
Patch 1 skips generic_write_sync() when the direct write was already made durable by REQ_FUA.
Patch 2 stops setting REQ_FUA on synchronous O_DSYNC direct writes to devices without FUA support, relying on generic_write_sync() instead and avoiding per-bio POSTFLUSH emulation. Asynchronous writes keep REQ_FUA, since they cannot fall back to generic_write_sync().
Changes since v1
[https://lore.kernel.org/linux-block/[email protected]/]:
- Split into two patches per Christoph's suggestion.
- Patch 1 tracks need_sync (default false) instead of dio_fua_done.
- Added Patch 2 for the no-FUA case; asynchronous submitters stay at parity with mainline.
Tested on Linux 7.2.0-rc7 with a Seagate ST20000NM007D (fua=1), 4 KiB O_DSYNC direct writes, mainline vs this series:
Sequential: 119.7 -> 7497.0 IOPS
Random: 156.1 -> 666.4 IOPS
On a loop device without FUA there was no regression, and async O_DSYNC direct writes were verified durable with fio verify=crc32c (262144 IOs, no errors).
Zhenxian Ma (2):
block: skip redundant flush for O_DSYNC direct writes
block: only use REQ_FUA for direct writes if the device supports it
block/fops.c | 40 +++++++++++++++++++++++++++++++++------- 1 file changed, 33 insertions(+), 7 deletions(-)
Signature · 1 line
2.43.5
PATCH v2 1/2 block: skip redundant flush for O_DSYNC direct writes
For an O_DIRECT | O_DSYNC write, dio_bio_write_op() adds REQ_FUA to the bio, so the data is durable once the direct I/O returns. The unconditional generic_write_sync() in blkdev_write_iter() then issues a REQ_PREFLUSH that is redundant.
Skip it when the direct path already provided durability via FUA. A need_sync flag, clear by default, is set only for buffered writes and for the buffered fallback after a partial direct write.
Measured on a Seagate ST20000NM007D (20 TB, 7200 rpm, fua=1, write_cache=write back), Linux v7.2.0-rc7, single-threaded pwrite() loop opening the raw block device with O_WRONLY | O_DIRECT | O_DSYNC, 4 KiB writes for 60 s:
Sequential 4 KiB writes:
baseline patched
IOPS 119.7 7497.0
avg latency (us) 8357 133
p50 latency (us) 8346 127
p99 latency (us) 8368 395
p99.9 latency (us) 8728 569Random 4 KiB writes (100 GiB span):
baseline patched
IOPS 156.1 666.4
avg latency (us) 6405 1500
p50 latency (us) 6186 1450
p99 latency (us) 16133 2285
p99.9 latency (us) 17250 9916--- block/fops.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
block/fops.c +6 -2
@@ -726,6 +726,7 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from)bool atomic = iocb->ki_flags &IOCB_ATOMIC ;loff_t size =bdev_nr_bytes (bdev);size_t shorted =0 ; +bool need_sync = false;ssize_t ret;if (bdev_read_only (bdev))
@@ -763,9 +764,11 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from)if (iocb->ki_flags &IOCB_DIRECT ) { ret =blkdev_direct_write (iocb, from); -if (ret >=0 &&iov_iter_count (from)) +if (ret >=0 &&iov_iter_count (from)) { ret =direct_write_fallback (iocb, from, ret,blkdev_buffered_write (iocb, from)); + need_sync = true; + } }else {/* * Take i_rwsem and invalidate_lock to avoid racing with
@@ -775,9 +778,10 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from)inode_lock_shared (bd_inode); ret =blkdev_buffered_write (iocb, from);inode_unlock_shared (bd_inode); + need_sync = true; } -if (ret >0 ) +if (ret >0 && need_sync) ret =generic_write_sync (iocb, ret);iov_iter_reexpand (from,iov_iter_count (from) + shorted);return ret;
Signature · 1 line
2.43.5
Looks good:
PATCH v2 2/2 block: only use REQ_FUA for direct writes if the device supports it
In reply to Zhenxian MaFrom: Zhenxian Ma <[email protected]>
When a block device does not support FUA natively, the block layer emulates it by adding a cache flush to every write bio. An O_DSYNC direct write that spans N bios then costs N flushes, rather than the single generic_write_sync() issued after the write completes.
Introduce blkdev_dio_fua() to decide when REQ_FUA is set. A synchronous write can rely on generic_write_sync() when the device lacks FUA, so it sets REQ_FUA only when bdev_fua() is true. An asynchronous write completes in blkdev_bio_end_io() and cannot call the blocking generic_write_sync(), so it keeps REQ_FUA (emulated when needed) to stay durable.
--- block/fops.c | 32 +++++++++++++++++++++++++++----- 1 file changed, 27 insertions(+), 5 deletions(-)
block/fops.c +27 -5
@@ -26,12 +26,26 @@ static inline struct inode *bdev_file_inode(struct file *file)return file->f_mapping->host; } -static blk_opf_t dio_bio_write_op (struct kiocb *iocb) +static bool blkdev_dio_fua (struct kiocb *iocb,struct block_device *bdev) +{ +if (!iocb_is_dsync (iocb)) +return false; +/* +* Async writes cannot fall back to generic_write_sync(), so they must +* use FUA (emulated if needed); sync writes only need it when the +* device supports FUA natively. +*/ +if (!is_sync_kiocb (iocb)) +return true; +return bdev_fua (bdev); +} + +static blk_opf_t dio_bio_write_op (struct kiocb *iocb,struct block_device *bdev) {blk_opf_t opf =REQ_OP_WRITE |REQ_SYNC |REQ_IDLE ;/* avoid the need for a I/O completion work item */ -if (iocb_is_dsync (iocb)) +if (blkdev_dio_fua (iocb, bdev)) opf |=REQ_FUA ;return opf; }
@@ -75,7 +89,8 @@ static ssize_t __blkdev_direct_IO_simple(struct kiocb *iocb,if (user_backed_iter (iter)) should_dirty = true; }else { -bio_init (&bio, bdev, vecs, nr_pages,dio_bio_write_op (iocb)); +bio_init (&bio, bdev, vecs, nr_pages, +dio_bio_write_op (iocb, bdev)); } bio.bi_iter.bi_sector = pos >>SECTOR_SHIFT ; bio.bi_write_hint =file_inode (iocb->ki_filp)->i_write_hint;
@@ -179,7 +194,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter,struct blkdev_dio *dio;struct bio *bio;bool is_read = (iov_iter_rw (iter) ==READ ), is_sync; -blk_opf_t opf = is_read ?REQ_OP_READ :dio_bio_write_op (iocb); +blk_opf_t opf = is_read ?REQ_OP_READ :dio_bio_write_op (iocb, bdev);loff_t pos = iocb->ki_pos;int ret =0 ;
@@ -323,7 +338,7 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb, unsigned int nr_pages) {bool is_read =iov_iter_rw (iter) ==READ ; -blk_opf_t opf = is_read ?REQ_OP_READ :dio_bio_write_op (iocb); +blk_opf_t opf = is_read ?REQ_OP_READ :dio_bio_write_op (iocb, bdev);struct blkdev_dio *dio;struct bio *bio;loff_t pos = iocb->ki_pos;
@@ -768,6 +783,13 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from) ret =direct_write_fallback (iocb, from, ret,blkdev_buffered_write (iocb, from)); need_sync = true; + }else if (ret >0 && !blkdev_dio_fua (iocb, bdev)) { +/* +* The device does not support FUA, so REQ_FUA was not +* set and the O_DSYNC direct write still needs a flush +* to be made durable. +*/ + need_sync = true; } }else {/*
Signature · 1 line
2.43.5
Re: [PATCH v2 2/2] block: only use REQ_FUA for direct writes if the device supports it
Quoted 29 lines · Zhenxian Ma
From: Zhenxian Ma <[email protected]>
When a block device does not support FUA natively, the block layer emulates it by adding a cache flush to every write bio. An
O_DSYNCdirect write that spans N bios then costs N flushes, rather than the singlegeneric_write_sync()issued after the write completes.Introduce
blkdev_dio_fua()to decide whenREQ_FUAis set. A synchronous write can rely ongeneric_write_sync()when the device lacks FUA, so it setsREQ_FUAonly whenbdev_fua()is true. An asynchronous write completes inblkdev_bio_end_io()and cannot call the blockinggeneric_write_sync(), so it keepsREQ_FUA(emulated when needed) to stay durable.Suggested-by Christoph Hellwig <[email protected]>Signed-off-by Zhenxian Ma <[email protected]>Signed-off-by Zhenxian Ma <[email protected]>--- block/fops.c | 32 +++++++++++++++++++++++++++----- 1 file changed, 27 insertions(+), 5 deletions(-)block/fops.c +0
-0@@ -26,12 +26,26 @@ static inline struct inode *bdev_file_inode(struct file *file)return file->f_mapping->host; }
block/fops.c static inline struct inode *bdev_file_inode(struct file *file) · line 37
Quoted 10 lines
block/fops.c +9
-1-static blk_opf_t dio_bio_write_op (struct kiocb *iocb) +static bool blkdev_dio_fua (struct kiocb *iocb,struct block_device *bdev) +{ +if (!iocb_is_dsync (iocb)) +return false; +/* +* Async writes cannot fall back to generic_write_sync(), so they must +* use FUA (emulated if needed); sync writes only need it when the +* device supports FUA natively. +*/
Ah, I guess this explain why the original code unconditionally uses FUA. Let's hope no one cares about the performance of large async writes on devices without FUA.
Looks good:
Re: [PATCH v2 0/2] block: avoid redundant flushes for O_DSYNC direct writes
In reply to Zhenxian MaHi Jens,
Gentle ping, both patches have been reviewed by Christoph Hellwig.
Happy to rebase if that helps.
Thanks, Zhenxian
Quoted 40 lines · Zhenxian Ma
This series removes redundant cache flushes on the
O_DIRECT|O_DSYNCwrite path to a block device, following Christoph's review of v1.Patch 1 skips
generic_write_sync()when the direct write was already made durable byREQ_FUA.Patch 2 stops setting
REQ_FUAon synchronousO_DSYNCdirect writes to devices without FUA support, relying ongeneric_write_sync()instead and avoiding per-bio POSTFLUSH emulation. Asynchronous writes keepREQ_FUA, since they cannot fall back togeneric_write_sync().Changes since v1
[https://lore.kernel.org/linux-block/[email protected]/]:- Split into two patches per Christoph's suggestion.
- Patch 1 tracksneed_sync(default false) instead ofdio_fua_done.
- Added Patch 2 for the no-FUA case; asynchronous submitters stay at parity with mainline.Tested on Linux 7.2.0-rc7 with a Seagate ST20000NM007D (fua=1), 4 KiB
O_DSYNCdirect writes, mainline vs this series:Sequential: 119.7 -> 7497.0 IOPS
Random: 156.1 -> 666.4 IOPSOn a loop device without FUA there was no regression, and async
O_DSYNCdirect writes were verified durable with fio verify=crc32c (262144 IOs, no errors).Zhenxian Ma (2):
block: skip redundant flush forO_DSYNCdirect writes
block: only useREQ_FUAfor direct writes if the device supports itblock/fops.c | 40 +++++++++++++++++++++++++++++++++------- 1 file changed, 33 insertions(+), 7 deletions(-)--
2.43.5
Re: <外部邮件>[PATCH v2 0/2] block: avoid redundant flushes for O_DSYNC direct writes
In reply to Zhenxian MaHi Jens,
Gentle ping, both patches have been reviewed by Christoph Hellwig.
Happy to rebase if that helps.
Thanks,
Zhenxian
Quoted 35 lines
2026年8月15日 17:06,Zhenxian Ma <[email protected]> 写道:
This series removes redundant cache flushes on the
O_DIRECT|O_DSYNCwrite path to a block device, following Christoph's review of v1.Patch 1 skips
generic_write_sync()when the direct write was already made durable byREQ_FUA.Patch 2 stops setting
REQ_FUAon synchronousO_DSYNCdirect writes to devices without FUA support, relying ongeneric_write_sync()instead and avoiding per-bio POSTFLUSH emulation. Asynchronous writes keepREQ_FUA, since they cannot fall back togeneric_write_sync().Changes since v1
[https://lore.kernel.org/linux-block/[email protected]/]:- Split into two patches per Christoph's suggestion.
- Patch 1 tracksneed_sync(default false) instead ofdio_fua_done.
- Added Patch 2 for the no-FUA case; asynchronous submitters stay at parity with mainline.Tested on Linux 7.2.0-rc7 with a Seagate ST20000NM007D (fua=1), 4 KiB
O_DSYNCdirect writes, mainline vs this series:Sequential: 119.7 -> 7497.0 IOPS
Random: 156.1 -> 666.4 IOPSOn a loop device without FUA there was no regression, and async
O_DSYNCdirect writes were verified durable with fio verify=crc32c (262144 IOs, no errors).Zhenxian Ma (2):
block: skip redundant flush forO_DSYNCdirect writes
block: only useREQ_FUAfor direct writes if the device supports it
Quoted 6 lines
block/fops.c| 40 +++++++++++++++++++++++++++++++++-------1 file changed, 33 insertions(+), 7 deletions(-)--
2.43.5
本邮件及其附件含有小红书公司的保密信息,仅限于发送给以上收件人或群组。禁止任何其他人以任何形式使用(包括但不限于全部或部分地泄露、复制、或散发)本邮件中的信息。如果您错收了本邮件,请您立即电话或邮件通知发件人并删除本邮件!
This email should be accessible only to the recipients and groups mentioned above for the privileged and confidential information in it concerning Xiaohongshu. The utilization of any information in it (includes but is not limited to leak, copy, or transfer parts of or the whole email content) by anyone outside the permitted group is prohibited. If you receive it by mistake, please notify the sender by phone call or email immediately and delete the message. Thank you.
Re: [PATCH v2 0/2] block: avoid redundant flushes for O_DSYNC direct writes
In reply to Zhenxian MaQuoted 6 lines · Zhenxian Ma
This series removes redundant cache flushes on the
O_DIRECT|O_DSYNCwrite path to a block device, following Christoph's review of v1.Patch 1 skips
generic_write_sync()when the direct write was already made durable byREQ_FUA.
Quoted 6 lines
Patch 2 stops setting
REQ_FUAon synchronousO_DSYNCdirect writes to devices without FUA support, relying ongeneric_write_sync()instead and avoiding per-bio POSTFLUSH emulation. Asynchronous writes keepREQ_FUA, since they cannot fall back togeneric_write_sync().[...]
Applied, thanks!
[1/2] block: skip redundant flush for O_DSYNC direct writes
commit: bec7d36a6514153ded6ffe1323ab14912ced42df
[2/2] block: only use REQ_FUA for direct writes if the device supports it
commit: 0d492f40c4ad7af7b1625fd81dd58a7978ab97f6Best regards,
Signature · 1 line
Jens Axboe