Skip to contentSkip to conversation

block: skip redundant flush for O_DSYNC direct writes

PATCH Superseded Zhenxian Ma 5 participants 12 messages Aug 12 – Sep 15 Block layer

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 block: skip redundant flush for O_DSYNC direct writes

For an O_DIRECT | O_DSYNC write to a block device, dio_bio_write_op() adds REQ_FUA to the bio, so the data is durable when the direct-IO returns. blkdev_write_iter() then falls through to generic_write_sync(), which for O_DSYNC turns into an additional REQ_PREFLUSH. That flush is redundant: the FUA write already made the data persistent.

Skip generic_write_sync() when the direct-IO path already provided durability via FUA. The buffered and direct_write_fallback() paths are unchanged.

Measured on a Seagate ST20000NM007D (20 TB, 7200 rpm, fua=1, write_cache=write back), Linux v7.2.0-rc7, single-threaded pwrite() loop opening the raw block device with O_WRONLY | O_DIRECT | O_DSYNC, 4 KiB writes for 60 s:

Sequential 4 KiB writes:

                            baseline    patched
    IOPS                       119.7     7497.0
    avg latency (us)            8357        133
    p50 latency (us)            8346        127
    p99 latency (us)            8368        395
    p99.9 latency (us)          8728        569

Random 4 KiB writes (100 GiB span):

                            baseline    patched
    IOPS                       156.1      666.4
    avg latency (us)            6405       1500
    p50 latency (us)            6186       1450
    p99 latency (us)           16133       2285
    p99.9 latency (us)         17250       9916
---
 block/fops.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
block/fops.c +7 -2
index 15783a6180de..9c34a372b476 100644
@@ -726,6 +726,7 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from)
 	bool atomic = iocb->ki_flags & IOCB_ATOMIC;
 	loff_t size = bdev_nr_bytes(bdev);
 	size_t shorted = 0;
+	bool dio_fua_done = false;
 	ssize_t ret;
 
 	if (bdev_read_only(bdev))
@@ -763,9 +764,13 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from)
 
 	if (iocb->ki_flags & IOCB_DIRECT) {
 		ret = blkdev_direct_write(iocb, from);
-		if (ret >= 0 && iov_iter_count(from))
+		if (ret >= 0 && iov_iter_count(from)) {
 			ret = direct_write_fallback(iocb, from, ret,
 					blkdev_buffered_write(iocb, from));
+		} else if (ret > 0 && iocb_is_dsync(iocb)) {
+			/* FUA from dio_bio_write_op() already made it durable */
+			dio_fua_done = true;
+		}
 	} else {
 		/*
 		 * Take i_rwsem and invalidate_lock to avoid racing with
@@ -777,7 +782,7 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from)
 		inode_unlock_shared(bd_inode);
 	}
 
-	if (ret > 0)
+	if (ret > 0 && !dio_fua_done)
 		ret = generic_write_sync(iocb, ret);
 	iov_iter_reexpand(from, iov_iter_count(from) + shorted);
 	return ret;
Signature · 1 line
2.43.5
#1 lore raw
Quoted 5 lines
block/fops.c +1 -0
+	bool dio_fua_done = false;
 	ssize_t ret;
 
 	if (bdev_read_only(bdev))

block/fops.c static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from) · line 772

Quoted 10 lines
block/fops.c +4 -1
@@ -763,9 +764,13 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from)
 
 	if (iocb->ki_flags & IOCB_DIRECT) {
 		ret = blkdev_direct_write(iocb, from);
-		if (ret >= 0 && iov_iter_count(from))
+		if (ret >= 0 && iov_iter_count(from)) {
 			ret = direct_write_fallback(iocb, from, ret,
 					blkdev_buffered_write(iocb, from));
+		} else if (ret > 0 && iocb_is_dsync(iocb)) {
+			/* FUA from dio_bio_write_op() already made it durable */
+			dio_fua_done = true;

This is kinda the wrong way aroud, it should just have a need_sync variable that is only set for the fallback.

But looking at this, the original logic here actually is very fishy. If we have a driver that does not actually support FUA, emulating it for every write command is a bad idea. So this should be fixed to only do FUA when actually useful, and then return from the non-fallback implementations if that actually happened.

author
#2 lore raw
Quoted 7 lines · Christoph Hellwig
Quoted 7 lines
block/fops.c +1 -0
+     bool dio_fua_done = false;
      ssize_t ret;
 
      if (bdev_read_only(bdev))
@@ -763,9 +764,13 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from)
 
Quoted 12 lines

block/fops.c static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from) · line 772

Quoted 9 lines
block/fops.c +4 -1
      if (iocb->ki_flags & IOCB_DIRECT) {
              ret = blkdev_direct_write(iocb, from);
-             if (ret >= 0 && iov_iter_count(from))
+             if (ret >= 0 && iov_iter_count(from)) {
                      ret = direct_write_fallback(iocb, from, ret,
                                      blkdev_buffered_write(iocb, from));
+             } else if (ret > 0 && iocb_is_dsync(iocb)) {
+                     /* FUA from dio_bio_write_op() already made it durable */
+                     dio_fua_done = true;

This is kinda the wrong way aroud, it should just have a need_sync variable that is only set for the fallback.

Agreed, that reads much cleaner, I'll modify the flag in v2.

Quoted 5 lines

But looking at this, the original logic here actually is very fishy. If we have a driver that does not actually support FUA, emulating it for every write command is a bad idea. So this should be fixed to only do FUA when actually useful, and then return from the non-fallback implementations if that actually happened.

Let me make sure:
blkdev_write_iter() then only skips generic_write_sync() when the direct-IO path actually issued REQ_FUA (i.e. iocb_is_dsync(iocb) && bdev_fua(bdev) && the write completed without falling back to buffered). Is that the direction you had in mind? If yes, I'll send patch v2 later.

Thanks for the review,
Zhenxian

#3 lore raw
Quoted 11 lines · 马振先
Quoted 5 lines

But looking at this, the original logic here actually is very fishy. If we have a driver that does not actually support FUA, emulating it for every write command is a bad idea. So this should be fixed to only do FUA when actually useful, and then return from the non-fallback implementations if that actually happened.

Let me make sure:
blkdev_write_iter() then only skips generic_write_sync() when the direct-IO path actually issued REQ_FUA (i.e. iocb_is_dsync(iocb) && bdev_fua(bdev) && the write completed without falling back to buffered). Is that the direction you had in mind? If yes, I'll send patch

Yes. If you want you can probably do this as an incremental patch on top of the new version of this patch.

#4 lore raw

v2 cover block: avoid redundant flushes for O_DSYNC direct writes

In reply to Zhenxian Ma

This series removes redundant cache flushes on the O_DIRECT | O_DSYNC write path to a block device, following Christoph's review of v1.

Patch 1 skips generic_write_sync() when the direct write was already made durable by REQ_FUA.

Patch 2 stops setting REQ_FUA on synchronous O_DSYNC direct writes to devices without FUA support, relying on generic_write_sync() instead and avoiding per-bio POSTFLUSH emulation. Asynchronous writes keep REQ_FUA, since they cannot fall back to generic_write_sync().

Changes since v1
[https://lore.kernel.org/linux-block/[email protected]/]:

- Split into two patches per Christoph's suggestion.
- Patch 1 tracks need_sync (default false) instead of dio_fua_done.
- Added Patch 2 for the no-FUA case; asynchronous submitters stay at parity with mainline.

Tested on Linux 7.2.0-rc7 with a Seagate ST20000NM007D (fua=1), 4 KiB O_DSYNC direct writes, mainline vs this series:

Sequential: 119.7 -> 7497.0 IOPS
Random: 156.1 -> 666.4 IOPS

On a loop device without FUA there was no regression, and async O_DSYNC direct writes were verified durable with fio verify=crc32c (262144 IOs, no errors).

Zhenxian Ma (2):
block: skip redundant flush for O_DSYNC direct writes
block: only use REQ_FUA for direct writes if the device supports it

 block/fops.c | 40 +++++++++++++++++++++++++++++++++-------
 1 file changed, 33 insertions(+), 7 deletions(-)
Signature · 1 line
2.43.5
#5 lore raw

PATCH v2 1/2 block: skip redundant flush for O_DSYNC direct writes

For an O_DIRECT | O_DSYNC write, dio_bio_write_op() adds REQ_FUA to the bio, so the data is durable once the direct I/O returns. The unconditional generic_write_sync() in blkdev_write_iter() then issues a REQ_PREFLUSH that is redundant.

Skip it when the direct path already provided durability via FUA. A need_sync flag, clear by default, is set only for buffered writes and for the buffered fallback after a partial direct write.

Measured on a Seagate ST20000NM007D (20 TB, 7200 rpm, fua=1, write_cache=write back), Linux v7.2.0-rc7, single-threaded pwrite() loop opening the raw block device with O_WRONLY | O_DIRECT | O_DSYNC, 4 KiB writes for 60 s:

Sequential 4 KiB writes:

                            baseline    patched
    IOPS                       119.7     7497.0
    avg latency (us)            8357        133
    p50 latency (us)            8346        127
    p99 latency (us)            8368        395
    p99.9 latency (us)          8728        569

Random 4 KiB writes (100 GiB span):

                            baseline    patched
    IOPS                       156.1      666.4
    avg latency (us)            6405       1500
    p50 latency (us)            6186       1450
    p99 latency (us)           16133       2285
    p99.9 latency (us)         17250       9916
---
 block/fops.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
block/fops.c +6 -2
index 15783a6180de..a2c3af38106b 100644
@@ -726,6 +726,7 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from)
 	bool atomic = iocb->ki_flags & IOCB_ATOMIC;
 	loff_t size = bdev_nr_bytes(bdev);
 	size_t shorted = 0;
+	bool need_sync = false;
 	ssize_t ret;
 
 	if (bdev_read_only(bdev))
@@ -763,9 +764,11 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from)
 
 	if (iocb->ki_flags & IOCB_DIRECT) {
 		ret = blkdev_direct_write(iocb, from);
-		if (ret >= 0 && iov_iter_count(from))
+		if (ret >= 0 && iov_iter_count(from)) {
 			ret = direct_write_fallback(iocb, from, ret,
 					blkdev_buffered_write(iocb, from));
+			need_sync = true;
+		}
 	} else {
 		/*
 		 * Take i_rwsem and invalidate_lock to avoid racing with
@@ -775,9 +778,10 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from)
 		inode_lock_shared(bd_inode);
 		ret = blkdev_buffered_write(iocb, from);
 		inode_unlock_shared(bd_inode);
+		need_sync = true;
 	}
 
-	if (ret > 0)
+	if (ret > 0 && need_sync)
 		ret = generic_write_sync(iocb, ret);
 	iov_iter_reexpand(from, iov_iter_count(from) + shorted);
 	return ret;
Signature · 1 line
2.43.5
#7 lore raw

PATCH v2 2/2 block: only use REQ_FUA for direct writes if the device supports it

In reply to Zhenxian Ma

From: Zhenxian Ma <[email protected]>

When a block device does not support FUA natively, the block layer emulates it by adding a cache flush to every write bio. An O_DSYNC direct write that spans N bios then costs N flushes, rather than the single generic_write_sync() issued after the write completes.

Introduce blkdev_dio_fua() to decide when REQ_FUA is set. A synchronous write can rely on generic_write_sync() when the device lacks FUA, so it sets REQ_FUA only when bdev_fua() is true. An asynchronous write completes in blkdev_bio_end_io() and cannot call the blocking generic_write_sync(), so it keeps REQ_FUA (emulated when needed) to stay durable.

---
 block/fops.c | 32 +++++++++++++++++++++++++++-----
 1 file changed, 27 insertions(+), 5 deletions(-)
block/fops.c +27 -5
index a2c3af38106b..e7d073620271 100644
@@ -26,12 +26,26 @@ static inline struct inode *bdev_file_inode(struct file *file)
 	return file->f_mapping->host;
 }
 
-static blk_opf_t dio_bio_write_op(struct kiocb *iocb)
+static bool blkdev_dio_fua(struct kiocb *iocb, struct block_device *bdev)
+{
+	if (!iocb_is_dsync(iocb))
+		return false;
+	/*
+	 * Async writes cannot fall back to generic_write_sync(), so they must
+	 * use FUA (emulated if needed); sync writes only need it when the
+	 * device supports FUA natively.
+	 */
+	if (!is_sync_kiocb(iocb))
+		return true;
+	return bdev_fua(bdev);
+}
+
+static blk_opf_t dio_bio_write_op(struct kiocb *iocb, struct block_device *bdev)
 {
 	blk_opf_t opf = REQ_OP_WRITE | REQ_SYNC | REQ_IDLE;
 
 	/* avoid the need for a I/O completion work item */
-	if (iocb_is_dsync(iocb))
+	if (blkdev_dio_fua(iocb, bdev))
 		opf |= REQ_FUA;
 	return opf;
 }
@@ -75,7 +89,8 @@ static ssize_t __blkdev_direct_IO_simple(struct kiocb *iocb,
 		if (user_backed_iter(iter))
 			should_dirty = true;
 	} else {
-		bio_init(&bio, bdev, vecs, nr_pages, dio_bio_write_op(iocb));
+		bio_init(&bio, bdev, vecs, nr_pages,
+			 dio_bio_write_op(iocb, bdev));
 	}
 	bio.bi_iter.bi_sector = pos >> SECTOR_SHIFT;
 	bio.bi_write_hint = file_inode(iocb->ki_filp)->i_write_hint;
@@ -179,7 +194,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter,
 	struct blkdev_dio *dio;
 	struct bio *bio;
 	bool is_read = (iov_iter_rw(iter) == READ), is_sync;
-	blk_opf_t opf = is_read ? REQ_OP_READ : dio_bio_write_op(iocb);
+	blk_opf_t opf = is_read ? REQ_OP_READ : dio_bio_write_op(iocb, bdev);
 	loff_t pos = iocb->ki_pos;
 	int ret = 0;
 
@@ -323,7 +338,7 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb,
 					unsigned int nr_pages)
 {
 	bool is_read = iov_iter_rw(iter) == READ;
-	blk_opf_t opf = is_read ? REQ_OP_READ : dio_bio_write_op(iocb);
+	blk_opf_t opf = is_read ? REQ_OP_READ : dio_bio_write_op(iocb, bdev);
 	struct blkdev_dio *dio;
 	struct bio *bio;
 	loff_t pos = iocb->ki_pos;
@@ -768,6 +783,13 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from)
 			ret = direct_write_fallback(iocb, from, ret,
 					blkdev_buffered_write(iocb, from));
 			need_sync = true;
+		} else if (ret > 0 && !blkdev_dio_fua(iocb, bdev)) {
+			/*
+			 * The device does not support FUA, so REQ_FUA was not
+			 * set and the O_DSYNC direct write still needs a flush
+			 * to be made durable.
+			 */
+			need_sync = true;
 		}
 	} else {
 		/*
Signature · 1 line
2.43.5
Reviewed-by on v2 2/2
#8 lore raw

Re: [PATCH v2 2/2] block: only use REQ_FUA for direct writes if the device supports it

Quoted 29 lines · Zhenxian Ma

From: Zhenxian Ma <[email protected]>

When a block device does not support FUA natively, the block layer emulates it by adding a cache flush to every write bio. An O_DSYNC direct write that spans N bios then costs N flushes, rather than the single generic_write_sync() issued after the write completes.

Introduce blkdev_dio_fua() to decide when REQ_FUA is set. A synchronous write can rely on generic_write_sync() when the device lacks FUA, so it sets REQ_FUA only when bdev_fua() is true. An asynchronous write completes in blkdev_bio_end_io() and cannot call the blocking generic_write_sync(), so it keeps REQ_FUA (emulated when needed) to stay durable.

---
 block/fops.c | 32 +++++++++++++++++++++++++++-----
 1 file changed, 27 insertions(+), 5 deletions(-)
block/fops.c +0 -0
index a2c3af38106b..e7d073620271 100644
@@ -26,12 +26,26 @@ static inline struct inode *bdev_file_inode(struct file *file)
 	return file->f_mapping->host;
 }
 

block/fops.c static inline struct inode *bdev_file_inode(struct file *file) · line 37

Quoted 10 lines
block/fops.c +9 -1
-static blk_opf_t dio_bio_write_op(struct kiocb *iocb)
+static bool blkdev_dio_fua(struct kiocb *iocb, struct block_device *bdev)
+{
+	if (!iocb_is_dsync(iocb))
+		return false;
+	/*
+	 * Async writes cannot fall back to generic_write_sync(), so they must
+	 * use FUA (emulated if needed); sync writes only need it when the
+	 * device supports FUA natively.
+	 */

Ah, I guess this explain why the original code unconditionally uses FUA. Let's hope no one cares about the performance of large async writes on devices without FUA.

Looks good:

#9 lore raw

Re: [PATCH v2 0/2] block: avoid redundant flushes for O_DSYNC direct writes

In reply to Zhenxian Ma

Hi Jens,
Gentle ping, both patches have been reviewed by Christoph Hellwig.

Happy to rebase if that helps.
Thanks, Zhenxian

Quoted 40 lines · Zhenxian Ma

This series removes redundant cache flushes on the O_DIRECT | O_DSYNC write path to a block device, following Christoph's review of v1.

Patch 1 skips generic_write_sync() when the direct write was already made durable by REQ_FUA.

Patch 2 stops setting REQ_FUA on synchronous O_DSYNC direct writes to devices without FUA support, relying on generic_write_sync() instead and avoiding per-bio POSTFLUSH emulation. Asynchronous writes keep REQ_FUA, since they cannot fall back to generic_write_sync().

Changes since v1
[https://lore.kernel.org/linux-block/[email protected]/]:

- Split into two patches per Christoph's suggestion.
- Patch 1 tracks need_sync (default false) instead of dio_fua_done.
- Added Patch 2 for the no-FUA case; asynchronous submitters stay at parity with mainline.

Tested on Linux 7.2.0-rc7 with a Seagate ST20000NM007D (fua=1), 4 KiB O_DSYNC direct writes, mainline vs this series:

Sequential: 119.7 -> 7497.0 IOPS
Random: 156.1 -> 666.4 IOPS

On a loop device without FUA there was no regression, and async O_DSYNC direct writes were verified durable with fio verify=crc32c (262144 IOs, no errors).

Zhenxian Ma (2):
block: skip redundant flush for O_DSYNC direct writes
block: only use REQ_FUA for direct writes if the device supports it

 block/fops.c | 40 +++++++++++++++++++++++++++++++++-------
 1 file changed, 33 insertions(+), 7 deletions(-)

--
2.43.5

#10 lore raw

Re: <外部邮件>[PATCH v2 0/2] block: avoid redundant flushes for O_DSYNC direct writes

In reply to Zhenxian Ma

Hi Jens,

Gentle ping, both patches have been reviewed by Christoph Hellwig.

Happy to rebase if that helps.

Thanks,
Zhenxian

Quoted 35 lines

2026年8月15日 17:06,Zhenxian Ma <[email protected]> 写道:

This series removes redundant cache flushes on the O_DIRECT | O_DSYNC write path to a block device, following Christoph's review of v1.

Patch 1 skips generic_write_sync() when the direct write was already made durable by REQ_FUA.

Patch 2 stops setting REQ_FUA on synchronous O_DSYNC direct writes to devices without FUA support, relying on generic_write_sync() instead and avoiding per-bio POSTFLUSH emulation. Asynchronous writes keep REQ_FUA, since they cannot fall back to generic_write_sync().

Changes since v1
[https://lore.kernel.org/linux-block/[email protected]/]:

- Split into two patches per Christoph's suggestion.
- Patch 1 tracks need_sync (default false) instead of dio_fua_done.
- Added Patch 2 for the no-FUA case; asynchronous submitters stay at parity with mainline.

Tested on Linux 7.2.0-rc7 with a Seagate ST20000NM007D (fua=1), 4 KiB O_DSYNC direct writes, mainline vs this series:

Sequential: 119.7 -> 7497.0 IOPS
Random: 156.1 -> 666.4 IOPS

On a loop device without FUA there was no regression, and async O_DSYNC direct writes were verified durable with fio verify=crc32c (262144 IOs, no errors).

Zhenxian Ma (2):
block: skip redundant flush for O_DSYNC direct writes
block: only use REQ_FUA for direct writes if the device supports it

Quoted 6 lines

block/fops.c | 40 +++++++++++++++++++++++++++++++++-------

1 file changed, 33 insertions(+), 7 deletions(-)

--
2.43.5

本邮件及其附件含有小红书公司的保密信息,仅限于发送给以上收件人或群组。禁止任何其他人以任何形式使用(包括但不限于全部或部分地泄露、复制、或散发)本邮件中的信息。如果您错收了本邮件,请您立即电话或邮件通知发件人并删除本邮件!
This email should be accessible only to the recipients and groups mentioned above for the privileged and confidential information in it concerning Xiaohongshu. The utilization of any information in it (includes but is not limited to leak, copy, or transfer parts of or the whole email content) by anyone outside the permitted group is prohibited. If you receive it by mistake, please notify the sender by phone call or email immediately and delete the message. Thank you.

maintainer Applied
#11 lore raw

Re: [PATCH v2 0/2] block: avoid redundant flushes for O_DSYNC direct writes

In reply to Zhenxian Ma
Quoted 6 lines · Zhenxian Ma

This series removes redundant cache flushes on the O_DIRECT | O_DSYNC write path to a block device, following Christoph's review of v1.

Patch 1 skips generic_write_sync() when the direct write was already made durable by REQ_FUA.

Quoted 6 lines

Patch 2 stops setting REQ_FUA on synchronous O_DSYNC direct writes to devices without FUA support, relying on generic_write_sync() instead and avoiding per-bio POSTFLUSH emulation. Asynchronous writes keep REQ_FUA, since they cannot fall back to generic_write_sync().

[...]

Applied, thanks!

[1/2] block: skip redundant flush for O_DSYNC direct writes

      commit: bec7d36a6514153ded6ffe1323ab14912ced42df
[2/2] block: only use REQ_FUA for direct writes if the device supports it
      commit: 0d492f40c4ad7af7b1625fd81dd58a7978ab97f6

Best regards,

Signature · 1 line
Jens Axboe