Skip to contentSkip to conversation

RFC: iomap write invalidation

PATCH In review Christoph Hellwig 7 participants 27 messages Jul 13, 2020 – Jul 21, 2020

Subsystems (8)
lore
Share & tools
lore
#0 lore raw

Hi all,

this series has two parts: the first one picks up Dave's patch to avoid invalidation entierly for reads, picked up deep down from the btrfs iomap thread. The second one falls back to buffered writes if invalidation fails instead of leaving a stale cache around. Let me know what you think about this approch.

#1 lore raw

PATCH 1/2 iomap: Only invalidate page cache pages on direct IO writes

From: Dave Chinner <[email protected]>

The historic requirement for XFS to invalidate cached pages on direct IO reads has been lost in the twisty pages of history - it was inherited from Irix, which implemented page cache invalidation on read as a method of working around problems synchronising page cache state with uncached IO.

XFS has carried this ever since. In the initial linux ports it was necessary to get mmap and DIO to play "ok" together and not immediately corrupt data. This was the state of play until the linux kernel had infrastructure to track unwritten extents and synchronise page faults with allocations and unwritten extent conversions (->page_mkwrite infrastructure). IOws, the page cache invalidation on DIO read was necessary to prevent trivial data corruptions. This didn't solve all the problems, though.

There were peformance problems if we didn't invalidate the entire page cache over the file on read - we couldn't easily determine if the cached pages were over the range of the IO, and invalidation required taking a serialising lock (i_mutex) on the inode. This serialising lock was an issue for XFS, as it was the only exclusive lock in the direct Io read path.

Hence if there were any cached pages, we'd just invalidate the entire file in one go so that subsequent IOs didn't need to take the serialising lock. This was a problem that prevented ranged invalidation from being particularly useful for avoiding the remaining coherency issues. This was solved with the conversion of i_mutex to i_rwsem and the conversion of the XFS inode IO lock to use i_rwsem. Hence we could now just do ranged invalidation and the performance problem went away.

However, page cache invalidation was still needed to serialise sub-page/sub-block zeroing via direct IO against buffered IO because bufferhead state attached to the cached page could get out of whack when direct IOs were issued. We've removed bufferheads from the XFS code, and we don't carry any extent state on the cached pages anymore, and so this problem has gone away, too.

IOWs, it would appear that we don't have any good reason to be invalidating the page cache on DIO reads anymore. Hence remove the invalidation on read because it is unnecessary overhead, not needed to maintain coherency between mmap/buffered access and direct IO anymore, and prevents anyone from using direct IO reads from intentionally invalidating the page cache of a file.

---
 fs/iomap/direct-io.c | 31 +++++++++++++++----------------
 1 file changed, 15 insertions(+), 16 deletions(-)
fs/iomap/direct-io.c +15 -16
index ec7b78e6fecaf9..190967e87b69e4 100644
@@ -475,23 +475,22 @@ iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 	if (ret)
 		goto out_free_dio;
 
-	/*
-	 * Try to invalidate cache pages for the range we're direct
-	 * writing.  If this invalidation fails, tough, the write will
-	 * still work, but racing two incompatible write paths is a
-	 * pretty crazy thing to do, so we don't support it 100%.
-	 */
-	ret = invalidate_inode_pages2_range(mapping,
-			pos >> PAGE_SHIFT, end >> PAGE_SHIFT);
-	if (ret)
-		dio_warn_stale_pagecache(iocb->ki_filp);
-	ret = 0;
+	if (iov_iter_rw(iter) == WRITE) {
+		/*
+		 * Try to invalidate cache pages for the range we are writing.
+		 * If this invalidation fails, tough, the write will still work,
+		 * but racing two incompatible write paths is a pretty crazy
+		 * thing to do, so we don't support it 100%.
+		 */
+		if (invalidate_inode_pages2_range(mapping, pos >> PAGE_SHIFT,
+				end >> PAGE_SHIFT))
+			dio_warn_stale_pagecache(iocb->ki_filp);
 
-	if (iov_iter_rw(iter) == WRITE && !wait_for_completion &&
-	    !inode->i_sb->s_dio_done_wq) {
-		ret = sb_init_dio_done_wq(inode->i_sb);
-		if (ret < 0)
-			goto out_free_dio;
+		if (!wait_for_completion && !inode->i_sb->s_dio_done_wq) {
+			ret = sb_init_dio_done_wq(inode->i_sb);
+			if (ret < 0)
+				goto out_free_dio;
+		}
 	}
 
 	inode_dio_begin(inode);
Signature · 1 line
2.26.2
#2 lore raw

PATCH 2/2 iomap: fall back to buffered writes for invalidation failures

In reply to Christoph Hellwig

Failing to invalid the page cache means data in incoherent, which is a very bad state for the system. Always fall back to buffered I/O through the page cache if we can't invalidate mappings.

---
 fs/ext4/file.c       |  2 ++
 fs/gfs2/file.c       |  3 ++-
 fs/iomap/direct-io.c | 13 ++++++++-----
 fs/iomap/trace.h     |  1 +
 fs/xfs/xfs_file.c    |  4 ++--
 fs/zonefs/super.c    |  7 +++++--
 6 files changed, 20 insertions(+), 10 deletions(-)
fs/ext4/file.c +2 -0
index 2a01e31a032c4c..0da6c2a2c32c1e 100644
@@ -544,6 +544,8 @@ static ssize_t ext4_dio_write_iter(struct kiocb *iocb, struct iov_iter *from)
 		iomap_ops = &ext4_iomap_overwrite_ops;
 	ret = iomap_dio_rw(iocb, from, iomap_ops, &ext4_dio_write_ops,
 			   is_sync_kiocb(iocb) || unaligned_io || extend);
+	if (ret == -EREMCHG)
+		ret = 0;
 
 	if (extend)
 		ret = ext4_handle_inode_extension(inode, offset, ret, count);
fs/gfs2/file.c +2 -1
index fe305e4bfd3734..c7907d40c61d17 100644
@@ -814,7 +814,8 @@ static ssize_t gfs2_file_direct_write(struct kiocb *iocb, struct iov_iter *from)
 
 	ret = iomap_dio_rw(iocb, from, &gfs2_iomap_ops, NULL,
 			   is_sync_kiocb(iocb));
-
+	if (ret == -EREMCHG)
+		ret = 0;
 out:
 	gfs2_glock_dq(&gh);
 out_uninit:
fs/iomap/direct-io.c +8 -5
index 190967e87b69e4..62626235cdbe8d 100644
@@ -10,6 +10,7 @@
 #include <linux/backing-dev.h>
 #include <linux/uio.h>
 #include <linux/task_io_accounting_ops.h>
+#include "trace.h"
 
 #include "../internal.h"
 
@@ -478,13 +479,15 @@ iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 	if (iov_iter_rw(iter) == WRITE) {
 		/*
 		 * Try to invalidate cache pages for the range we are writing.
-		 * If this invalidation fails, tough, the write will still work,
-		 * but racing two incompatible write paths is a pretty crazy
-		 * thing to do, so we don't support it 100%.
+		 * If this invalidation fails, let the caller fall back to
+		 * buffered I/O.
 		 */
 		if (invalidate_inode_pages2_range(mapping, pos >> PAGE_SHIFT,
-				end >> PAGE_SHIFT))
-			dio_warn_stale_pagecache(iocb->ki_filp);
+				end >> PAGE_SHIFT)) {
+			trace_iomap_dio_invalidate_fail(inode, pos, count);
+			ret = -EREMCHG;
+			goto out_free_dio;
+		}
 
 		if (!wait_for_completion && !inode->i_sb->s_dio_done_wq) {
 			ret = sb_init_dio_done_wq(inode->i_sb);
fs/iomap/trace.h +1 -0
index 5693a39d52fb63..fdc7ae388476f5 100644
@@ -74,6 +74,7 @@ DEFINE_EVENT(iomap_range_class, name,	\
 DEFINE_RANGE_EVENT(iomap_writepage);
 DEFINE_RANGE_EVENT(iomap_releasepage);
 DEFINE_RANGE_EVENT(iomap_invalidatepage);
+DEFINE_RANGE_EVENT(iomap_dio_invalidate_fail);
 
 #define IOMAP_TYPE_STRINGS \
 	{ IOMAP_HOLE,		"HOLE" }, \
fs/xfs/xfs_file.c +2 -2
index 00db81eac80d6c..551cca39fa3ba6 100644
@@ -553,8 +553,8 @@ xfs_file_dio_aio_write(
 	xfs_iunlock(ip, iolock);
 
 	/*
-	 * No fallback to buffered IO on errors for XFS, direct IO will either
-	 * complete fully or fail.
+	 * No partial fallback to buffered IO on errors for XFS, direct IO will
+	 * either complete fully or fail.
 	 */
 	ASSERT(ret < 0 || ret == count);
 	return ret;
fs/zonefs/super.c +5 -2
index 07bc42d62673ce..793850454b752f 100644
@@ -786,8 +786,11 @@ static ssize_t zonefs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
 	if (iocb->ki_pos >= ZONEFS_I(inode)->i_max_size)
 		return -EFBIG;
 
-	if (iocb->ki_flags & IOCB_DIRECT)
-		return zonefs_file_dio_write(iocb, from);
+	if (iocb->ki_flags & IOCB_DIRECT) {
+		ret = zonefs_file_dio_write(iocb, from);
+		if (ret != -EREMCHG)
+			return ret;
+	}
 
 	return zonefs_file_buffered_write(iocb, from);
 }
Signature · 1 line
2.26.2
#3 lore raw

Re: [PATCH 2/2] iomap: fall back to buffered writes for invalidation failures

Quoted 3 lines · Christoph Hellwig

Failing to invalid the page cache means data in incoherent, which is a very bad state for the system. Always fall back to buffered I/O through the page cache if we can't invalidate mappings.

Is that the right approach though? I don't have a full picture in my head, but wouldn't we be better off marking these pages as !Uptodate and doing the direct I/O?

#4 lore raw

Re: [PATCH 2/2] iomap: fall back to buffered writes for invalidation failures

Quoted 8 lines · Matthew Wilcox
Quoted 3 lines · Christoph Hellwig

Failing to invalid the page cache means data in incoherent, which is a very bad state for the system. Always fall back to buffered I/O through the page cache if we can't invalidate mappings.

Is that the right approach though? I don't have a full picture in my head, but wouldn't we be better off marking these pages as !Uptodate and doing the direct I/O?

Isn't that a problem if e.g. pages are mapped into userspace and mlocked?

Reviewed-by on v1 2/2
#5 lore raw

Re: [PATCH 2/2] iomap: fall back to buffered writes for invalidation failures

In reply to Christoph Hellwig
Quoted 5 lines · Christoph Hellwig

Failing to invalid the page cache means data in incoherent, which is a very bad state for the system. Always fall back to buffered I/O through the page cache if we can't invalidate mappings.

Thanks. This will help btrfs. The current next tree contains the iomap changes I recomended and would need to be reverted in order to incorporate this. Once this is in the next tree I will (re)format the btrfs iomap dio patches.

Quoted 117 lines
---
 fs/ext4/file.c       |  2 ++
 fs/gfs2/file.c       |  3 ++-
 fs/iomap/direct-io.c | 13 ++++++++-----
 fs/iomap/trace.h     |  1 +
 fs/xfs/xfs_file.c    |  4 ++--
 fs/zonefs/super.c    |  7 +++++--
 6 files changed, 20 insertions(+), 10 deletions(-)
fs/ext4/file.c +2 -0
index 2a01e31a032c4c..0da6c2a2c32c1e 100644
@@ -544,6 +544,8 @@ static ssize_t ext4_dio_write_iter(struct kiocb *iocb, struct iov_iter *from)
 		iomap_ops = &ext4_iomap_overwrite_ops;
 	ret = iomap_dio_rw(iocb, from, iomap_ops, &ext4_dio_write_ops,
 			   is_sync_kiocb(iocb) || unaligned_io || extend);
+	if (ret == -EREMCHG)
+		ret = 0;
 
 	if (extend)
 		ret = ext4_handle_inode_extension(inode, offset, ret, count);
fs/gfs2/file.c +2 -1
index fe305e4bfd3734..c7907d40c61d17 100644
@@ -814,7 +814,8 @@ static ssize_t gfs2_file_direct_write(struct kiocb *iocb, struct iov_iter *from)
 
 	ret = iomap_dio_rw(iocb, from, &gfs2_iomap_ops, NULL,
 			   is_sync_kiocb(iocb));
-
+	if (ret == -EREMCHG)
+		ret = 0;
 out:
 	gfs2_glock_dq(&gh);
 out_uninit:
fs/iomap/direct-io.c +8 -5
index 190967e87b69e4..62626235cdbe8d 100644
@@ -10,6 +10,7 @@
 #include <linux/backing-dev.h>
 #include <linux/uio.h>
 #include <linux/task_io_accounting_ops.h>
+#include "trace.h"
 
 #include "../internal.h"
 
@@ -478,13 +479,15 @@ iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 	if (iov_iter_rw(iter) == WRITE) {
 		/*
 		 * Try to invalidate cache pages for the range we are writing.
-		 * If this invalidation fails, tough, the write will still work,
-		 * but racing two incompatible write paths is a pretty crazy
-		 * thing to do, so we don't support it 100%.
+		 * If this invalidation fails, let the caller fall back to
+		 * buffered I/O.
 		 */
 		if (invalidate_inode_pages2_range(mapping, pos >> PAGE_SHIFT,
-				end >> PAGE_SHIFT))
-			dio_warn_stale_pagecache(iocb->ki_filp);
+				end >> PAGE_SHIFT)) {
+			trace_iomap_dio_invalidate_fail(inode, pos, count);
+			ret = -EREMCHG;
+			goto out_free_dio;
+		}
 
 		if (!wait_for_completion && !inode->i_sb->s_dio_done_wq) {
 			ret = sb_init_dio_done_wq(inode->i_sb);
fs/iomap/trace.h +1 -0
index 5693a39d52fb63..fdc7ae388476f5 100644
@@ -74,6 +74,7 @@ DEFINE_EVENT(iomap_range_class, name,	\
 DEFINE_RANGE_EVENT(iomap_writepage);
 DEFINE_RANGE_EVENT(iomap_releasepage);
 DEFINE_RANGE_EVENT(iomap_invalidatepage);
+DEFINE_RANGE_EVENT(iomap_dio_invalidate_fail);
 
 #define IOMAP_TYPE_STRINGS \
 	{ IOMAP_HOLE,		"HOLE" }, \
fs/xfs/xfs_file.c +2 -2
index 00db81eac80d6c..551cca39fa3ba6 100644
@@ -553,8 +553,8 @@ xfs_file_dio_aio_write(
 	xfs_iunlock(ip, iolock);
 
 	/*
-	 * No fallback to buffered IO on errors for XFS, direct IO will either
-	 * complete fully or fail.
+	 * No partial fallback to buffered IO on errors for XFS, direct IO will
+	 * either complete fully or fail.
 	 */
 	ASSERT(ret < 0 || ret == count);
 	return ret;
fs/zonefs/super.c +5 -2
index 07bc42d62673ce..793850454b752f 100644
@@ -786,8 +786,11 @@ static ssize_t zonefs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
 	if (iocb->ki_pos >= ZONEFS_I(inode)->i_max_size)
 		return -EFBIG;
 
-	if (iocb->ki_flags & IOCB_DIRECT)
-		return zonefs_file_dio_write(iocb, from);
+	if (iocb->ki_flags & IOCB_DIRECT) {
+		ret = zonefs_file_dio_write(iocb, from);
+		if (ret != -EREMCHG)
+			return ret;
+	}
 
 	return zonefs_file_buffered_write(iocb, from);
 }
Signature · 1 line
2.26.2
Signature · 1 line
Goldwyn
#6 lore raw

Re: [PATCH 2/2] iomap: fall back to buffered writes for invalidation failures

Quoted 11 lines · Goldwyn Rodrigues
Quoted 5 lines · Christoph Hellwig

Failing to invalid the page cache means data in incoherent, which is a very bad state for the system. Always fall back to buffered I/O through the page cache if we can't invalidate mappings.

Thanks. This will help btrfs. The current next tree contains the iomap changes I recomended and would need to be reverted in order to incorporate this. Once this is in the next tree I will (re)format the btrfs iomap dio patches.

Patches in next don't need to be reverted, if you put together a branch with any iomap or other prep patches + btrfs iomap-dio I'll replace it in the for-next snapshot I push.

#7 lore raw

Re: [PATCH 2/2] iomap: fall back to buffered writes for invalidation failures

In reply to Christoph Hellwig
Quoted 59 lines · Christoph Hellwig

Failing to invalid the page cache means data in incoherent, which is a very bad state for the system. Always fall back to buffered I/O through the page cache if we can't invalidate mappings.

---
 fs/ext4/file.c       |  2 ++
 fs/gfs2/file.c       |  3 ++-
 fs/iomap/direct-io.c | 13 ++++++++-----
 fs/iomap/trace.h     |  1 +
 fs/xfs/xfs_file.c    |  4 ++--
 fs/zonefs/super.c    |  7 +++++--
 6 files changed, 20 insertions(+), 10 deletions(-)
fs/ext4/file.c +2 -0
index 2a01e31a032c4c..0da6c2a2c32c1e 100644
@@ -544,6 +544,8 @@ static ssize_t ext4_dio_write_iter(struct kiocb *iocb, struct iov_iter *from)
 		iomap_ops = &ext4_iomap_overwrite_ops;
 	ret = iomap_dio_rw(iocb, from, iomap_ops, &ext4_dio_write_ops,
 			   is_sync_kiocb(iocb) || unaligned_io || extend);
+	if (ret == -EREMCHG)
+		ret = 0;
 
 	if (extend)
 		ret = ext4_handle_inode_extension(inode, offset, ret, count);
fs/gfs2/file.c +2 -1
index fe305e4bfd3734..c7907d40c61d17 100644
@@ -814,7 +814,8 @@ static ssize_t gfs2_file_direct_write(struct kiocb *iocb, struct iov_iter *from)
 
 	ret = iomap_dio_rw(iocb, from, &gfs2_iomap_ops, NULL,
 			   is_sync_kiocb(iocb));
-
+	if (ret == -EREMCHG)
+		ret = 0;
 out:
 	gfs2_glock_dq(&gh);
 out_uninit:
fs/iomap/direct-io.c +1 -2
index 190967e87b69e4..62626235cdbe8d 100644
@@ -10,6 +10,7 @@
 #include <linux/backing-dev.h>
 #include <linux/uio.h>
 #include <linux/task_io_accounting_ops.h>
+#include "trace.h"
 
 #include "../internal.h"
 
@@ -478,13 +479,15 @@ iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 	if (iov_iter_rw(iter) == WRITE) {
 		/*
 		 * Try to invalidate cache pages for the range we are writing.
-		 * If this invalidation fails, tough, the write will still work,
-		 * but racing two incompatible write paths is a pretty crazy

fs/iomap/direct-io.c iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, · line 488

Quoted 10 lines
fs/iomap/direct-io.c +5 -3
-		 * thing to do, so we don't support it 100%.
+		 * If this invalidation fails, let the caller fall back to
+		 * buffered I/O.
 		 */
 		if (invalidate_inode_pages2_range(mapping, pos >> PAGE_SHIFT,
-				end >> PAGE_SHIFT))
-			dio_warn_stale_pagecache(iocb->ki_filp);
+				end >> PAGE_SHIFT)) {
+			trace_iomap_dio_invalidate_fail(inode, pos, count);
+			ret = -EREMCHG;

-ENOTBLK is already being used as a "magic" return code that means "retry this direct write as a buffered write". Shouldn't we use that instead?

-EREMCHG was a private hack we put in XFS for the one case where a direct write had to be done through the page cache (non block-aligned COW), but maybe it's time we put that to rest since the rest of the world apparently thinks the magic fallback code is -ENOTBLK.

--D

Quoted 53 lines
fs/iomap/direct-io.c +2 -0
+			goto out_free_dio;
+		}
 
 		if (!wait_for_completion && !inode->i_sb->s_dio_done_wq) {
 			ret = sb_init_dio_done_wq(inode->i_sb);
fs/iomap/trace.h +1 -0
index 5693a39d52fb63..fdc7ae388476f5 100644
@@ -74,6 +74,7 @@ DEFINE_EVENT(iomap_range_class, name,	\
 DEFINE_RANGE_EVENT(iomap_writepage);
 DEFINE_RANGE_EVENT(iomap_releasepage);
 DEFINE_RANGE_EVENT(iomap_invalidatepage);
+DEFINE_RANGE_EVENT(iomap_dio_invalidate_fail);
 
 #define IOMAP_TYPE_STRINGS \
 	{ IOMAP_HOLE,		"HOLE" }, \
fs/xfs/xfs_file.c +2 -2
index 00db81eac80d6c..551cca39fa3ba6 100644
@@ -553,8 +553,8 @@ xfs_file_dio_aio_write(
 	xfs_iunlock(ip, iolock);
 
 	/*
-	 * No fallback to buffered IO on errors for XFS, direct IO will either
-	 * complete fully or fail.
+	 * No partial fallback to buffered IO on errors for XFS, direct IO will
+	 * either complete fully or fail.
 	 */
 	ASSERT(ret < 0 || ret == count);
 	return ret;
fs/zonefs/super.c +5 -2
index 07bc42d62673ce..793850454b752f 100644
@@ -786,8 +786,11 @@ static ssize_t zonefs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
 	if (iocb->ki_pos >= ZONEFS_I(inode)->i_max_size)
 		return -EFBIG;
 
-	if (iocb->ki_flags & IOCB_DIRECT)
-		return zonefs_file_dio_write(iocb, from);
+	if (iocb->ki_flags & IOCB_DIRECT) {
+		ret = zonefs_file_dio_write(iocb, from);
+		if (ret != -EREMCHG)
+			return ret;
+	}
 
 	return zonefs_file_buffered_write(iocb, from);
 }
Signature · 1 line
2.26.2
#8 lore raw

Re: [PATCH 2/2] iomap: fall back to buffered writes for invalidation failures

Quoted 8 lines · Darrick J. Wong

-ENOTBLK is already being used as a "magic" return code that means "retry this direct write as a buffered write". Shouldn't we use that instead?

-EREMCHG was a private hack we put in XFS for the one case where a direct write had to be done through the page cache (non block-aligned COW), but maybe it's time we put that to rest since the rest of the world apparently thinks the magic fallback code is -ENOTBLK.

Sure, I can switch the error code.

#9 lore raw

Re: [PATCH 2/2] iomap: fall back to buffered writes for invalidation failures

In reply to Christoph Hellwig
Quoted 1 line · Christoph Hellwig

Failing to invalid the page cache means data in incoherent, which is

s/in incoherent/is incoherent

Quoted 58 lines

a very bad state for the system. Always fall back to buffered I/O through the page cache if we can't invalidate mappings.

---
 fs/ext4/file.c       |  2 ++
 fs/gfs2/file.c       |  3 ++-
 fs/iomap/direct-io.c | 13 ++++++++-----
 fs/iomap/trace.h     |  1 +
 fs/xfs/xfs_file.c    |  4 ++--
 fs/zonefs/super.c    |  7 +++++--
 6 files changed, 20 insertions(+), 10 deletions(-)
fs/ext4/file.c +2 -0
index 2a01e31a032c4c..0da6c2a2c32c1e 100644
@@ -544,6 +544,8 @@ static ssize_t ext4_dio_write_iter(struct kiocb *iocb, struct iov_iter *from)
 		iomap_ops = &ext4_iomap_overwrite_ops;
 	ret = iomap_dio_rw(iocb, from, iomap_ops, &ext4_dio_write_ops,
 			   is_sync_kiocb(iocb) || unaligned_io || extend);
+	if (ret == -EREMCHG)
+		ret = 0;
 
 	if (extend)
 		ret = ext4_handle_inode_extension(inode, offset, ret, count);
fs/gfs2/file.c +2 -1
index fe305e4bfd3734..c7907d40c61d17 100644
@@ -814,7 +814,8 @@ static ssize_t gfs2_file_direct_write(struct kiocb *iocb, struct iov_iter *from)
 
 	ret = iomap_dio_rw(iocb, from, &gfs2_iomap_ops, NULL,
 			   is_sync_kiocb(iocb));
-
+	if (ret == -EREMCHG)
+		ret = 0;
 out:
 	gfs2_glock_dq(&gh);
 out_uninit:
fs/iomap/direct-io.c +1 -2
index 190967e87b69e4..62626235cdbe8d 100644
@@ -10,6 +10,7 @@
 #include <linux/backing-dev.h>
 #include <linux/uio.h>
 #include <linux/task_io_accounting_ops.h>
+#include "trace.h"
 
 #include "../internal.h"
 
@@ -478,13 +479,15 @@ iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 	if (iov_iter_rw(iter) == WRITE) {
 		/*
 		 * Try to invalidate cache pages for the range we are writing.
-		 * If this invalidation fails, tough, the write will still work,
-		 * but racing two incompatible write paths is a pretty crazy

fs/iomap/direct-io.c iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, · line 488

Quoted 10 lines
fs/iomap/direct-io.c +5 -3
-		 * thing to do, so we don't support it 100%.
+		 * If this invalidation fails, let the caller fall back to
+		 * buffered I/O.
 		 */
 		if (invalidate_inode_pages2_range(mapping, pos >> PAGE_SHIFT,
-				end >> PAGE_SHIFT))
-			dio_warn_stale_pagecache(iocb->ki_filp);
+				end >> PAGE_SHIFT)) {
+			trace_iomap_dio_invalidate_fail(inode, pos, count);
+			ret = -EREMCHG;

I am wondering if it is OK to unconditionally always return -EREMCHG here. Shouldn't this depend on the return code of invalidate_inode_pages2_range() ? ret may be the value returned by mapping->a_ops->launder_page(page) instead of -EBUSY that invalidate_inode_pages2_range() otherwise returns for a failed invalidation. Isn't their any error condition that would be better served by not forcing the fallback to buffered write ? E.g. -ENOMEM ?

Quoted 37 lines
fs/iomap/direct-io.c +2 -0
+			goto out_free_dio;
+		}
 
 		if (!wait_for_completion && !inode->i_sb->s_dio_done_wq) {
 			ret = sb_init_dio_done_wq(inode->i_sb);
fs/iomap/trace.h +1 -0
index 5693a39d52fb63..fdc7ae388476f5 100644
@@ -74,6 +74,7 @@ DEFINE_EVENT(iomap_range_class, name,	\
 DEFINE_RANGE_EVENT(iomap_writepage);
 DEFINE_RANGE_EVENT(iomap_releasepage);
 DEFINE_RANGE_EVENT(iomap_invalidatepage);
+DEFINE_RANGE_EVENT(iomap_dio_invalidate_fail);
 
 #define IOMAP_TYPE_STRINGS \
 	{ IOMAP_HOLE,		"HOLE" }, \
fs/xfs/xfs_file.c +2 -2
index 00db81eac80d6c..551cca39fa3ba6 100644
@@ -553,8 +553,8 @@ xfs_file_dio_aio_write(
 	xfs_iunlock(ip, iolock);
 
 	/*
-	 * No fallback to buffered IO on errors for XFS, direct IO will either
-	 * complete fully or fail.
+	 * No partial fallback to buffered IO on errors for XFS, direct IO will
+	 * either complete fully or fail.
 	 */
 	ASSERT(ret < 0 || ret == count);
 	return ret;
fs/zonefs/super.c +0 -0
index 07bc42d62673ce..793850454b752f 100644

fs/zonefs/super.c static ssize_t zonefs_file_write_iter(struct kiocb *iocb, struct iov_iter *from) · line 793

Quoted 10 lines
fs/zonefs/super.c +5 -2
@@ -786,8 +786,11 @@ static ssize_t zonefs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
 	if (iocb->ki_pos >= ZONEFS_I(inode)->i_max_size)
 		return -EFBIG;
 
-	if (iocb->ki_flags & IOCB_DIRECT)
-		return zonefs_file_dio_write(iocb, from);
+	if (iocb->ki_flags & IOCB_DIRECT) {
+		ret = zonefs_file_dio_write(iocb, from);
+		if (ret != -EREMCHG)
+			return ret;
+	}

This looks fine. This would happen only for conventional zone writes since sequential zone writes cannot ever issue direct IOs on top of cached data as that would be a forbidden "overwrite" operation.

Quoted 4 lines
fs/zonefs/super.c +0 -0
 	return zonefs_file_buffered_write(iocb, from);
 }
 
Signature · 2 lines
Damien Le Moal
Western Digital Research
Acked-by
#10 lore raw
In reply to Christoph Hellwig
Quoted 7 lines · Christoph Hellwig

Hi all,

this series has two parts: the first one picks up Dave's patch to avoid invalidation entierly for reads, picked up deep down from the btrfs iomap thread. The second one falls back to buffered writes if invalidation fails instead of leaving a stale cache around. Let me know what you think about this approch.

Either we maintain application level concurrency for direct IOs and ignore the stale data in the page cache, or we kill application IO concurrency and keep the page cache coherent.

It's a lose-lose choice and I'm on the fence as to which is the lesser of two evils.

The main factor is whether the buffered IO fallback can be diagnosed. There's a new tracepoint for that case, so at least we will be able to tell if the fallback co-incides with application performance cratering. Hopefully this will only be a rare event.

So, to hoist myself on my own petard: correctness first, performance second.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
david at fromorbit.com
#11 lore raw
In reply to Christoph Hellwig

Hi Christoph,

Quoted 7 lines · Christoph Hellwig

Hi all,

this series has two parts: the first one picks up Dave's patch to avoid invalidation entierly for reads, picked up deep down from the btrfs iomap thread. The second one falls back to buffered writes if invalidation fails instead of leaving a stale cache around. Let me know what you think about this approch.

Which kernel version are these changes expected?
btrfs dio switch to iomap depends on this.

Signature · 1 line
Goldwyn
#12 lore raw
Quoted 5 lines · Goldwyn Rodrigues

Hi Christoph,

Quoted 2 lines · Christoph Hellwig

Hi all,

Quoted 8 lines
Quoted 5 lines

this series has two parts: the first one picks up Dave's patch to avoid invalidation entierly for reads, picked up deep down from the btrfs iomap thread. The second one falls back to buffered writes if invalidation fails instead of leaving a stale cache around. Let me know what you think about this approch.

Which kernel version are these changes expected?
btrfs dio switch to iomap depends on this.

No idea. Darrick, are you ok with picking this up soon so that Goldwyn can pull it in?

#13 lore raw
Quoted 12 lines · Christoph Hellwig
Quoted 11 lines · Goldwyn Rodrigues

Hi Christoph,

Quoted 7 lines · Christoph Hellwig

Hi all,

this series has two parts: the first one picks up Dave's patch to avoid invalidation entierly for reads, picked up deep down from the btrfs iomap thread. The second one falls back to buffered writes if invalidation fails instead of leaving a stale cache around. Let me know what you think about this approch.

Quoted 5 lines
Quoted 2 lines

Which kernel version are these changes expected?
btrfs dio switch to iomap depends on this.

No idea. Darrick, are you ok with picking this up soon so that Goldwyn can pull it in?

Yes; I was nearly about to ping you to see if you were going to have time to push out a corrected patch 2 in time for ... 5.9? 5.10?

--D

#14 lore raw
In reply to Christoph Hellwig
Quoted 12 lines · Christoph Hellwig
Quoted 11 lines · Goldwyn Rodrigues

Hi Christoph,

Quoted 7 lines · Christoph Hellwig

Hi all,

this series has two parts: the first one picks up Dave's patch to avoid invalidation entierly for reads, picked up deep down from the btrfs iomap thread. The second one falls back to buffered writes if invalidation fails instead of leaving a stale cache around. Let me know what you think about this approch.

Quoted 5 lines
Quoted 2 lines

Which kernel version are these changes expected?
btrfs dio switch to iomap depends on this.

No idea. Darrick, are you ok with picking this up soon so that Goldwyn can pull it in?

I thought you were going to respin this with EREMCHG changed to ENOTBLK?

#15 lore raw
Quoted 13 lines · Matthew Wilcox
Quoted 12 lines · Christoph Hellwig
Quoted 11 lines · Goldwyn Rodrigues

Hi Christoph,

Quoted 7 lines · Christoph Hellwig

Hi all,

this series has two parts: the first one picks up Dave's patch to avoid invalidation entierly for reads, picked up deep down from the btrfs iomap thread. The second one falls back to buffered writes if invalidation fails instead of leaving a stale cache around. Let me know what you think about this approch.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Which kernel version are these changes expected?
btrfs dio switch to iomap depends on this.

No idea. Darrick, are you ok with picking this up soon so that Goldwyn can pull it in?

I thought you were going to respin this with EREMCHG changed to ENOTBLK?

Oh, true. I'll do that ASAP.

#16 lore raw
Quoted 4 lines · Christoph Hellwig
Quoted 1 line · Matthew Wilcox

I thought you were going to respin this with EREMCHG changed to ENOTBLK?

Oh, true. I'll do that ASAP.

Michael, could we add this to manpages?

write.2 renamed from write.2.old +6 -0
@@ -192,6 +192,12 @@
 .IR count ,
 or the file offset is not suitably aligned.
 .TP
+.B ENOTBLK
+The file was opened with the
+.B O_DIRECT
+flag, and the I/O could not be completed due to another process using
+the file.
+.TP
 .B EIO
 A low-level I/O error occurred while modifying the inode.
 This error may relate to the write-back of data written by an earlier
#17 lore raw
Quoted 7 lines · Matthew Wilcox
Quoted 4 lines · Christoph Hellwig
Quoted 1 line · Matthew Wilcox

I thought you were going to respin this with EREMCHG changed to ENOTBLK?

Oh, true. I'll do that ASAP.

Michael, could we add this to manpages?

Umm, no. -ENOTBLK is internal - the file systems will retry using buffered I/O and the error shall never escape to userspace (or even the VFS for that matter).

#18 lore raw
Quoted 5 lines · Christoph Hellwig
Quoted 4 lines · Matthew Wilcox
Quoted 3 lines · Christoph Hellwig
Quoted 1 line · Matthew Wilcox

I thought you were going to respin this with EREMCHG changed to ENOTBLK?

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Oh, true. I'll do that ASAP.

Michael, could we add this to manpages?

Umm, no. -ENOTBLK is internal - the file systems will retry using buffered I/O and the error shall never escape to userspace (or even the VFS for that matter).

It's worth dropping a comment somewhere that ENOTBLK is the desired "fall back to buffered" errcode, seeing as Dave and I missed that in XFS...

--D

#19 lore raw
Quoted 10 lines · Darrick J. Wong
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Matthew Wilcox
Quoted 4 lines · Christoph Hellwig
Quoted 1 line · Matthew Wilcox

I thought you were going to respin this with EREMCHG changed to ENOTBLK?

Oh, true. I'll do that ASAP.

Michael, could we add this to manpages?

Quoted 7 lines
Quoted 3 lines

Umm, no. -ENOTBLK is internal - the file systems will retry using buffered I/O and the error shall never escape to userspace (or even the VFS for that matter).

It's worth dropping a comment somewhere that ENOTBLK is the desired "fall back to buffered" errcode, seeing as Dave and I missed that in XFS...

Sounds like a good idea, but what would a good place be?

#20 lore raw
Quoted 15 lines · Christoph Hellwig
Quoted 14 lines · Darrick J. Wong
Quoted 12 lines · Christoph Hellwig
Quoted 7 lines · Matthew Wilcox
Quoted 4 lines · Christoph Hellwig
Quoted 1 line · Matthew Wilcox

I thought you were going to respin this with EREMCHG changed to ENOTBLK?

Oh, true. I'll do that ASAP.

Michael, could we add this to manpages?

Umm, no. -ENOTBLK is internal - the file systems will retry using buffered I/O and the error shall never escape to userspace (or even the VFS for that matter).

Quoted 5 lines
Quoted 3 lines

It's worth dropping a comment somewhere that ENOTBLK is the desired "fall back to buffered" errcode, seeing as Dave and I missed that in XFS...

Sounds like a good idea, but what would a good place be?

In the comment that precedes iomap_dio_rw() for the iomap version, and...

...ye $deity, the old direct-io.c file is a mess of wrappers. Uh... a new comment preceding __blockdev_direct_IO? Or blockdev_direct_IO? Or both?

Or I guess the direct_IO documentation in vfs.rst...?

``direct_IO``
called by the generic read/write routines to perform direct_IO - that is IO requests which bypass the page cache and transfer data directly between the storage and the application's address space. This function can return -ENOTBLK to signal that it is necessary to fallback to buffered IO. Note that blockdev_direct_IO and variants can also return -ENOTBLK.

--D

#21 lore raw
Quoted 1 line · Darrick J. Wong

In the comment that precedes iomap_dio_rw() for the iomap version,

maybe let's just do that..

Quoted 7 lines

``direct_IO``
called by the generic read/write routines to perform direct_IO - that is IO requests which bypass the page cache and transfer data directly between the storage and the application's address space. This function can return -ENOTBLK to signal that it is necessary to fallback to buffered IO. Note that blockdev_direct_IO and variants can also return -ENOTBLK.

->direct_IO is not used for iomap and various other implementations. In fact it is a horrible hack that I've been trying to get rid of for a while.

#22 lore raw
Quoted 13 lines · Christoph Hellwig
Quoted 1 line · Darrick J. Wong

In the comment that precedes iomap_dio_rw() for the iomap version,

maybe let's just do that..

Quoted 7 lines

``direct_IO``
called by the generic read/write routines to perform direct_IO - that is IO requests which bypass the page cache and transfer data directly between the storage and the application's address space. This function can return -ENOTBLK to signal that it is necessary to fallback to buffered IO. Note that blockdev_direct_IO and variants can also return -ENOTBLK.

Quoted 3 lines

->direct_IO is not used for iomap and various other implementations. In fact it is a horrible hack that I've been trying to get rid of for a while.

Agreed, but for now there are still a number of fses who are still on the old directio code; let's try to keep the drainbamage/confusion potential to a minimum so it doesn't spread to our iomap shinyness. :)

--D

#23 lore raw
In reply to Christoph Hellwig
Quoted 5 lines · Christoph Hellwig
Quoted 4 lines · Matthew Wilcox
Quoted 3 lines · Christoph Hellwig
Quoted 1 line · Matthew Wilcox

I thought you were going to respin this with EREMCHG changed to ENOTBLK?

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Oh, true. I'll do that ASAP.

Michael, could we add this to manpages?

Umm, no. -ENOTBLK is internal - the file systems will retry using buffered I/O and the error shall never escape to userspace (or even the VFS for that matter).

Ah, I made the mistake of believing the comments that I could see in your patch instead of reading the code.

Can I suggest deleting this comment:

/*
* No fallback to buffered IO on errors for XFS, direct IO will either
* complete fully or fail.
*/

and rewording this one:

/*
* Allow a directio write to fall back to a buffered
* write *only* in the case that we're doing a reflink
* CoW. In all other directio scenarios we do not
* allow an operation to fall back to buffered mode. */

as part of your revised patchset?

#24 lore raw
Quoted 16 lines · Matthew Wilcox
Quoted 3 lines

Umm, no. -ENOTBLK is internal - the file systems will retry using buffered I/O and the error shall never escape to userspace (or even the VFS for that matter).

Ah, I made the mistake of believing the comments that I could see in your patch instead of reading the code.

Can I suggest deleting this comment:

/*
* No fallback to buffered IO on errors for XFS, direct IO will either
* complete fully or fail.
*/

and rewording this one:

Quoted 8 lines

/*
* Allow a directio write to fall back to a buffered
* write *only* in the case that we're doing a reflink
* CoW. In all other directio scenarios we do not
* allow an operation to fall back to buffered mode. */

as part of your revised patchset?

That isn't actually true. In current mainline we only fallback on reflink RMW cases, but with this series we also fall back for invalidation failures.

#25 lore raw
Quoted 24 lines · Christoph Hellwig
Quoted 23 lines · Matthew Wilcox
Quoted 3 lines

Umm, no. -ENOTBLK is internal - the file systems will retry using buffered I/O and the error shall never escape to userspace (or even the VFS for that matter).

Ah, I made the mistake of believing the comments that I could see in your patch instead of reading the code.

Can I suggest deleting this comment:

/*
* No fallback to buffered IO on errors for XFS, direct IO will either
* complete fully or fail.
*/

and rewording this one:

/*
* Allow a directio write to fall back to a buffered
* write *only* in the case that we're doing a reflink
* CoW. In all other directio scenarios we do not
* allow an operation to fall back to buffered mode. */

Quoted 5 lines
Quoted 1 line

as part of your revised patchset?

That isn't actually true. In current mainline we only fallback on reflink RMW cases, but with this series we also fall back for invalidation failures.

... that's why I'm suggesting that you delete the first one and rewrite the second one. Because they aren't true.

#26 lore raw
Quoted 25 lines · Matthew Wilcox
Quoted 24 lines · Christoph Hellwig
Quoted 23 lines · Matthew Wilcox
Quoted 3 lines

Umm, no. -ENOTBLK is internal - the file systems will retry using buffered I/O and the error shall never escape to userspace (or even the VFS for that matter).

Ah, I made the mistake of believing the comments that I could see in your patch instead of reading the code.

Can I suggest deleting this comment:

/*
* No fallback to buffered IO on errors for XFS, direct IO will either
* complete fully or fail.
*/

and rewording this one:

/*
* Allow a directio write to fall back to a buffered
* write *only* in the case that we're doing a reflink
* CoW. In all other directio scenarios we do not
* allow an operation to fall back to buffered mode. */

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

as part of your revised patchset?

That isn't actually true. In current mainline we only fallback on reflink RMW cases, but with this series we also fall back for invalidation failures.

... that's why I'm suggesting that you delete the first one and rewrite the second one. Because they aren't true.

/*
* We allow only three outcomes of a directio: (1) it succeeds
* completely; (2) it fails with a negative error code; or (3) it
* returns -ENOTBLK to signal that we should fall back to buffered IO. *
* The third scenario should only happen if iomap refuses to perform the
* direct IO, or the direct write request requires CoW but is not aligned
* to the filesystem block size.
*/

?

--D