Uncached buffered IO

Add a page flag that file IO can use to indicate that the IO being done is uncached, as in it should not persist in the page cache after the IO has been completed.

Quoted 3 lines · Jens Axboe

Add a page flag that file IO can use to indicate that the IO being done is uncached, as in it should not persist in the page cache after the IO has been completed.

I have not found a way to avoid using a new bit. I am unsure if we have enough bits on 32-bit systems with all possible features enabled.

In the worst-case scenario, we may need to make the feature 64-bit only. I believe it should be acceptable as long as userspace is prepared for the possibility that RWF_UNCACHED may fail. It is not going to be supported by all filesystems anyway.

If the folio is marked as uncached, drop pages when writeback completes. Intended to be used with RWF_UNCACHED, to avoid needing sync writes for uncached IO.

Signed-off-by Jens Axboe <[email protected]>

---
 mm/filemap.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

mm/filemap.c +23 -0

index 3d0614ea5f59..40debe742abe 100644
@@ -1600,6 +1600,27 @@ int folio_wait_private_2_killable(struct folio *folio)
 }
 EXPORT_SYMBOL(folio_wait_private_2_killable);
 
+/*
+ * If folio was marked as uncached, then pages should be dropped when writeback
+ * completes. Do that now. If we fail, it's likely because of a big folio -
+ * just reset uncached for that case and latter completions should invalidate.
+ */
+static void folio_end_uncached(struct folio *folio)
+{
+	/*
+	 * Hitting !in_task() should not happen off RWF_UNCACHED writeback, but
+	 * can happen if normal writeback just happens to find dirty folios
+	 * that were created as part of uncached writeback, and that writeback
+	 * would otherwise not need non-IRQ handling. Just skip the
+	 * invalidation in that case.
+	 */
+	if (in_task() && folio_trylock(folio)) {
+		if (folio->mapping)
+			folio_unmap_invalidate(folio->mapping, folio, 0);
+		folio_unlock(folio);
+	}
+}
+
 /**
  * folio_end_writeback - End writeback against a folio.
  * @folio: The folio.

@@ -1610,6 +1631,8 @@ EXPORT_SYMBOL(folio_wait_private_2_killable);
  */
 void folio_end_writeback(struct folio *folio)
 {
+	bool folio_uncached = false;
+
 	VM_BUG_ON_FOLIO(!folio_test_writeback(folio), folio);
 
 	/*

mm/filemap.c +3 -0

 	if (__folio_end_writeback(folio))
 		folio_wake_bit(folio, PG_writeback);
 	acct_reclaim_writeback(folio);
+
+	if (folio_uncached)
+		folio_end_uncached(folio);
 	folio_put(folio);
 }
 EXPORT_SYMBOL(folio_end_writeback);

Signature · 1 line

2.45.2

Quoted 51 lines · Jens Axboe

If the folio is marked as uncached, drop pages when writeback completes. Intended to be used with RWF_UNCACHED, to avoid needing sync writes for uncached IO.

Signed-off-by Jens Axboe <[email protected]>

---
 mm/filemap.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

mm/filemap.c +23 -0

index 3d0614ea5f59..40debe742abe 100644
@@ -1600,6 +1600,27 @@ int folio_wait_private_2_killable(struct folio *folio)
 }
 EXPORT_SYMBOL(folio_wait_private_2_killable);
 
+/*
+ * If folio was marked as uncached, then pages should be dropped when writeback
+ * completes. Do that now. If we fail, it's likely because of a big folio -
+ * just reset uncached for that case and latter completions should invalidate.
+ */
+static void folio_end_uncached(struct folio *folio)
+{
+	/*
+	 * Hitting !in_task() should not happen off RWF_UNCACHED writeback, but
+	 * can happen if normal writeback just happens to find dirty folios
+	 * that were created as part of uncached writeback, and that writeback
+	 * would otherwise not need non-IRQ handling. Just skip the
+	 * invalidation in that case.
+	 */
+	if (in_task() && folio_trylock(folio)) {
+		if (folio->mapping)
+			folio_unmap_invalidate(folio->mapping, folio, 0);
+		folio_unlock(folio);
+	}
+}
+
 /**
  * folio_end_writeback - End writeback against a folio.
  * @folio: The folio.

@@ -1610,6 +1631,8 @@ EXPORT_SYMBOL(folio_wait_private_2_killable);
  */
 void folio_end_writeback(struct folio *folio)
 {
+	bool folio_uncached = false;
+
 	VM_BUG_ON_FOLIO(!folio_test_writeback(folio), folio);
 
 	/*

mm/filemap.c void folio_end_writeback(struct folio *folio) · line 1658

Quoted 5 lines

mm/filemap.c +2 -0

@@ -1631,9 +1654,14 @@ void folio_end_writeback(struct folio *folio)
 	 * reused before the folio_wake_bit().
 	 */
 	folio_get(folio);
+	if (folio_test_uncached(folio) && folio_test_clear_uncached(folio))
+		folio_uncached = true;

Hm? Maybe

	folio_uncached = folio_test_clear_uncached(folio);

?

If RWF_UNCACHED is set for a write, mark new folios being written with uncached. This is done by passing in the fact that it's an uncached write through the folio pointer. We can only get there when IOCB_UNCACHED was allowed, which can only happen if the file system opts in. Opting in means they need to check for the LSB in the folio pointer to know if it's an uncached write or not. If it is, then FGP_UNCACHED should be used if creating new folios is necessary.

Uncached writes will drop any folios they create upon writeback completion, but leave folios that may exist in that range alone. Since ->write_begin() doesn't currently take any flags, and to avoid needing to change the callback kernel wide, use the foliop being passed in to ->write_begin() to signal if this is an uncached write or not. File systems can then use that to mark newly created folios as uncached.

Add a helper, generic_uncached_write(), that generic_file_write_iter() calls upon successful completion of an uncached write.

include/linux/pagemap.h +9 -0

@@ -70,6 +71,34 @@ static inline int filemap_write_and_wait(struct address_space *mapping)
 	return filemap_write_and_wait_range(mapping, 0, LLONG_MAX);
 }
 
+/*
+ * generic_uncached_write - start uncached writeback
+ * @iocb: the iocb that was written
+ * @written: the amount of bytes written
+ *
+ * When writeback has been handled by write_iter, this helper should be called
+ * if the file system supports uncached writes. If %IOCB_UNCACHED is set, it
+ * will kick off writeback for the specified range.
+ */

include/linux/pagemap.h +9 -0

+
+/*
+ * Value passed in to ->write_begin() if IOCB_UNCACHED is set for the write,
+ * and the ->write_begin() handler on a file system supporting FOP_UNCACHED
+ * must check for this and pass FGP_UNCACHED for folio creation.
+ */
+#define foliop_uncached			((struct folio *) 0xfee1c001)
+#define foliop_is_uncached(foliop)	(*(foliop) == foliop_uncached)
+
 /**
  * filemap_set_wb_err - set a writeback error on an address_space
  * @mapping: mapping in which to set writeback error

mm/filemap.c +12 -1

index 40debe742abe..0d312de4e20c 100644
@@ -430,6 +430,7 @@ int __filemap_fdatawrite_range(struct address_space *mapping, loff_t start,
 
 	return filemap_fdatawrite_wbc(mapping, &wbc);
 }
+EXPORT_SYMBOL_GPL(__filemap_fdatawrite_range);
 
 static inline int __filemap_fdatawrite(struct address_space *mapping,
 	int sync_mode)

@@ -4076,7 +4077,7 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i)
 	ssize_t written = 0;
 
 	do {
-		struct folio *folio;
+		struct folio *folio = NULL;
 		size_t offset;		/* Offset into folio */
 		size_t bytes;		/* Bytes to write to folio */
 		size_t copied;		/* Bytes copied from user */

@@ -4104,6 +4105,16 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i)
 			break;
 		}
 
+		/*
+		 * If IOCB_UNCACHED is set here, we now the file system
+		 * supports it. And hence it'll know to check folip for being
+		 * set to this magic value. If so, it's an uncached write.
+		 * Whenever ->write_begin() changes prototypes again, this
+		 * can go away and just pass iocb or iocb flags.
+		 */
+		if (iocb->ki_flags & IOCB_UNCACHED)
+			folio = foliop_uncached;
+
 		status = a_ops->write_begin(file, mapping, pos, bytes,
 						&folio, &fsdata);
 		if (unlikely(status < 0))

Quoted 15 lines · Jens Axboe

If RWF_UNCACHED is set for a write, mark new folios being written with uncached. This is done by passing in the fact that it's an uncached write through the folio pointer. We can only get there when IOCB_UNCACHED was allowed, which can only happen if the file system opts in. Opting in means they need to check for the LSB in the folio pointer to know if it's an uncached write or not. If it is, then FGP_UNCACHED should be used if creating new folios is necessary.

Uncached writes will drop any folios they create upon writeback completion, but leave folios that may exist in that range alone. Since ->write_begin() doesn't currently take any flags, and to avoid needing to change the callback kernel wide, use the foliop being passed in to ->write_begin() to signal if this is an uncached write or not. File systems can then use that to mark newly created folios as uncached.

Quoted 2 lines

Add a helper, generic_uncached_write(), that generic_file_write_iter() calls upon successful completion of an uncached write.

This doesn't implement an "uncached" write operation. This implements a cache write-through operation.

We've actually been talking about this for some time as a desirable general buffered write trait on fast SSDs. Excessive write-behind caching is a real problem in general, especially when doing streaming sequential writes to pcie 4 and 5 nvme SSDs that can do more than 7GB/s to disk. When the page cache fills up, we see all

the same problems you are trying to work around in this series with "uncached" writes.

IOWS, what we really want is page cache write-through as an automatic feature for buffered writes.

Quoted 17 lines

include/linux/pagemap.h +13 -0

@@ -70,6 +71,34 @@ static inline int filemap_write_and_wait(struct address_space *mapping)
 	return filemap_write_and_wait_range(mapping, 0, LLONG_MAX);
 }
 
+/*
+ * generic_uncached_write - start uncached writeback
+ * @iocb: the iocb that was written
+ * @written: the amount of bytes written
+ *
+ * When writeback has been handled by write_iter, this helper should be called
+ * if the file system supports uncached writes. If %IOCB_UNCACHED is set, it
+ * will kick off writeback for the specified range.
+ */
+static inline void generic_uncached_write(struct kiocb *iocb, ssize_t written)
+{
+	if (iocb->ki_flags & IOCB_UNCACHED) {
+		struct address_space *mapping = iocb->ki_filp->f_mapping;

include/linux/pagemap.h static inline int filemap_write_and_wait(struct address_space *mapping) · line 92

Quoted 6 lines

include/linux/pagemap.h +6 -0

+
+		/* kick off uncached writeback */
+		__filemap_fdatawrite_range(mapping, iocb->ki_pos,
+					   iocb->ki_pos + written, WB_SYNC_NONE);
+	}
+}

Yup, this is basically write-through.

Quoted 49 lines

include/linux/pagemap.h +7 -0

+ * Value passed in to ->write_begin() if IOCB_UNCACHED is set for the write,
+ * and the ->write_begin() handler on a file system supporting FOP_UNCACHED
+ * must check for this and pass FGP_UNCACHED for folio creation.
+ */
+#define foliop_uncached			((struct folio *) 0xfee1c001)
+#define foliop_is_uncached(foliop)	(*(foliop) == foliop_uncached)
+
 /**
  * filemap_set_wb_err - set a writeback error on an address_space
  * @mapping: mapping in which to set writeback error

mm/filemap.c +12 -1

index 40debe742abe..0d312de4e20c 100644
@@ -430,6 +430,7 @@ int __filemap_fdatawrite_range(struct address_space *mapping, loff_t start,
 
 	return filemap_fdatawrite_wbc(mapping, &wbc);
 }
+EXPORT_SYMBOL_GPL(__filemap_fdatawrite_range);
 
 static inline int __filemap_fdatawrite(struct address_space *mapping,
 	int sync_mode)

@@ -4076,7 +4077,7 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i)
 	ssize_t written = 0;
 
 	do {
-		struct folio *folio;
+		struct folio *folio = NULL;
 		size_t offset;		/* Offset into folio */
 		size_t bytes;		/* Bytes to write to folio */
 		size_t copied;		/* Bytes copied from user */

@@ -4104,6 +4105,16 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i)
 			break;
 		}
 
+		/*
+		 * If IOCB_UNCACHED is set here, we now the file system
+		 * supports it. And hence it'll know to check folip for being
+		 * set to this magic value. If so, it's an uncached write.
+		 * Whenever ->write_begin() changes prototypes again, this
+		 * can go away and just pass iocb or iocb flags.
+		 */
+		if (iocb->ki_flags & IOCB_UNCACHED)
+			folio = foliop_uncached;
+
 		status = a_ops->write_begin(file, mapping, pos, bytes,
 						&folio, &fsdata);
 		if (unlikely(status < 0))

mm/filemap.c ssize_t generic_file_write_iter(struct kiocb *iocb, struct iov_iter *from) · line 4250

Quoted 7 lines

mm/filemap.c +2 -1

@@ -4234,8 +4245,10 @@ ssize_t generic_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
 		ret = __generic_file_write_iter(iocb, from);
 	inode_unlock(inode);
 
-	if (ret > 0)
+	if (ret > 0) {
+		generic_uncached_write(iocb, ret);
 		ret = generic_write_sync(iocb, ret);

Why isn't the writethrough check inside generic_write_sync()? Having to add it to every filesystem that supports write-through is unwieldy. If the IO is DSYNC or SYNC, we're going to run WB_SYNC_ALL writeback through the generic_write_sync() path already, so the only time we actually want to run WB_SYNC_NONE write-through here is if the iocb is not marked as dsync.

Hence I think this write-through check should be done conditionally inside generic_write_sync(), not in addition to the writeback generic_write_sync() might need to do...

That also gives us a common place for adding cache write-through trigger logic (think writebehind trigger logic similar to readahead) and this is also a place where we could automatically tag mapping ranges for reclaim on writeback completion....

Quoted 16 lines · Dave Chinner

Quoted 15 lines · Jens Axboe

If RWF_UNCACHED is set for a write, mark new folios being written with uncached. This is done by passing in the fact that it's an uncached write through the folio pointer. We can only get there when IOCB_UNCACHED was allowed, which can only happen if the file system opts in. Opting in means they need to check for the LSB in the folio pointer to know if it's an uncached write or not. If it is, then FGP_UNCACHED should be used if creating new folios is necessary.

Uncached writes will drop any folios they create upon writeback completion, but leave folios that may exist in that range alone. Since ->write_begin() doesn't currently take any flags, and to avoid needing to change the callback kernel wide, use the foliop being passed in to ->write_begin() to signal if this is an uncached write or not. File systems can then use that to mark newly created folios as uncached.

Quoted 5 lines

Quoted 2 lines

Add a helper, generic_uncached_write(), that generic_file_write_iter() calls upon successful completion of an uncached write.

This doesn't implement an "uncached" write operation. This implements a cache write-through operation.

It's uncached in the sense that the range gets pruned on writeback completion.

For write-through, I'd consider that just the fact that it gets kicked off once dirtied rather than wait for writeback to get kicked at some point.

So I'd say write-through is a subset of that.

Quoted 5 lines

the same problems you are trying to work around in this series with "uncached" writes.

IOWS, what we really want is page cache write-through as an automatic feature for buffered writes.

I don't know who "we" is here - what I really want is for the write to get kicked off, but also reclaimed as part of completion. I don't want kswapd to do that, as it's inefficient.

Quoted 4 lines

That also gives us a common place for adding cache write-through trigger logic (think writebehind trigger logic similar to readahead) and this is also a place where we could automatically tag mapping ranges for reclaim on writeback completion....

I appreciate that you seemingly like the concept, but not that you are also seemingly trying to commandeer this to be something else. Unless you like the automatic reclaiming as well, it's not clear to me.

I think the post-IO invalidation that these IOs do is largely irrelevant to how the page cache processes the write. Indeed, from userspace, the functionality in this patchset would be implemented like this:

oneshot_data_write(fd, buf, len, off)

{
	/* write into page cache */
	pwrite(fd, buf, len, off);

	/* force the write through the page cache */
	sync_file_range(fd, off, len, SYNC_FILE_RANGE_WRITE | SYNC_FILE_RANGE_WAIT_AFTER);

	/* Invalidate the single use data in the cache now it is on disk */
	posix_fadvise(fd, off, len, POSIX_FADV_DONTNEED);
}

Allowing the application to control writeback and invalidation granularity is a much more flexible solution to the problem here; when IO is sequential, delayed allocation will be allowed to ensure large contiguous extents are created and that will greatly reduce file fragmentation on XFS, btrfs, bcachefs and ext4. For random writes, it'll submit async IOs in batches...

Given that io_uring already supports sync_file_range() and posix_fadvise(), I'm wondering why we need an new IO API to perform this specific write-through behaviour in a way that is less flexible than what applications can already implement through existing APIs....

Quoted 24 lines · Dave Chinner

I think the post-IO invalidation that these IOs do is largely irrelevant to how the page cache processes the write. Indeed, from userspace, the functionality in this patchset would be implemented like this:

oneshot_data_write(fd, buf, len, off)

{
	/* write into page cache */
	pwrite(fd, buf, len, off);

	/* force the write through the page cache */
	sync_file_range(fd, off, len, SYNC_FILE_RANGE_WRITE | SYNC_FILE_RANGE_WAIT_AFTER);

	/* Invalidate the single use data in the cache now it is on disk */
	posix_fadvise(fd, off, len, POSIX_FADV_DONTNEED);
}

Allowing the application to control writeback and invalidation granularity is a much more flexible solution to the problem here; when IO is sequential, delayed allocation will be allowed to ensure large contiguous extents are created and that will greatly reduce file fragmentation on XFS, btrfs, bcachefs and ext4. For random writes, it'll submit async IOs in batches...

Quoted 5 lines

Given that io_uring already supports sync_file_range() and posix_fadvise(), I'm wondering why we need an new IO API to perform this specific write-through behaviour in a way that is less flexible than what applications can already implement through existing APIs....

Attaching the hint to the IO operation allows kernel to keep the data in page cache if it is there for other reason. You cannot do it with a separate syscall.

Consider a scenario of a nightly backup of the data. The same data is in cache because the actual workload needs it. You don't want backup task to invalidate the data from cache. Your snippet would do that.

Quoted 23 lines · Jens Axboe

Quoted 21 lines · Dave Chinner

Quoted 15 lines · Jens Axboe

If RWF_UNCACHED is set for a write, mark new folios being written with uncached. This is done by passing in the fact that it's an uncached write through the folio pointer. We can only get there when IOCB_UNCACHED was allowed, which can only happen if the file system opts in. Opting in means they need to check for the LSB in the folio pointer to know if it's an uncached write or not. If it is, then FGP_UNCACHED should be used if creating new folios is necessary.

Uncached writes will drop any folios they create upon writeback completion, but leave folios that may exist in that range alone. Since ->write_begin() doesn't currently take any flags, and to avoid needing to change the callback kernel wide, use the foliop being passed in to ->write_begin() to signal if this is an uncached write or not. File systems can then use that to mark newly created folios as uncached.

Quoted 2 lines

Add a helper, generic_uncached_write(), that generic_file_write_iter() calls upon successful completion of an uncached write.

This doesn't implement an "uncached" write operation. This implements a cache write-through operation.

Quoted 2 lines

It's uncached in the sense that the range gets pruned on writeback completion.

That's not the definition of "uncached". Direct IO is, by definition, "uncached" because it bypasses the cache and is not coherent with the contents of the cache.

This IO, however, is moving the data coherently through the cache (both on read and write). The cached folios are transient - i.e. -temporarily resident- in the cache whilst the IO is in progress - but this behaviour does not make it "uncached IO".

Calling it "uncached IO " is simply wrong from any direction I look at it....

Quoted 5 lines

For write-through, I'd consider that just the fact that it gets kicked off once dirtied rather than wait for writeback to get kicked at some point.

So I'd say write-through is a subset of that.

I think the post-IO invalidation that these IOs do is largely irrelevant to how the page cache processes the write. Indeed, from userspace, the functionality in this patchset would be implemented like this:

oneshot_data_write(fd, buf, len, off)

{
	/* write into page cache */
	pwrite(fd, buf, len, off);

	/* force the write through the page cache */
	sync_file_range(fd, off, len, SYNC_FILE_RANGE_WRITE | SYNC_FILE_RANGE_WAIT_AFTER);

	/* Invalidate the single use data in the cache now it is on disk */
	posix_fadvise(fd, off, len, POSIX_FADV_DONTNEED);
}

Allowing the application to control writeback and invalidation granularity is a much more flexible solution to the problem here; when IO is sequential, delayed allocation will be allowed to ensure large contiguous extents are created and that will greatly reduce file fragmentation on XFS, btrfs, bcachefs and ext4. For random writes, it'll submit async IOs in batches...

Given that io_uring already supports sync_file_range() and posix_fadvise(), I'm wondering why we need an new IO API to perform this specific write-through behaviour in a way that is less flexible than what applications can already implement through existing APIs....

Quoted 8 lines

Quoted 4 lines

That also gives us a common place for adding cache write-through trigger logic (think writebehind trigger logic similar to readahead) and this is also a place where we could automatically tag mapping ranges for reclaim on writeback completion....

I appreciate that you seemingly like the concept, but not that you are also seemingly trying to commandeer this to be something else. Unless you like the automatic reclaiming as well, it's not clear to me.

I'm not trying to commandeer anything.

Having thought about it more, I think this new API is unneccesary for custom written applications to perform fine grained control of page cache residency of one-shot data. We already have APIs that allow applications to do exactly what this patchset is doing. rather than choosing to modify whatever benchmark being used to use existing APIs, a choice was made to modify both the applicaiton and the kernel to implement a whole new API....

I think that was the -wrong choice-.

I think this partially because the kernel modifications are don't really help further us towards the goal of transparent mode switching in the page cache.

Read-through should be a mode that the readahead control activates, not be something triggered by a special read() syscall flag. We already have access patterns and fadvise modes guiding this. Write-through should be controlled in a similar way.

And making the data being read and written behave as transient page caceh objects should be done via an existing fadvise mode, too, because the model you have implemented here exactly matches the definition of FADV_NOREUSE:

POSIX_FADV_NOREUSE
The specified data will be accessed only once.

Having a new per-IO flag that effectively collides existing control functionality into a single inflexible API bit doesn't really make a whole lot of sense to me.

IOWs, I'm not questioning whether we need rw-through modes and/or IO-transient residency for page cache based IO - it's been on our radar for a while. I'm more concerned that the chosen API in this patchset is a poor one as it cannot replace any of the existing controls we already have for these sorts of application directed page cache manipulations...

IOCB_UNCACHED IO needs to prune writeback regions on IO completion, and hence need the worker punt that ext4 also does for unwritten extents. Add an io_end flag to manage that.

If foliop is set to foliop_uncached in ext4_write_begin(), then set FGP_UNCACHED so that __filemap_get_folio() will mark newly created folios as uncached. That in turn will make writeback completion drop these ranges from the page cache.

Now that ext4 supports both uncached reads and writes, add the fop_flag FOP_UNCACHED to enable it.

Signed-off-by Jens Axboe <[email protected]>

---
 fs/ext4/ext4.h    |  1 +
 fs/ext4/file.c    |  2 +-
 fs/ext4/inline.c  |  7 ++++++-
 fs/ext4/inode.c   | 18 ++++++++++++++++--
 fs/ext4/page-io.c | 28 ++++++++++++++++------------
 5 files changed, 40 insertions(+), 16 deletions(-)

fs/ext4/inode.c +10 -0

index 54bdd4884fe6..afae3ab64c9e 100644
@@ -1138,6 +1138,7 @@ static int ext4_write_begin(struct file *file, struct address_space *mapping,
 	int ret, needed_blocks;
 	handle_t *handle;
 	int retries = 0;
+	fgf_t fgp_flags;
 	struct folio *folio;
 	pgoff_t index;
 	unsigned from, to;

@@ -1164,6 +1165,15 @@ static int ext4_write_begin(struct file *file, struct address_space *mapping,
 			return 0;
 	}
 
+	/*
+	 * Set FGP_WRITEBEGIN, and FGP_UNCACHED if foliop contains
+	 * foliop_uncached. That's how generic_perform_write() informs us
+	 * that this is an uncached write.
+	 */
+	fgp_flags = FGP_WRITEBEGIN;
+	if (*foliop == foliop_uncached)
+		fgp_flags |= FGP_UNCACHED;
+
 	/*
 	 * __filemap_get_folio() can take a long time if the
 	 * system is thrashing due to memory pressure, or if the folio

fs/ext4/inode.c +5 -1

@@ -2903,6 +2913,7 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
 	struct folio *folio;
 	pgoff_t index;
 	struct inode *inode = mapping->host;
+	fgf_t fgp_flags;
 
 	if (unlikely(ext4_forced_shutdown(inode->i_sb)))
 		return -EIO;

@@ -2926,8 +2937,11 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
 			return 0;
 	}
 
+	fgp_flags = FGP_WRITEBEGIN;
+	if (*foliop == foliop_uncached)
+		fgp_flags |= FGP_UNCACHED;
 retry:
-	folio = __filemap_get_folio(mapping, index, FGP_WRITEBEGIN,
+	folio = __filemap_get_folio(mapping, index, fgp_flags,
 			mapping_gfp_mask(mapping));
 	if (IS_ERR(folio))
 		return PTR_ERR(folio);

fs/ext4/page-io.c +16 -12

index ad5543866d21..10447c3c4ff1 100644
@@ -226,8 +226,6 @@ static void ext4_add_complete_io(ext4_io_end_t *io_end)
 	unsigned long flags;
 
 	/* Only reserved conversions from writeback should enter here */
-	WARN_ON(!(io_end->flag & EXT4_IO_END_UNWRITTEN));
-	WARN_ON(!io_end->handle && sbi->s_journal);
 	spin_lock_irqsave(&ei->i_completed_io_lock, flags);
 	wq = sbi->rsv_conversion_wq;
 	if (list_empty(&ei->i_rsv_conversion_list))

@@ -252,7 +250,7 @@ static int ext4_do_flush_completed_IO(struct inode *inode,
 
 	while (!list_empty(&unwritten)) {
 		io_end = list_entry(unwritten.next, ext4_io_end_t, list);
-		BUG_ON(!(io_end->flag & EXT4_IO_END_UNWRITTEN));
+		BUG_ON(!(io_end->flag & (EXT4_IO_END_UNWRITTEN|EXT4_IO_UNCACHED)));
 		list_del_init(&io_end->list);
 
 		err = ext4_end_io_end(io_end);

@@ -287,14 +285,15 @@ ext4_io_end_t *ext4_init_io_end(struct inode *inode, gfp_t flags)
 
 void ext4_put_io_end_defer(ext4_io_end_t *io_end)
 {
-	if (refcount_dec_and_test(&io_end->count)) {
-		if (!(io_end->flag & EXT4_IO_END_UNWRITTEN) ||
-				list_empty(&io_end->list_vec)) {
-			ext4_release_io_end(io_end);
-			return;
-		}
-		ext4_add_complete_io(io_end);
+	if (!refcount_dec_and_test(&io_end->count))
+		return;
+	if ((!(io_end->flag & EXT4_IO_END_UNWRITTEN) ||
+	    list_empty(&io_end->list_vec)) &&
+	    !(io_end->flag & EXT4_IO_UNCACHED)) {
+		ext4_release_io_end(io_end);
+		return;
 	}
+	ext4_add_complete_io(io_end);
 }
 
 int ext4_put_io_end(ext4_io_end_t *io_end)

@@ -348,7 +347,7 @@ static void ext4_end_bio(struct bio *bio)
 				blk_status_to_errno(bio->bi_status));
 	}
 
-	if (io_end->flag & EXT4_IO_END_UNWRITTEN) {
+	if (io_end->flag & (EXT4_IO_END_UNWRITTEN|EXT4_IO_UNCACHED)) {
 		/*
 		 * Link bio into list hanging from io_end. We have to do it
 		 * atomically as bio completions can be racing against each

@@ -417,8 +416,13 @@ static void io_submit_add_bh(struct ext4_io_submit *io,
 submit_and_retry:
 		ext4_io_submit(io);
 	}
-	if (io->io_bio == NULL)
+	if (io->io_bio == NULL) {
 		io_submit_init_bio(io, bh);
+		if (folio_test_uncached(folio)) {
+			ext4_io_end_t *io_end = io->io_bio->bi_private;
+			io_end->flag |= EXT4_IO_UNCACHED;
+		}
+	}
 	if (!bio_add_folio(io->io_bio, io_folio, bh->b_size, bh_offset(bh)))
 		goto submit_and_retry;
 	wbc_account_cgroup_owner(io->io_wbc, &folio->page, bh->b_size);

Signature · 1 line

2.45.2

Quoted 21 lines · Jens Axboe

IOCB_UNCACHED IO needs to prune writeback regions on IO completion, and hence need the worker punt that ext4 also does for unwritten extents. Add an io_end flag to manage that.

If foliop is set to foliop_uncached in ext4_write_begin(), then set FGP_UNCACHED so that __filemap_get_folio() will mark newly created folios as uncached. That in turn will make writeback completion drop these ranges from the page cache.

Now that ext4 supports both uncached reads and writes, add the fop_flag FOP_UNCACHED to enable it.

Signed-off-by Jens Axboe <[email protected]>

---
 fs/ext4/ext4.h    |  1 +
 fs/ext4/file.c    |  2 +-
 fs/ext4/inline.c  |  7 ++++++-
 fs/ext4/inode.c   | 18 ++++++++++++++++--
 fs/ext4/page-io.c | 28 ++++++++++++++++------------
 5 files changed, 40 insertions(+), 16 deletions(-)

...

Quoted 29 lines

fs/ext4/inode.c +10 -0

index 54bdd4884fe6..afae3ab64c9e 100644
@@ -1138,6 +1138,7 @@ static int ext4_write_begin(struct file *file, struct address_space *mapping,
 	int ret, needed_blocks;
 	handle_t *handle;
 	int retries = 0;
+	fgf_t fgp_flags;
 	struct folio *folio;
 	pgoff_t index;
 	unsigned from, to;

@@ -1164,6 +1165,15 @@ static int ext4_write_begin(struct file *file, struct address_space *mapping,
 			return 0;
 	}
 
+	/*
+	 * Set FGP_WRITEBEGIN, and FGP_UNCACHED if foliop contains
+	 * foliop_uncached. That's how generic_perform_write() informs us
+	 * that this is an uncached write.
+	 */
+	fgp_flags = FGP_WRITEBEGIN;
+	if (*foliop == foliop_uncached)
+		fgp_flags |= FGP_UNCACHED;
+
 	/*
 	 * __filemap_get_folio() can take a long time if the
 	 * system is thrashing due to memory pressure, or if the folio

fs/ext4/inode.c static int ext4_write_begin(struct file *file, struct address_space *mapping, · line 1188

Quoted 8 lines

fs/ext4/inode.c +1 -1

@@ -1172,7 +1182,7 @@ static int ext4_write_begin(struct file *file, struct address_space *mapping,
 	 * the folio (if needed) without using GFP_NOFS.
 	 */
 retry_grab:
-	folio = __filemap_get_folio(mapping, index, FGP_WRITEBEGIN,
+	folio = __filemap_get_folio(mapping, index, fgp_flags,
 					mapping_gfp_mask(mapping));
 	if (IS_ERR(folio))
 		return PTR_ERR(folio);

JFYI, I notice that ext4 cycles the folio lock here in this path and thus follows up with a couple checks presumably to accommodate that. One is whether i_mapping has changed, which I assume means uncached state would have been handled/cleared externally somewhere..? I.e., if an uncached folio is somehow truncated/freed without ever having been written back?

The next is a folio_wait_stable() call "in case writeback began ..." It's not immediately clear to me if that is possible here, but taking that at face value, is it an issue if we were to create an uncached folio, drop the folio lock, then have some other task dirty and writeback the folio (due to a sync write or something), then have writeback completion invalidate the folio before we relock it here?

Quoted 68 lines · Brian Foster

Quoted 21 lines · Jens Axboe

IOCB_UNCACHED IO needs to prune writeback regions on IO completion, and hence need the worker punt that ext4 also does for unwritten extents. Add an io_end flag to manage that.

If foliop is set to foliop_uncached in ext4_write_begin(), then set FGP_UNCACHED so that __filemap_get_folio() will mark newly created folios as uncached. That in turn will make writeback completion drop these ranges from the page cache.

Now that ext4 supports both uncached reads and writes, add the fop_flag FOP_UNCACHED to enable it.

Signed-off-by Jens Axboe <[email protected]>

---
 fs/ext4/ext4.h    |  1 +
 fs/ext4/file.c    |  2 +-
 fs/ext4/inline.c  |  7 ++++++-
 fs/ext4/inode.c   | 18 ++++++++++++++++--
 fs/ext4/page-io.c | 28 ++++++++++++++++------------
 5 files changed, 40 insertions(+), 16 deletions(-)

...

Quoted 29 lines

fs/ext4/inode.c +10 -0

index 54bdd4884fe6..afae3ab64c9e 100644
@@ -1138,6 +1138,7 @@ static int ext4_write_begin(struct file *file, struct address_space *mapping,
 	int ret, needed_blocks;
 	handle_t *handle;
 	int retries = 0;
+	fgf_t fgp_flags;
 	struct folio *folio;
 	pgoff_t index;
 	unsigned from, to;

@@ -1164,6 +1165,15 @@ static int ext4_write_begin(struct file *file, struct address_space *mapping,
 			return 0;
 	}
 
+	/*
+	 * Set FGP_WRITEBEGIN, and FGP_UNCACHED if foliop contains
+	 * foliop_uncached. That's how generic_perform_write() informs us
+	 * that this is an uncached write.
+	 */
+	fgp_flags = FGP_WRITEBEGIN;
+	if (*foliop == foliop_uncached)
+		fgp_flags |= FGP_UNCACHED;
+
 	/*
 	 * __filemap_get_folio() can take a long time if the
 	 * system is thrashing due to memory pressure, or if the folio

fs/ext4/inode.c static int ext4_write_begin(struct file *file, struct address_space *mapping, · line 1188

Quoted 8 lines

fs/ext4/inode.c +1 -1

@@ -1172,7 +1182,7 @@ static int ext4_write_begin(struct file *file, struct address_space *mapping,
 	 * the folio (if needed) without using GFP_NOFS.
 	 */
 retry_grab:
-	folio = __filemap_get_folio(mapping, index, FGP_WRITEBEGIN,
+	folio = __filemap_get_folio(mapping, index, fgp_flags,
 					mapping_gfp_mask(mapping));
 	if (IS_ERR(folio))
 		return PTR_ERR(folio);

JFYI, I notice that ext4 cycles the folio lock here in this path and thus follows up with a couple checks presumably to accommodate that. One is whether i_mapping has changed, which I assume means uncached state would have been handled/cleared externally somewhere..? I.e., if an uncached folio is somehow truncated/freed without ever having been written back?

Quoted 6 lines

The next is a folio_wait_stable() call "in case writeback began ..." It's not immediately clear to me if that is possible here, but taking that at face value, is it an issue if we were to create an uncached folio, drop the folio lock, then have some other task dirty and writeback the folio (due to a sync write or something), then have writeback completion invalidate the folio before we relock it here?

I don't either of those are an issue. The UNCACHED flag will only be set on a newly created folio, it does not get inherited for folios that already exist.

Signature · 1 line

Jens Axboe

Quoted 76 lines · Jens Axboe

Quoted 68 lines · Brian Foster

Quoted 21 lines · Jens Axboe

IOCB_UNCACHED IO needs to prune writeback regions on IO completion, and hence need the worker punt that ext4 also does for unwritten extents. Add an io_end flag to manage that.

If foliop is set to foliop_uncached in ext4_write_begin(), then set FGP_UNCACHED so that __filemap_get_folio() will mark newly created folios as uncached. That in turn will make writeback completion drop these ranges from the page cache.

Now that ext4 supports both uncached reads and writes, add the fop_flag FOP_UNCACHED to enable it.

Signed-off-by Jens Axboe <[email protected]>

---
 fs/ext4/ext4.h    |  1 +
 fs/ext4/file.c    |  2 +-
 fs/ext4/inline.c  |  7 ++++++-
 fs/ext4/inode.c   | 18 ++++++++++++++++--
 fs/ext4/page-io.c | 28 ++++++++++++++++------------
 5 files changed, 40 insertions(+), 16 deletions(-)

...

Quoted 29 lines

fs/ext4/inode.c +10 -0

index 54bdd4884fe6..afae3ab64c9e 100644
@@ -1138,6 +1138,7 @@ static int ext4_write_begin(struct file *file, struct address_space *mapping,
 	int ret, needed_blocks;
 	handle_t *handle;
 	int retries = 0;
+	fgf_t fgp_flags;
 	struct folio *folio;
 	pgoff_t index;
 	unsigned from, to;

@@ -1164,6 +1165,15 @@ static int ext4_write_begin(struct file *file, struct address_space *mapping,
 			return 0;
 	}
 
+	/*
+	 * Set FGP_WRITEBEGIN, and FGP_UNCACHED if foliop contains
+	 * foliop_uncached. That's how generic_perform_write() informs us
+	 * that this is an uncached write.
+	 */
+	fgp_flags = FGP_WRITEBEGIN;
+	if (*foliop == foliop_uncached)
+		fgp_flags |= FGP_UNCACHED;
+
 	/*
 	 * __filemap_get_folio() can take a long time if the
 	 * system is thrashing due to memory pressure, or if the folio

fs/ext4/inode.c static int ext4_write_begin(struct file *file, struct address_space *mapping, · line 1188

Quoted 8 lines

fs/ext4/inode.c +1 -1

@@ -1172,7 +1182,7 @@ static int ext4_write_begin(struct file *file, struct address_space *mapping,
 	 * the folio (if needed) without using GFP_NOFS.
 	 */
 retry_grab:
-	folio = __filemap_get_folio(mapping, index, FGP_WRITEBEGIN,
+	folio = __filemap_get_folio(mapping, index, fgp_flags,
 					mapping_gfp_mask(mapping));
 	if (IS_ERR(folio))
 		return PTR_ERR(folio);

JFYI, I notice that ext4 cycles the folio lock here in this path and thus follows up with a couple checks presumably to accommodate that. One is whether i_mapping has changed, which I assume means uncached state would have been handled/cleared externally somewhere..? I.e., if an uncached folio is somehow truncated/freed without ever having been written back?

Quoted 6 lines

The next is a folio_wait_stable() call "in case writeback began ..." It's not immediately clear to me if that is possible here, but taking that at face value, is it an issue if we were to create an uncached folio, drop the folio lock, then have some other task dirty and writeback the folio (due to a sync write or something), then have writeback completion invalidate the folio before we relock it here?

Quoted 4 lines

I don't either of those are an issue. The UNCACHED flag will only be set on a newly created folio, it does not get inherited for folios that already exist.

Right.. but what I was wondering for that latter case is if the folio is created here by ext4, so uncached is set before it is unlocked.

On second look I guess the uncached completion invalidation should clear mapping and thus trigger the retry logic here. That seems reasonable enough, but is it still possible to race with writeback?

Maybe this is a better way to ask.. what happens if a write completes to an uncached folio that is already under writeback? For example, uncached write 1 completes, submits for writeback and returns to userspace. Then write 2 begins and redirties the same folio before the uncached writeback completes.

If I follow correctly, if write 2 is also uncached, it eventually blocks in writeback submission (folio_prepare_writeback() -> folio_wait_writeback()). It looks like folio lock is held there, so presumably that would bypass the completion time invalidation in folio_end_uncached(). But what if write 2 was not uncached or perhaps writeback completion won the race for folio lock vs. the write side (between locking the folio for dirtying and later for writeback submission)? Does anything prevent invalidation of the folio before the second write is submitted for writeback?

IOW, I'm wondering if the uncached completion time invalidation also needs a folio dirty check..?

Add iomap buffered write support for RWF_UNCACHED. If RWF_UNCACHED is set for a write, mark the folios being written with drop_writeback. Then writeback completion will drop the pages. The write_iter handler simply kicks off writeback for the pages, and writeback completion will take care of the rest.

This still needs the user of the iomap buffered write helpers to call iocb_uncached_write() upon successful issue of the writes.

Signed-off-by Jens Axboe <[email protected]>

---
 fs/iomap/buffered-io.c | 15 +++++++++++++--
 include/linux/iomap.h  |  4 +++-
 2 files changed, 16 insertions(+), 3 deletions(-)

fs/iomap/buffered-io.c +13 -2

index ef0b68bccbb6..2f2a5db04a68 100644
@@ -603,6 +603,8 @@ struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len)
 
 	if (iter->flags & IOMAP_NOWAIT)
 		fgp |= FGP_NOWAIT;
+	if (iter->flags & IOMAP_UNCACHED)
+		fgp |= FGP_UNCACHED;
 	fgp |= fgf_set_order(len);
 
 	return __filemap_get_folio(iter->inode->i_mapping, pos >> PAGE_SHIFT,

@@ -1023,8 +1025,9 @@ ssize_t
 iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i,
 		const struct iomap_ops *ops, void *private)
 {
+	struct address_space *mapping = iocb->ki_filp->f_mapping;
 	struct iomap_iter iter = {
-		.inode		= iocb->ki_filp->f_mapping->host,
+		.inode		= mapping->host,
 		.pos		= iocb->ki_pos,
 		.len		= iov_iter_count(i),
 		.flags		= IOMAP_WRITE,

@@ -1034,9 +1037,14 @@ iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i,
 
 	if (iocb->ki_flags & IOCB_NOWAIT)
 		iter.flags |= IOMAP_NOWAIT;
+	if (iocb->ki_flags & IOCB_UNCACHED)
+		iter.flags |= IOMAP_UNCACHED;
 
-	while ((ret = iomap_iter(&iter, ops)) > 0)
+	while ((ret = iomap_iter(&iter, ops)) > 0) {
+		if (iocb->ki_flags & IOCB_UNCACHED)
+			iter.iomap.flags |= IOMAP_F_UNCACHED;
 		iter.processed = iomap_write_iter(&iter, i);
+	}
 
 	if (unlikely(iter.pos == iocb->ki_pos))
 		return ret;

@@ -1770,6 +1778,9 @@ static int iomap_add_to_ioend(struct iomap_writepage_ctx *wpc,
 	size_t poff = offset_in_folio(folio, pos);
 	int error;
 
+	if (folio_test_uncached(folio))
+		wpc->iomap.flags |= IOMAP_F_UNCACHED;
+
 	if (!wpc->ioend || !iomap_can_add_to_ioend(wpc, pos)) {
 new_ioend:
 		error = iomap_submit_ioend(wpc, 0);

include/linux/iomap.h +0 -0

index f61407e3b121..2efc72df19a2 100644

include/linux/iomap.h +2 -1

@@ -173,8 +174,9 @@ struct iomap_folio_ops {
 #define IOMAP_NOWAIT		(1 << 5) /* do not block */
 #define IOMAP_OVERWRITE_ONLY	(1 << 6) /* only pure overwrites allowed */
 #define IOMAP_UNSHARE		(1 << 7) /* unshare_file_range */
+#define IOMAP_UNCACHED		(1 << 8) /* uncached IO */
 #ifdef CONFIG_FS_DAX
-#define IOMAP_DAX		(1 << 8) /* DAX mapping */
+#define IOMAP_DAX		(1 << 9) /* DAX mapping */
 #else
 #define IOMAP_DAX		0
 #endif /* CONFIG_FS_DAX */

Signature · 1 line

2.45.2

Quoted 68 lines · Jens Axboe

Add iomap buffered write support for RWF_UNCACHED. If RWF_UNCACHED is set for a write, mark the folios being written with drop_writeback. Then writeback completion will drop the pages. The write_iter handler simply kicks off writeback for the pages, and writeback completion will take care of the rest.

This still needs the user of the iomap buffered write helpers to call iocb_uncached_write() upon successful issue of the writes.

Signed-off-by Jens Axboe <[email protected]>

---
 fs/iomap/buffered-io.c | 15 +++++++++++++--
 include/linux/iomap.h  |  4 +++-
 2 files changed, 16 insertions(+), 3 deletions(-)

fs/iomap/buffered-io.c +13 -2

index ef0b68bccbb6..2f2a5db04a68 100644
@@ -603,6 +603,8 @@ struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len)
 
 	if (iter->flags & IOMAP_NOWAIT)
 		fgp |= FGP_NOWAIT;
+	if (iter->flags & IOMAP_UNCACHED)
+		fgp |= FGP_UNCACHED;
 	fgp |= fgf_set_order(len);
 
 	return __filemap_get_folio(iter->inode->i_mapping, pos >> PAGE_SHIFT,

@@ -1023,8 +1025,9 @@ ssize_t
 iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i,
 		const struct iomap_ops *ops, void *private)
 {
+	struct address_space *mapping = iocb->ki_filp->f_mapping;
 	struct iomap_iter iter = {
-		.inode		= iocb->ki_filp->f_mapping->host,
+		.inode		= mapping->host,
 		.pos		= iocb->ki_pos,
 		.len		= iov_iter_count(i),
 		.flags		= IOMAP_WRITE,

@@ -1034,9 +1037,14 @@ iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i,
 
 	if (iocb->ki_flags & IOCB_NOWAIT)
 		iter.flags |= IOMAP_NOWAIT;
+	if (iocb->ki_flags & IOCB_UNCACHED)
+		iter.flags |= IOMAP_UNCACHED;
 
-	while ((ret = iomap_iter(&iter, ops)) > 0)
+	while ((ret = iomap_iter(&iter, ops)) > 0) {
+		if (iocb->ki_flags & IOCB_UNCACHED)
+			iter.iomap.flags |= IOMAP_F_UNCACHED;
 		iter.processed = iomap_write_iter(&iter, i);
+	}
 
 	if (unlikely(iter.pos == iocb->ki_pos))
 		return ret;

@@ -1770,6 +1778,9 @@ static int iomap_add_to_ioend(struct iomap_writepage_ctx *wpc,
 	size_t poff = offset_in_folio(folio, pos);
 	int error;
 
+	if (folio_test_uncached(folio))
+		wpc->iomap.flags |= IOMAP_F_UNCACHED;
+
 	if (!wpc->ioend || !iomap_can_add_to_ioend(wpc, pos)) {
 new_ioend:
 		error = iomap_submit_ioend(wpc, 0);

include/linux/iomap.h +0 -1

index f61407e3b121..2efc72df19a2 100644
--- a/include/linux/iomap.h

include/linux/iomap.h struct vm_fault; · line 67

Quoted 6 lines

+++ b/include/linux/iomap.h

include/linux/iomap.h +1 -0

@@ -64,6 +64,7 @@ struct vm_fault;
 #define IOMAP_F_BUFFER_HEAD	0
 #endif /* CONFIG_BUFFER_HEAD */
 #define IOMAP_F_XATTR		(1U << 5)
+#define IOMAP_F_UNCACHED	(1U << 6)

This value ^^^ is set only by the core iomap code, right?

include/linux/iomap.h struct vm_fault; · line 70

Quoted 2 lines

include/linux/iomap.h +0 -0

 /*
  * Flags set by the core iomap code during operations:

...in which case it should be set down here. It probably ought to have a description of what it does, too:

"IOMAP_F_UNCACHED is set to indicate that writes to the page cache (and hence writeback) will result in folios being evicted as soon as the updated bytes are written back to the storage."

If the writeback fails, does that mean that the dirty data will /not/ be retained in the page cache? IIRC we finally got to the point where the major filesystems leave pagecache alone after writeback EIO.

The rest of the mechanics looks nifty to me; there's plenty of places where this could be useful to me personally. :)

Add iomap buffered write support for RWF_UNCACHED. If RWF_UNCACHED is set for a write, mark the folios being written with drop_writeback. Then

writeback completion will drop the pages. The write_iter handler simply kicks off writeback for the pages, and writeback completion will take care of the rest.

This still needs the user of the iomap buffered write helpers to call iocb_uncached_write() upon successful issue of the writes.

Signed-off-by Jens Axboe <[email protected]>

---
 fs/iomap/buffered-io.c | 15 +++++++++++++--
 include/linux/iomap.h  |  4 +++-
 2 files changed, 16 insertions(+), 3 deletions(-)

fs/iomap/buffered-io.c +13 -2

index ef0b68bccbb6..2f2a5db04a68 100644
@@ -603,6 +603,8 @@ struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len)
 
 	if (iter->flags & IOMAP_NOWAIT)
 		fgp |= FGP_NOWAIT;
+	if (iter->flags & IOMAP_UNCACHED)
+		fgp |= FGP_UNCACHED;
 	fgp |= fgf_set_order(len);
 
 	return __filemap_get_folio(iter->inode->i_mapping, pos >> PAGE_SHIFT,

@@ -1023,8 +1025,9 @@ ssize_t
 iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i,
 		const struct iomap_ops *ops, void *private)
 {
+	struct address_space *mapping = iocb->ki_filp->f_mapping;
 	struct iomap_iter iter = {
-		.inode		= iocb->ki_filp->f_mapping->host,
+		.inode		= mapping->host,
 		.pos		= iocb->ki_pos,
 		.len		= iov_iter_count(i),
 		.flags		= IOMAP_WRITE,

@@ -1034,9 +1037,14 @@ iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i,
 
 	if (iocb->ki_flags & IOCB_NOWAIT)
 		iter.flags |= IOMAP_NOWAIT;
+	if (iocb->ki_flags & IOCB_UNCACHED)
+		iter.flags |= IOMAP_UNCACHED;
 
-	while ((ret = iomap_iter(&iter, ops)) > 0)
+	while ((ret = iomap_iter(&iter, ops)) > 0) {
+		if (iocb->ki_flags & IOCB_UNCACHED)
+			iter.iomap.flags |= IOMAP_F_UNCACHED;
 		iter.processed = iomap_write_iter(&iter, i);
+	}
 
 	if (unlikely(iter.pos == iocb->ki_pos))
 		return ret;

@@ -1770,6 +1778,9 @@ static int iomap_add_to_ioend(struct iomap_writepage_ctx *wpc,
 	size_t poff = offset_in_folio(folio, pos);
 	int error;
 
+	if (folio_test_uncached(folio))
+		wpc->iomap.flags |= IOMAP_F_UNCACHED;
+
 	if (!wpc->ioend || !iomap_can_add_to_ioend(wpc, pos)) {
 new_ioend:
 		error = iomap_submit_ioend(wpc, 0);

include/linux/iomap.h +3 -1

index f61407e3b121..2efc72df19a2 100644
@@ -64,6 +64,7 @@ struct vm_fault;
 #define IOMAP_F_BUFFER_HEAD	0
 #endif /* CONFIG_BUFFER_HEAD */
 #define IOMAP_F_XATTR		(1U << 5)
+#define IOMAP_F_UNCACHED	(1U << 6)
 
 /*
  * Flags set by the core iomap code during operations:

@@ -173,8 +174,9 @@ struct iomap_folio_ops {
 #define IOMAP_NOWAIT		(1 << 5) /* do not block */
 #define IOMAP_OVERWRITE_ONLY	(1 << 6) /* only pure overwrites allowed */
 #define IOMAP_UNSHARE		(1 << 7) /* unshare_file_range */
+#define IOMAP_UNCACHED		(1 << 8) /* uncached IO */
 #ifdef CONFIG_FS_DAX
-#define IOMAP_DAX		(1 << 8) /* DAX mapping */
+#define IOMAP_DAX		(1 << 9) /* DAX mapping */
 #else
 #define IOMAP_DAX		0
 #endif /* CONFIG_FS_DAX */

Signature · 1 line

2.45.2

Quoted 2 lines · Jens Axboe

Add iomap buffered write support for RWF_UNCACHED. If RWF_UNCACHED is set for a write, mark the folios being written with drop_writeback. Then

s/drop_writeback/uncached/ ?

BTW, this might be getting into wonky "don't care that much" territory, but something else to be aware of is that certain writes can potentially change pagecache state as a side effect outside of the actual buffered write itself.

For example, xfs calls iomap_zero_range() on write extension (i.e. pos > isize), which uses buffered writes and thus could populate a pagecache folio without setting it uncached, even if done on behalf of an uncached write.

I've only made a first pass and could be missing some details, but IIUC I _think_ this means something like writing out a stream of small, sparse and file extending uncached writes could actually end up behaving more like sync I/O. Again, not saying that's something we really care about, just raising it in case it's worth considering or documenting..

Quoted 5 lines · Brian Foster

Quoted 2 lines · Jens Axboe

Add iomap buffered write support for RWF_UNCACHED. If RWF_UNCACHED is set for a write, mark the folios being written with drop_writeback. Then

s/drop_writeback/uncached/ ?

Ah indeed, guess that never got changed. Thanks, will fix that in the commit message.

Quoted 10 lines

BTW, this might be getting into wonky "don't care that much" territory, but something else to be aware of is that certain writes can potentially change pagecache state as a side effect outside of the actual buffered write itself.

For example, xfs calls iomap_zero_range() on write extension (i.e. pos > isize), which uses buffered writes and thus could populate a pagecache folio without setting it uncached, even if done on behalf of an uncached write.

Quoted 5 lines

I've only made a first pass and could be missing some details, but IIUC I _think_ this means something like writing out a stream of small, sparse and file extending uncached writes could actually end up behaving more like sync I/O. Again, not saying that's something we really care about, just raising it in case it's worth considering or documenting..

No that's useful info, I'm not really surprised that there would still be cases where UNCACHED goes unnoticed. In other words, I'd be surprised if the current patches for eg xfs/ext4 cover all the cases where new folios are created and should be marked as UNCACHED of IOCB_UNCACHED is set in the iocb.

I think those can be sorted out or documented as we move forward. UNCACHED is really just a hint - the kernel should do its best to not have permanent folios for this IO, but there are certainly cases where it won't be honored if you're racing with regular buffered IO or mmap. For the case above, sounds like we could cover that, however, and probably should.

Signature · 1 line

Jens Axboe

I ran this through xfstests, and it found some of the issue listed as fixed below. This posted version passes the whole generic suite of xfstests. The xfstests patch is here: