Uncached buffered IO

mm/filemap.c static int filemap_update_page(struct kiocb *iocb, · line 2463

Quoted 2 lines · Jens Axboe

mm/filemap.c +2 -0

+static int filemap_create_folio(struct kiocb *iocb,
+		struct address_space *mapping, struct folio_batch *fbatch)

We might as well drop passing the mapping and deriving it from the iocb as well.

Otherwise this looks fine to me.

Rather than use the page_cache_sync_readahead() helper, define our own ractl and use page_cache_sync_ra() directly. In preparation for needing to modify ractl inside filemap_get_pages().

No functional changes in this patch.

Add a folio_unmap_invalidate() helper, which unmaps and invalidates a given folio. The caller must already have locked the folio. Use this new helper in invalidate_inode_pages2_range(), rather than duplicate the code there.

In preparation for using this elsewhere as well, have it take a gfp_t mask rather than assume GFP_KERNEL is the right choice. This bubbles back to invalidate_complete_folio2() as well.

Quoted 4 lines · Jens Axboe

Add a folio_unmap_invalidate() helper, which unmaps and invalidates a given folio. The caller must already have locked the folio. Use this new helper in invalidate_inode_pages2_range(), rather than duplicate the code there.

This new helper ends up the only caller of invalidate_complete_folio2, so you might as well merge the two instead of having yet another invalidate/unmap helper, which are getting impossible to track of.

Also it is only used in mm/, so add the prototype to mm/internal.h insead of the public pagemap.h. And a little comment what the function does would be pretty useful as well.

Quoted 3 lines

In preparation for using this elsewhere as well, have it take a gfp_t mask rather than assume GFP_KERNEL is the right choice. This bubbles back to invalidate_complete_folio2() as well.

Looking at the callers the gfp_t looks a bit odd to me, as it is either GFP_KERNEL or 0 which is a valid but rather unusuable gfp_t value, but I guess this comes form filemap_release_folio which works similarly.

If a file system supports uncached buffered IO, it may set FOP_UNCACHED and enable RWF_UNCACHED. If RWF_UNCACHED is attempted without the file system supporting it, it'll get errored with -EOPNOTSUPP.

Signed-off-by Jens Axboe <[email protected]>

---
 include/linux/fs.h      | 14 +++++++++++++-
 include/uapi/linux/fs.h |  6 +++++-
 2 files changed, 18 insertions(+), 2 deletions(-)

include/linux/fs.h +5 -1

index 7e29433c5ecc..b64a78582f06 100644
@@ -322,6 +322,7 @@ struct readahead_control;
 #define IOCB_NOWAIT		(__force int) RWF_NOWAIT
 #define IOCB_APPEND		(__force int) RWF_APPEND
 #define IOCB_ATOMIC		(__force int) RWF_ATOMIC
+#define IOCB_UNCACHED		(__force int) RWF_UNCACHED
 
 /* non-RWF related bits - start at 16 */
 #define IOCB_EVENTFD		(1 << 16)

@@ -356,7 +357,8 @@ struct readahead_control;
 	{ IOCB_SYNC,		"SYNC" }, \
 	{ IOCB_NOWAIT,		"NOWAIT" }, \
 	{ IOCB_APPEND,		"APPEND" }, \
-	{ IOCB_ATOMIC,		"ATOMIC"}, \
+	{ IOCB_ATOMIC,		"ATOMIC" }, \
+	{ IOCB_UNCACHED,	"UNCACHED" }, \
 	{ IOCB_EVENTFD,		"EVENTFD"}, \
 	{ IOCB_DIRECT,		"DIRECT" }, \
 	{ IOCB_WRITE,		"WRITE" }, \

@@ -2127,6 +2129,8 @@ struct file_operations {
 #define FOP_UNSIGNED_OFFSET	((__force fop_flags_t)(1 << 5))
 /* Supports asynchronous lock callbacks */
 #define FOP_ASYNC_LOCK		((__force fop_flags_t)(1 << 6))
+/* File system supports uncached read/write buffered IO */
+#define FOP_UNCACHED		((__force fop_flags_t)(1 << 7))
 
 /* Wrap a directory iterator that needs exclusive inode access */
 int wrap_directory_iterator(struct file *, struct dir_context *,

include/linux/fs.h +7 -0

+		/* file system must support it */
+		if (!(ki->ki_filp->f_op->fop_flags & FOP_UNCACHED))
+			return -EOPNOTSUPP;
+		/* DAX mappings not supported */
+		if (IS_DAX(ki->ki_filp->f_mapping->host))
+			return -EOPNOTSUPP;
+	}
 	kiocb_flags |= (__force int) (flags & RWF_SUPPORTED);
 	if (flags & RWF_SYNC)
 		kiocb_flags |= IOCB_DSYNC;

include/uapi/linux/fs.h +5 -1

index 753971770733..dc77cd8ae1a3 100644
@@ -332,9 +332,13 @@ typedef int __bitwise __kernel_rwf_t;
 /* Atomic Write */
 #define RWF_ATOMIC	((__force __kernel_rwf_t)0x00000040)
 
+/* buffered IO that drops the cache after reading or writing data */
+#define RWF_UNCACHED	((__force __kernel_rwf_t)0x00000080)
+
 /* mask of flags supported by the kernel */
 #define RWF_SUPPORTED	(RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
-			 RWF_APPEND | RWF_NOAPPEND | RWF_ATOMIC)
+			 RWF_APPEND | RWF_NOAPPEND | RWF_ATOMIC |\
+			 RWF_UNCACHED)
 
 #define PROCFS_IOCTL_MAGIC 'f'
 

Signature · 1 line

2.45.2

include/linux/fs.h static inline int kiocb_set_rw_flags(struct kiocb *ki, rwf_t flags, · line 3627

Quoted 7 lines · Jens Axboe

include/linux/fs.h +7 -0

+	if (flags & RWF_UNCACHED) {
+		/* file system must support it */
+		if (!(ki->ki_filp->f_op->fop_flags & FOP_UNCACHED))
+			return -EOPNOTSUPP;
+		/* DAX mappings not supported */
+		if (IS_DAX(ki->ki_filp->f_mapping->host))
+			return -EOPNOTSUPP;

I'd argue that DAX is always uncached and could just ignore the flag. Same for direct I/O.

If RWF_UNCACHED is set for a write, mark new folios being written with uncached. This is done by passing in the fact that it's an uncached write through the folio pointer. We can only get there when IOCB_UNCACHED was allowed, which can only happen if the file system opts in. Opting in means they need to check for the LSB in the folio pointer to know if it's an uncached write or not. If it is, then FGP_UNCACHED should be used if creating new folios is necessary.

Uncached writes will drop any folios they create upon writeback completion, but leave folios that may exist in that range alone. Since ->write_begin() doesn't currently take any flags, and to avoid needing to change the callback kernel wide, use the foliop being passed in to ->write_begin() to signal if this is an uncached write or not. File systems can then use that to mark newly created folios as uncached.

This provides similar benefits to using RWF_UNCACHED with reads. Testing buffered writes on 32 files:

writing bs 65536, uncached 0
1s: 196035MB/sec
2s: 132308MB/sec
3s: 132438MB/sec
4s: 116528MB/sec
5s: 103898MB/sec
6s: 108893MB/sec
7s: 99678MB/sec
8s: 106545MB/sec
9s: 106826MB/sec
10s: 101544MB/sec
11s: 111044MB/sec
12s: 124257MB/sec
13s: 116031MB/sec
14s: 114540MB/sec
15s: 115011MB/sec
16s: 115260MB/sec
17s: 116068MB/sec
18s: 116096MB/sec

where it's quite obvious where the page cache filled, and performance dropped from to about half of where it started, settling in at around 115GB/sec. Meanwhile, 32 kswapds were running full steam trying to reclaim pages.

Running the same test with uncached buffered writes:

writing bs 65536, uncached 1
1s: 198974MB/sec
2s: 189618MB/sec
3s: 193601MB/sec
4s: 188582MB/sec
5s: 193487MB/sec
6s: 188341MB/sec
7s: 194325MB/sec
8s: 188114MB/sec
9s: 192740MB/sec
10s: 189206MB/sec
11s: 193442MB/sec
12s: 189659MB/sec
13s: 191732MB/sec
14s: 190701MB/sec
15s: 191789MB/sec
16s: 191259MB/sec
17s: 190613MB/sec
18s: 191951MB/sec

and the behavior is fully predictable, performing the same throughout even after the page cache would otherwise have fully filled with dirty data. It's also about 65% faster, and using half the CPU of the system compared to the normal buffered write.

Signed-off-by Jens Axboe <[email protected]>

---
 include/linux/fs.h      |  5 +++++
 include/linux/pagemap.h |  9 +++++++++
 mm/filemap.c            | 12 +++++++++++-
 3 files changed, 25 insertions(+), 1 deletion(-)

include/linux/fs.h +5 -0

index 40383f5cc6a2..32255473f79d 100644
@@ -2912,6 +2912,11 @@ static inline ssize_t generic_write_sync(struct kiocb *iocb, ssize_t count)
 				(iocb->ki_flags & IOCB_SYNC) ? 0 : 1);
 		if (ret)
 			return ret;
+	} else if (iocb->ki_flags & IOCB_UNCACHED) {
+		struct address_space *mapping = iocb->ki_filp->f_mapping;
+
+		filemap_fdatawrite_range_kick(mapping, iocb->ki_pos,
+					      iocb->ki_pos + count);
 	}
 
 	return count;

include/linux/pagemap.h +1 -0

index f2d49dccb7c1..e49587c40157 100644
@@ -14,6 +14,7 @@
 #include <linux/gfp.h>
 #include <linux/bitops.h>
 #include <linux/hardirq.h> /* for in_interrupt() */
+#include <linux/writeback.h>
 #include <linux/hugetlb_inline.h>
 
 struct folio_batch;

include/linux/pagemap.h +0 -0

 /**
  * filemap_set_wb_err - set a writeback error on an address_space
  * @mapping: mapping in which to set writeback error

mm/filemap.c +11 -1

index 826df99e294f..00f3c6c58629 100644
@@ -4095,7 +4095,7 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i)
 	ssize_t written = 0;
 
 	do {
-		struct folio *folio;
+		struct folio *folio = NULL;
 		size_t offset;		/* Offset into folio */
 		size_t bytes;		/* Bytes to write to folio */
 		size_t copied;		/* Bytes copied from user */

@@ -4123,6 +4123,16 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i)
 			break;
 		}
 
+		/*
+		 * If IOCB_UNCACHED is set here, we now the file system
+		 * supports it. And hence it'll know to check folip for being
+		 * set to this magic value. If so, it's an uncached write.
+		 * Whenever ->write_begin() changes prototypes again, this
+		 * can go away and just pass iocb or iocb flags.
+		 */
+		if (iocb->ki_flags & IOCB_UNCACHED)
+			folio = foliop_uncached;
+
 		status = a_ops->write_begin(file, mapping, pos, bytes,
 						&folio, &fsdata);
 		if (unlikely(status < 0))

Signature · 1 line

2.45.2

Quoted 109 lines · Jens Axboe

If RWF_UNCACHED is set for a write, mark new folios being written with uncached. This is done by passing in the fact that it's an uncached write through the folio pointer. We can only get there when IOCB_UNCACHED was allowed, which can only happen if the file system opts in. Opting in means they need to check for the LSB in the folio pointer to know if it's an uncached write or not. If it is, then FGP_UNCACHED should be used if creating new folios is necessary.

Uncached writes will drop any folios they create upon writeback completion, but leave folios that may exist in that range alone. Since ->write_begin() doesn't currently take any flags, and to avoid needing to change the callback kernel wide, use the foliop being passed in to ->write_begin() to signal if this is an uncached write or not. File systems can then use that to mark newly created folios as uncached.

This provides similar benefits to using RWF_UNCACHED with reads. Testing buffered writes on 32 files:

writing bs 65536, uncached 0
1s: 196035MB/sec
2s: 132308MB/sec
3s: 132438MB/sec
4s: 116528MB/sec
5s: 103898MB/sec
6s: 108893MB/sec
7s: 99678MB/sec
8s: 106545MB/sec
9s: 106826MB/sec
10s: 101544MB/sec
11s: 111044MB/sec
12s: 124257MB/sec
13s: 116031MB/sec
14s: 114540MB/sec
15s: 115011MB/sec
16s: 115260MB/sec
17s: 116068MB/sec
18s: 116096MB/sec

where it's quite obvious where the page cache filled, and performance dropped from to about half of where it started, settling in at around 115GB/sec. Meanwhile, 32 kswapds were running full steam trying to reclaim pages.

Running the same test with uncached buffered writes:

writing bs 65536, uncached 1
1s: 198974MB/sec
2s: 189618MB/sec
3s: 193601MB/sec
4s: 188582MB/sec
5s: 193487MB/sec
6s: 188341MB/sec
7s: 194325MB/sec
8s: 188114MB/sec
9s: 192740MB/sec
10s: 189206MB/sec
11s: 193442MB/sec
12s: 189659MB/sec
13s: 191732MB/sec
14s: 190701MB/sec
15s: 191789MB/sec
16s: 191259MB/sec
17s: 190613MB/sec
18s: 191951MB/sec

and the behavior is fully predictable, performing the same throughout even after the page cache would otherwise have fully filled with dirty data. It's also about 65% faster, and using half the CPU of the system compared to the normal buffered write.

Signed-off-by Jens Axboe <[email protected]>

---
 include/linux/fs.h      |  5 +++++
 include/linux/pagemap.h |  9 +++++++++
 mm/filemap.c            | 12 +++++++++++-
 3 files changed, 25 insertions(+), 1 deletion(-)

include/linux/fs.h +5 -0

index 40383f5cc6a2..32255473f79d 100644
@@ -2912,6 +2912,11 @@ static inline ssize_t generic_write_sync(struct kiocb *iocb, ssize_t count)
 				(iocb->ki_flags & IOCB_SYNC) ? 0 : 1);
 		if (ret)
 			return ret;
+	} else if (iocb->ki_flags & IOCB_UNCACHED) {
+		struct address_space *mapping = iocb->ki_filp->f_mapping;
+
+		filemap_fdatawrite_range_kick(mapping, iocb->ki_pos,
+					      iocb->ki_pos + count);
 	}
 
 	return count;

include/linux/pagemap.h +1 -0

index f2d49dccb7c1..e49587c40157 100644
@@ -14,6 +14,7 @@
 #include <linux/gfp.h>
 #include <linux/bitops.h>
 #include <linux/hardirq.h> /* for in_interrupt() */
+#include <linux/writeback.h>
 #include <linux/hugetlb_inline.h>
 
 struct folio_batch;

@@ -70,6 +71,14 @@ static inline int filemap_write_and_wait(struct address_space *mapping)
 	return filemap_write_and_wait_range(mapping, 0, LLONG_MAX);
 }
 

include/linux/pagemap.h static inline int filemap_write_and_wait(struct address_space *mapping) · line 80

Quoted 7 lines

include/linux/pagemap.h +7 -0

+/*
+ * Value passed in to ->write_begin() if IOCB_UNCACHED is set for the write,
+ * and the ->write_begin() handler on a file system supporting FOP_UNCACHED
+ * must check for this and pass FGP_UNCACHED for folio creation.
+ */
+#define foliop_uncached			((struct folio *) 0xfee1c001)
+#define foliop_is_uncached(foliop)	(*(foliop) == foliop_uncached)

Honestly, I'm not a fan of foliop_uncached or foliop_is_uncached.

The first one because it's a magic value and can you guarantee that 0xfee1c001 will never be a pointer to an actual struct folio, even on 32-bit?

Second, they're both named "foliop" even though the first one doesn't return a (struct folio **) but the second one takes that as an arg.

I think these two macros are only used for ext4 (or really, !iomap) support, right? And that's only to avoid messing with ->write_begin?

What if you dropped ext4 support instead? :D

Quoted 1 line

Honestly, I'm not a fan of foliop_uncached or foliop_is_uncached.

It definitely is what I would elegantly refer to as somewhat of a hack... But it's not _that_ bad imho.

Quoted 2 lines

I think these two macros are only used for ext4 (or really, !iomap) support, right? And that's only to avoid messing with ->write_begin?

Indeed, ideally we'd change ->write_begin() instead. And that probably should still be done, I just did not want to deal with that nightmare in terms of managing the patchset. And honestly I think it'd be OK to defer that part until ->write_begin() needs to be changed for other reasons, it's a lot of churn just for this particular thing and dealing with the magic pointer value (at least to me) is liveable.

Quoted 1 line

What if you dropped ext4 support instead? :D

Hah, yes obviously that'd be a solution, then I'd need to drop btrfs as well. And I would kind of prefer not doing that ;-)

Quoted 4 lines · Jens Axboe

Quoted 1 line

Honestly, I'm not a fan of foliop_uncached or foliop_is_uncached.

It definitely is what I would elegantly refer to as somewhat of a hack... But it's not _that_ bad imho.

It's pretty horrible actually.

Quoted 9 lines

Quoted 2 lines

I think these two macros are only used for ext4 (or really, !iomap) support, right? And that's only to avoid messing with ->write_begin?

Indeed, ideally we'd change ->write_begin() instead. And that probably should still be done, I just did not want to deal with that nightmare in terms of managing the patchset. And honestly I think it'd be OK to defer that part until ->write_begin() needs to be changed for other reasons, it's a lot of churn just for this particular thing and dealing with the magic pointer value (at least to me) is liveable.

->write_begin() really should just go away, it is a horrible interface. Note that in that past it actually had a flags argument, but that got killed a while ago.

Quoted 2 lines

Quoted 1 line

What if you dropped ext4 support instead? :D

Quoted 2 lines

Hah, yes obviously that'd be a solution, then I'd need to drop btrfs as well. And I would kind of prefer not doing that ;-)

Btrfs doesn't need it. In fact the code would be cleaner and do less work with out, see the patch below. And for ext4 there already is an iomap conversion patch series on the list that just needs more review, so skipping it here and growing the uncached support through that sounds sensible.

So here's a new approach to the same concent, but using the page cache as synchronization. That makes RWF_UNCACHED less special, in that it's just page cache IO, except it prunes the ranges once IO is completed.

Quoted 4 lines · Jens Axboe

So here's a new approach to the same concent, but using the page cache as synchronization. That makes RWF_UNCACHED less special, in that it's just page cache IO, except it prunes the ranges once IO is completed.

Great idea and someting that is really important these days.

However, one nit that I have is the use of the term "uncached" for a folio/page. An uncached "page frame" refers to a page frame that requires accesses not going through the cpu cache. I.e. device mappings. This is an established mm/cpu term as far as I can tell.

So maybe be a bit more specific about which cache this is?

PAGE_CACHE_UNCACHED?

or use a different term. It is cached after all but only for a brief period. So this may be a "TEMPORAL_PAGE" or so? (Similar to the x86 "non-temporal" stores).

I actually did consider using some form of temporal, as it's the only other name I liked. But I do think cached_uncached becomes pretty unwieldy. Which is why I just stuck with uncached. Yes I know it means different things in different circles, but probably mostly an overlap with deeper technical things like that. An honestly almost impossible to avoid overlap these days, everything has been used already :-)

IOW, I think uncached is probably still the most descriptive thing out there, even if I'm certainly open to entertaining other names. Just not anything yet that has really resonated with me.

Quoted 7 lines · Jens Axboe

I actually did consider using some form of temporal, as it's the only other name I liked. But I do think cached_uncached becomes pretty unwieldy. Which is why I just stuck with uncached. Yes I know it means different things in different circles, but probably mostly an overlap with deeper technical things like that. An honestly almost impossible to avoid overlap these days, everything has been used already :-)

Quoted 3 lines

IOW, I think uncached is probably still the most descriptive thing out there, even if I'm certainly open to entertaining other names. Just not anything yet that has really resonated with me.

How about calling this a "transitory" page? It means fleeting, not persistent and I think we have not used that term with a page/folio yet.

Quoted 9 lines · Christoph Lameter (Ampere)

Quoted 7 lines · Jens Axboe

I actually did consider using some form of temporal, as it's the only other name I liked. But I do think cached_uncached becomes pretty unwieldy. Which is why I just stuck with uncached. Yes I know it means different things in different circles, but probably mostly an overlap with deeper technical things like that. An honestly almost impossible to avoid overlap these days, everything has been used already :-)

Quoted 6 lines

Quoted 3 lines

IOW, I think uncached is probably still the most descriptive thing out there, even if I'm certainly open to entertaining other names. Just not anything yet that has really resonated with me.

How about calling this a "transitory" page? It means fleeting, not persistent and I think we have not used that term with a page/folio yet.

I also hit the thesaurus ;-)

I'm honestly not too worried about the internal name, as developers can figure that out. It's more about presenting an external name that sys developers will not need a lot of explaining to know what it's about. And something that isn't too long. BRIEFLY_CACHED? TRANSIENT_CACHE?

Dunno, I keep going back to uncached as it's pretty easy to grok!

Signature · 1 line

Jens Axboe

Quoted 24 lines · Jens Axboe

Quoted 9 lines · Christoph Lameter (Ampere)

Quoted 7 lines · Jens Axboe

I actually did consider using some form of temporal, as it's the only other name I liked. But I do think cached_uncached becomes pretty unwieldy. Which is why I just stuck with uncached. Yes I know it means different things in different circles, but probably mostly an overlap with deeper technical things like that. An honestly almost impossible to avoid overlap these days, everything has been used already :-)

Quoted 6 lines

Quoted 3 lines

IOW, I think uncached is probably still the most descriptive thing out there, even if I'm certainly open to entertaining other names. Just not anything yet that has really resonated with me.

How about calling this a "transitory" page? It means fleeting, not persistent and I think we have not used that term with a page/folio yet.

I also hit the thesaurus ;-)

I'm honestly not too worried about the internal name, as developers can figure that out. It's more about presenting an external name that sys developers will not need a lot of explaining to know what it's about. And something that isn't too long. BRIEFLY_CACHED? TRANSIENT_CACHE?

Quoted 1 line

Dunno, I keep going back to uncached as it's pretty easy to grok!

<shrug> RWF_DONTCACHE, to match {I,DCACHE}_DONTCACHE ? ;)

They sound pretty similar ("load this so I can do something with it, evict it immediately if possible") though I wouldn't rely on people outside the kernel being familiar with the existing dontcaches.

<shrug> RWF_DONTCACHE, to match {I,DCACHE}_DONTCACHE ? ;)

They sound pretty similar ("load this so I can do something with it, evict it immediately if possible") though I wouldn't rely on people outside the kernel being familiar with the existing dontcaches.

Quoted 5 lines · Darrick J. Wong

<shrug> RWF_DONTCACHE, to match {I,DCACHE}_DONTCACHE ? ;)

They sound pretty similar ("load this so I can do something with it, evict it immediately if possible") though I wouldn't rely on people outside the kernel being familiar with the existing dontcaches.

FYI, another word for dontcache. uncached just has too many conotations in the kernel context.

Quoted 9 lines · Christoph Hellwig

Quoted 5 lines · Darrick J. Wong

<shrug> RWF_DONTCACHE, to match {I,DCACHE}_DONTCACHE ? ;)

They sound pretty similar ("load this so I can do something with it, evict it immediately if possible") though I wouldn't rely on people outside the kernel being familiar with the existing dontcaches.

FYI, another word for dontcache. uncached just has too many conotations in the kernel context.

Sure, we can go with DONTCACHE instead. Only thing I don't like about that is that you can use uncached as a verb and adjective, eg talking about uncached IO. Talking about dontcached IO sounds pretty weird.

As I've said previously in this and other threads, I don't feel too strongly about the in-kernel naming, I care more about the exposed name. And uncached does seem to be the most descriptive and most easily understandable by users.

Quoted 15 lines · Jens Axboe

Quoted 9 lines · Christoph Hellwig

Quoted 5 lines · Darrick J. Wong

<shrug> RWF_DONTCACHE, to match {I,DCACHE}_DONTCACHE ? ;)

They sound pretty similar ("load this so I can do something with it, evict it immediately if possible") though I wouldn't rely on people outside the kernel being familiar with the existing dontcaches.

FYI, another word for dontcache. uncached just has too many conotations in the kernel context.

Sure, we can go with DONTCACHE instead. Only thing I don't like about that is that you can use uncached as a verb and adjective, eg talking about uncached IO. Talking about dontcached IO sounds pretty weird.

Quoted 4 lines

As I've said previously in this and other threads, I don't feel too strongly about the in-kernel naming, I care more about the exposed name. And uncached does seem to be the most descriptive and most easily understandable by users.

The page is cached while the operation is ongoing. "Transitory" would be more accurate and it is a new term that was not used with pages before.

Like I mentioned earlier, the fact that it's cached for the duration of the operation is more of an implementation detail that developers need not worry about. What's important is that it's not cached AFTER. I still feel UNCACHED is the best description, but I'll change it to DONTCACHE for the next version just to avoid the overlap with other in-kernel uses.

Quoted 6 lines · Jens Axboe

Like I mentioned earlier, the fact that it's cached for the duration of the operation is more of an implementation detail that developers need not worry about. What's important is that it's not cached AFTER. I still feel UNCACHED is the best description, but I'll change it to DONTCACHE for the next version just to avoid the overlap with other in-kernel uses.

Regardless of the user API name, I like PG_streaming for the folio flag name.

Regardless of the user API name, I like PG_streaming for the folio flag name.

Quoted 6 lines · Jens Axboe

Like I mentioned earlier, the fact that it's cached for the duration of the operation is more of an implementation detail that developers need not worry about. What's important is that it's not cached AFTER. I still feel UNCACHED is the best description, but I'll change it to DONTCACHE for the next version just to avoid the overlap with other in-kernel uses.

Regardless of the user API name, I like PG_streaming for the folio flag name.

Quoted 10 lines · Matthew Wilcox

Quoted 6 lines · Jens Axboe

Like I mentioned earlier, the fact that it's cached for the duration of the operation is more of an implementation detail that developers need not worry about. What's important is that it's not cached AFTER. I still feel UNCACHED is the best description, but I'll change it to DONTCACHE for the next version just to avoid the overlap with other in-kernel uses.

Regardless of the user API name, I like PG_streaming for the folio flag name.

If we're throwing names in the ring, I'm partial to PG_dropbehind.

It's a term I think has been used to describe this type of behavior before; it juxtaposes nicely with readahead; it plainly names the action of what will happen to the page after the current IO operation against it has completed (i.e. pairs up with PG_reclaim).

Hi,

5 years ago I posted patches adding support for RWF_UNCACHED, as a way to do buffered IO that isn't page cache persistent. The approach back then was to have private pages for IO, and then get rid of them once IO was done. But that then runs into all the issues that O_DIRECT has, in terms of synchronizing with the page cache.

So here's a new approach to the same concent, but using the page cache as synchronization. That makes RWF_UNCACHED less special, in that it's just page cache IO, except it prunes the ranges once IO is completed.

Why do this, you may ask? The tldr is that device speeds are only getting faster, while reclaim is not. Doing normal buffered IO can be very unpredictable, and suck up a lot of resources on the reclaim side. This leads people to use O_DIRECT as a work-around, which has its own set of restrictions in terms of size, offset, and length of IO. It's also inherently synchronous, and now you need async IO as well. While the latter isn't necessarily a big problem as we have good options available there, it also should not be a requirement when all you want to do is read or write some data without caching.

Even on desktop type systems, a normal NVMe device can fill the entire page cache in seconds. On the big system I used for testing, there's a lot more RAM, but also a lot more devices. As can be seen in some of the results in the following patches, you can still fill RAM in seconds even when there's 1TB of it. Hence this problem isn't solely a "big hyperscaler system" issue, it's common across the board.

Common for both reads and writes with RWF_UNCACHED is that they use the page cache for IO. Reads work just like a normal buffered read would, with the only exception being that the touched ranges will get pruned after data has been copied. For writes, the ranges will get writeback kicked off before the syscall returns, and then writeback completion will prune the range. Hence writes aren't synchronous, and it's easy to pipeline writes using RWF_UNCACHED. Folios that aren't instantiated by RWF_UNCACHED IO are left untouched. This means you that uncached IO will take advantage of the page cache for uptodate data, but not leave anything it instantiated/created in cache.

File systems need to support this. The patches add support for the generic filemap helpers, and for iomap. Then ext4 and XFS are marked as supporting it. The last patch adds support for btrfs as well, lightly tested. The read side is already done by filemap, only the write side needs a bit of help. The amount of code here is really trivial, and the only reason the fs opt-in is necessary is to have an RWF_UNCACHED IO return -EOPNOTSUPP just in case the fs doesn't use either the generic paths or iomap. Adding "support" to other file systems should be trivial, most of the time just a one-liner adding FOP_UNCACHED to the fop_flags in the file_operations struct.

Performance results are in patch 8 for reads and patch 10 for writes, with the tldr being that I see about a 65% improvement in performance for both, with fully predictable IO times. CPU reduction is substantial as well, with no kswapd activity at all for reclaim when using uncached IO.

Using it from applications is trivial - just set RWF_UNCACHED for the read or write, using pwritev2(2) or preadv2(2). For io_uring, same thing, just set RWF_UNCACHED in sqe->rw_flags for a buffered read/write operation. And that's it.

Patches 1..7 are just prep patches, and should have no functional changes at all. Patch 8 adds support for the filemap path for RWF_UNCACHED reads, patch 11 adds support for filemap RWF_UNCACHED writes. In the below mentioned branch, there are then patches to adopt uncached reads and writes for ext4, xfs, and btrfs.

Passes full xfstests and fsx overnight runs, no issues observed. That includes the vm running the testing also using RWF_UNCACHED on the host. I'll post fsstress and fsx patches for RWF_UNCACHED separately. As far as I'm concerned, no further work needs doing here.

And git tree for the patches is here:

https://git.kernel.dk/cgit/linux/log/?h=buffered-uncached.8

 include/linux/fs.h             |  21 +++++-
 include/linux/page-flags.h     |   5 ++
 include/linux/pagemap.h        |  14 ++++
 include/trace/events/mmflags.h |   3 +-
 include/uapi/linux/fs.h        |   6 +-
 mm/filemap.c                   | 114 +++++++++++++++++++++++++++++----
 mm/readahead.c                 |  22 +++++--
 mm/swap.c                      |   2 +
 mm/truncate.c                  |  35 ++++++----
 9 files changed, 187 insertions(+), 35 deletions(-)

Since v5
- Skip invalidation in filemap_uncached_read() if the folio is dirty as well, retaining the uncached setting for later cleaning to do the actual invalidation.
- Use the same trylock approach in read invalidation as the writeback invalidation does.
- Swap order of patches 10 and 11 to fix a bisection issue.
- Split core mm changes and fs series patches. Once the generic side has been approved, I'll send out the fs series separately.
- Rebase on 6.13-rc1

Signature · 1 line

Jens Axboe

Hi Jens,

I ran a couple of variants of FIO to check how this patchset affects the FIO numbers. My other motivation with this patchset is to check if it improves the scalability issues that were discussed in [1] and [2]. But for now, here are some initial numbers.

Also note that I am using your buffered-uncached.8 branch from
https://git.kernel.dk/cgit/linux/log/?h=buffered-uncached.8 that has changes to enable uncached buffered IO for EXT4 and block devices.

In the below reported numbers,
'base' means kernel from buffered-uncached.8 branch and
'patched' means kernel from buffered-uncached.8 branch + above shown FIO change

FIO on EXT4 partitions
======================

nvme1n1     259:12   0   3.5T  0 disk 
??nvme1n1p1 259:13   0 894.3G  0 part /mnt1
??nvme1n1p2 259:14   0 894.3G  0 part /mnt2
??nvme1n1p3 259:15   0 894.3G  0 part /mnt3
??nvme1n1p4 259:16   0 894.1G  0 part /mnt4

fio -directory=/mnt4/ -direct=0 -thread -size=3G -rw=rw -rwmixwrite=30 --norandommap --randrepeat=0 -ioengine=pvsync2 -bs=64k -numjobs=252 -runtime=3600 --time_based -group_reporting -name=mytest fio -directory=/mnt3/ -direct=0 -thread -size=3G -rw=rw -rwmixwrite=30 --norandommap --randrepeat=0 -ioengine=pvsync2 -bs=64k -numjobs=252 -runtime=3600 --time_based -group_reporting -name=mytest fio -directory=/mnt1/ -direct=0 -thread -size=3G -rw=rw -rwmixwrite=30 --norandommap --randrepeat=0 -ioengine=pvsync2 -bs=64k -numjobs=252 -runtime=3600 --time_based -group_reporting -name=mytest fio -directory=/mnt2/ -direct=0 -thread -size=3G -rw=rw -rwmixwrite=30 --norandommap --randrepeat=0 -ioengine=pvsync2 -bs=64k -numjobs=252 -runtime=3600 --time_based -group_reporting -name=mytest

Four NVME devices are formatted with EXT4 and four parallel FIO instances are run on them with the options as shown above.

FIO output looks like this:

base:

   READ: bw=1233MiB/s (1293MB/s), 1233MiB/s-1233MiB/s (1293MB/s-1293MB/s), io=4335GiB (4654GB), run=3600097-3600097msec

WRITE: bw=529MiB/s (554MB/s), 529MiB/s-529MiB/s (554MB/s-554MB/s), io=1858GiB (1995GB), run=3600097-3600097msec
READ: bw=1248MiB/s (1308MB/s), 1248MiB/s-1248MiB/s (1308MB/s-1308MB/s), io=4387GiB (4710GB), run=3600091-3600091msec
WRITE: bw=535MiB/s (561MB/s), 535MiB/s-535MiB/s (561MB/s-561MB/s), io=1880GiB (2019GB), run=3600091-3600091msec
READ: bw=1235MiB/s (1294MB/s), 1235MiB/s-1235MiB/s (1294MB/s-1294MB/s), io=4340GiB (4660GB), run=3600094-3600094msec
WRITE: bw=529MiB/s (555MB/s), 529MiB/s-529MiB/s (555MB/s-555MB/s), io=1860GiB (1997GB), run=3600094-3600094msec
READ: bw=1234MiB/s (1294MB/s), 1234MiB/s-1234MiB/s (1294MB/s-1294MB/s), io=4337GiB (4657GB), run=3600093-3600093msec
WRITE: bw=529MiB/s (554MB/s), 529MiB/s-529MiB/s (554MB/s-554MB/s), io=1859GiB (1996GB), run=3600093-3600093msec

patched:

   READ: bw=1400MiB/s (1469MB/s), 1400MiB/s-1400MiB/s (1469MB/s-1469MB/s), io=4924GiB (5287GB), run=3600100-3600100msec

WRITE: bw=600MiB/s (629MB/s), 600MiB/s-600MiB/s (629MB/s-629MB/s), io=2110GiB (2266GB), run=3600100-3600100msec
READ: bw=1395MiB/s (1463MB/s), 1395MiB/s-1395MiB/s (1463MB/s-1463MB/s), io=4904GiB (5266GB), run=3600148-3600148msec
WRITE: bw=598MiB/s (627MB/s), 598MiB/s-598MiB/s (627MB/s-627MB/s), io=2102GiB (2257GB), run=3600148-3600148msec
READ: bw=1385MiB/s (1452MB/s), 1385MiB/s-1385MiB/s (1452MB/s-1452MB/s), io=4868GiB (5227GB), run=3600136-3600136msec
WRITE: bw=594MiB/s (622MB/s), 594MiB/s-594MiB/s (622MB/s-622MB/s), io=2087GiB (2241GB), run=3600136-3600136msec
READ: bw=1376MiB/s (1443MB/s), 1376MiB/s-1376MiB/s (1443MB/s-1443MB/s), io=4837GiB (5194GB), run=3600145-3600145msec
WRITE: bw=590MiB/s (618MB/s), 590MiB/s-590MiB/s (618MB/s-618MB/s), io=2073GiB (2226GB), run=3600145-3600145msec

FIO on block devices
====================

nvme1n1     259:12   0   3.5T  0 disk 
??nvme1n1p1 259:13   0 894.3G  0 part 
??nvme1n1p2 259:14   0 894.3G  0 part 
??nvme1n1p3 259:15   0 894.3G  0 part 
??nvme1n1p4 259:16   0 894.1G  0 part 

fio -filename=/dev/nvme1n1p4 -direct=0 -thread -size=800G -rw=rw -rwmixwrite=30 --norandommap --randrepeat=0 -ioengine=pvsync2 -bs=64k -numjobs=252 -runtime=3600 --time_based -group_reporting -name=mytest fio -filename=/dev/nvme1n1p2 -direct=0 -thread -size=800G -rw=rw -rwmixwrite=30 --norandommap --randrepeat=0 -ioengine=pvsync2 -bs=64k -numjobs=252 -runtime=3600 --time_based -group_reporting -name=mytest fio -filename=/dev/nvme1n1p1 -direct=0 -thread -size=800G -rw=rw -rwmixwrite=30 --norandommap --randrepeat=0 -ioengine=pvsync2 -bs=64k -numjobs=252 -runtime=3600 --time_based -group_reporting -name=mytest fio -filename=/dev/nvme1n1p3 -direct=0 -thread -size=800G -rw=rw -rwmixwrite=30 --norandommap --randrepeat=0 -ioengine=pvsync2 -bs=64k -numjobs=252 -runtime=3600 --time_based -group_reporting -name=mytest

Four instances of FIO are run on four different NVME block devices with the options as shown above.

base:

   READ: bw=8712MiB/s (9135MB/s), 8712MiB/s-8712MiB/s (9135MB/s-9135MB/s), io=29.9TiB (32.9TB), run=3600011-3600011msec

WRITE: bw=3734MiB/s (3915MB/s), 3734MiB/s-3734MiB/s (3915MB/s-3915MB/s), io=12.8TiB (14.1TB), run=3600011-3600011msec
READ: bw=8727MiB/s (9151MB/s), 8727MiB/s-8727MiB/s (9151MB/s-9151MB/s), io=30.0TiB (32.9TB), run=3600005-3600005msec
WRITE: bw=3740MiB/s (3922MB/s), 3740MiB/s-3740MiB/s (3922MB/s-3922MB/s), io=12.8TiB (14.1TB), run=3600005-3600005msec
READ: bw=8701MiB/s (9123MB/s), 8701MiB/s-8701MiB/s (9123MB/s-9123MB/s), io=29.9TiB (32.8TB), run=3600004-3600004msec
WRITE: bw=3729MiB/s (3910MB/s), 3729MiB/s-3729MiB/s (3910MB/s-3910MB/s), io=12.8TiB (14.1TB), run=3600004-3600004msec
READ: bw=8706MiB/s (9128MB/s), 8706MiB/s-8706MiB/s (9128MB/s-9128MB/s), io=29.9TiB (32.9TB), run=3600005-3600005msec
WRITE: bw=3731MiB/s (3913MB/s), 3731MiB/s-3731MiB/s (3913MB/s-3913MB/s), io=12.8TiB (14.1TB), run=3600005-3600005msec

patched:

   READ: bw=1844MiB/s (1933MB/s), 1844MiB/s-1844MiB/s (1933MB/s-1933MB/s), io=6500GiB (6980GB), run=3610641-3610641msec

WRITE: bw=790MiB/s (828MB/s), 790MiB/s-790MiB/s (828MB/s-828MB/s), io=2786GiB (2991GB), run=3610642-3610642msec
READ: bw=1753MiB/s (1838MB/s), 1753MiB/s-1753MiB/s (1838MB/s-1838MB/s), io=6235GiB (6695GB), run=3641973-3641973msec
WRITE: bw=751MiB/s (788MB/s), 751MiB/s-751MiB/s (788MB/s-788MB/s), io=2672GiB (2869GB), run=3641969-3641969msec
READ: bw=1078MiB/s (1130MB/s), 1078MiB/s-1078MiB/s (1130MB/s-1130MB/s), io=3788GiB (4068GB), run=3600007-3600007msec
WRITE: bw=462MiB/s (484MB/s), 462MiB/s-462MiB/s (484MB/s-484MB/s), io=1624GiB (1743GB), run=3600007-3600007msec
READ: bw=1752MiB/s (1838MB/s), 1752MiB/s-1752MiB/s (1838MB/s-1838MB/s), io=6234GiB (6694GB), run=3642657-3642657msec
WRITE: bw=751MiB/s (788MB/s), 751MiB/s-751MiB/s (788MB/s-788MB/s), io=2672GiB (2869GB), run=3642622-3642622msec

While FIO on FS shows improvement, FIO on block shows numbers going down. Is this expected or am I missing enabling anything else for the block option?