Skip to contentSkip to conversation

mm, netfs: Provide a means of invalidation without using launder_folio

RFC PATCH Superseded David Howells 3 participants 6 messages Mar 27, 2024

Subsystems (10)
lore
Share & tools
lore

David Howells

author maintainer
#0 lore raw

RFC PATCH 1/1 mm, netfs: Provide a means of invalidation without using launder_folio

Implement a replacement for launder_folio[1]. The key feature of invalidate_inode_pages2() is that it locks each folio individually, unmaps it to prevent mmap'd accesses interfering and calls the ->launder_folio() address_space op to flush it. This has problems: firstly, each folio is written individually as one or more small writes; secondly, adjacent folios cannot be added so easily into the laundry; thirdly, it's yet another op to implement.

Here's a bit of a hacked together solution which should probably be moved to mm/:

Use the mmap lock to cause future faulting to wait, then unmap all the folios if we have mmaps, then, conditionally, use ->writepages() to flush any dirty data back and then discard all pages. The caller needs to hold a lock to prevent ->write_iter() getting underfoot.

Note that this does not prevent ->read_iter() from accessing the file whilst we do this since that may operate without locking.

We also have the writeback_control available and so have the opportunity to set a flag in it to tell the filesystem that we're doing an invalidation.

---
 fs/netfs/misc.c       |   56 ++++++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/netfs.h |    3 ++
 mm/memory.c           |    3 +-
 3 files changed, 61 insertions(+), 1 deletion(-)
fs/netfs/misc.c +56 -0
index bc1fc54fb724..774ce825fbec 100644
@@ -250,3 +250,59 @@ bool netfs_release_folio(struct folio *folio, gfp_t gfp)
 	return true;
 }
 EXPORT_SYMBOL(netfs_release_folio);
+
+extern void unmap_mapping_range_tree(struct rb_root_cached *root,
+				     pgoff_t first_index,
+				     pgoff_t last_index,
+				     struct zap_details *details);
+
+/**
+ * netfs_invalidate_inode - Invalidate/forcibly write back an inode's pagecache
+ * @inode: The inode to flush
+ * @flush: Set to write back rather than simply invalidate.
+ *
+ * Invalidate all the folios on an inode, possibly writing them back first.
+ * Whilst the operation is undertaken, the mmap lock is held to prevent
+ * ->fault() from reinstalling the folios.  The caller must hold a lock on the
+ * inode sufficient to prevent ->write_iter() from dirtying more folios.
+ */
+int netfs_invalidate_inode(struct inode *inode, bool flush)
+{
+	struct address_space *mapping = inode->i_mapping;
+
+	if (!mapping || !mapping->nrpages)
+		goto out;
+
+	/* Prevent folios from being faulted in. */
+	i_mmap_lock_write(mapping);
+
+	if (!mapping->nrpages)
+		goto unlock;
+
+	/* Assume there are probably PTEs only if there are mmaps. */
+	if (unlikely(!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root))) {
+		struct zap_details details = { };
+
+		unmap_mapping_range_tree(&mapping->i_mmap, 0, LLONG_MAX, &details);
+	}
+
+	/* Write back the data if we're asked to. */
+	if (flush) {
+		struct writeback_control wbc = {
+			.sync_mode	= WB_SYNC_ALL,
+			.nr_to_write	= LONG_MAX,
+			.range_start	= 0,
+			.range_end	= LLONG_MAX,
+		};
+
+		filemap_fdatawrite_wbc(mapping, &wbc);
+	}
+
+	/* Wait for writeback to complete on all folios and discard. */
+	truncate_inode_pages_range(mapping, 0, LLONG_MAX);
+
+unlock:
+	i_mmap_unlock_write(mapping);
+out:
+	return filemap_check_errors(mapping);
+}
include/linux/netfs.h +3 -0
index 298552f5122c..40dc34ee291d 100644
@@ -400,6 +400,9 @@ ssize_t netfs_buffered_write_iter_locked(struct kiocb *iocb, struct iov_iter *fr
 ssize_t netfs_unbuffered_write_iter(struct kiocb *iocb, struct iov_iter *from);
 ssize_t netfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from);
 
+/* High-level invalidation API */
+int netfs_invalidate_inode(struct inode *inode, bool flush);
+
 /* Address operations API */
 struct readahead_control;
 void netfs_readahead(struct readahead_control *);
mm/memory.c +2 -1
index f2bc6dd15eb8..106f32c7d7fb 100644
@@ -3665,7 +3665,7 @@ static void unmap_mapping_range_vma(struct vm_area_struct *vma,
 	zap_page_range_single(vma, start_addr, end_addr - start_addr, details);
 }
 
-static inline void unmap_mapping_range_tree(struct rb_root_cached *root,
+inline void unmap_mapping_range_tree(struct rb_root_cached *root,
 					    pgoff_t first_index,
 					    pgoff_t last_index,
 					    struct zap_details *details)
@@ -3685,6 +3685,7 @@ static inline void unmap_mapping_range_tree(struct rb_root_cached *root,
 				details);
 	}
 }
+EXPORT_SYMBOL_GPL(unmap_mapping_range_tree);
 
 /**
  * unmap_mapping_folio() - Unmap single folio from processes.
#1 lore raw
Quoted 174 lines · David Howells

Implement a replacement for launder_folio[1].  The key feature of invalidate_inode_pages2() is that it locks each folio individually, unmaps
it to prevent mmap'd accesses interfering and calls the -

Quoted 1 line

launder_folio()

address_space op to flush it.  This has problems: firstly, each folio is
written individually as one or more small writes; secondly, adjacent folios
cannot be added so easily into the laundry; thirdly, it's yet another op to
implement.

Here's a bit of a hacked together solution which should probably be moved
to mm/:

Use the mmap lock to cause future faulting to wait, then unmap all the
folios if we have mmaps, then, conditionally, use ->writepages() to flush
any dirty data back and then discard all pages.  The caller needs to hold a
lock to prevent ->write_iter() getting underfoot.

Note that this does not prevent ->read_iter() from accessing the file whilst we do this since that may operate without locking.

We also have the writeback_control available and so have the opportunity to
set a flag in it to tell the filesystem that we're doing an invalidation.

Link:
https://lore.kernel.org/r/[email protected]/
[1]
---
 fs/netfs/misc.c       |   56
++++++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/netfs.h |    3 ++
 mm/memory.c           |    3 +-

 3 files changed, 61 insertions(+), 1 deletion(-)
fs/netfs/misc.c +0 -0
index bc1fc54fb724..774ce825fbec 100644
@@ -250,3 +250,59 @@ bool netfs_release_folio(struct folio *folio,
 gfp_t gfp)
  	return true;
  }

 EXPORT_SYMBOL(netfs_release_folio);

fs/netfs/misc.c +8 -0
+
+extern void unmap_mapping_range_tree(struct rb_root_cached *root,
+				     pgoff_t first_index,
+				     pgoff_t last_index,
+				     struct zap_details *details);
+
+/**
+ * netfs_invalidate_inode - Invalidate/forcibly write back an

inode's pagecache
+ * @inode: The inode to flush
+ * @flush: Set to write back rather than simply invalidate.
+ *
+ * Invalidate all the folios on an inode, possibly writing them back first.
+ * Whilst the operation is undertaken, the mmap lock is held to prevent
+ * ->fault() from reinstalling the folios.  The caller must hold a lock on the
+ * inode sufficient to prevent ->write_iter() from dirtying more folios.
+ */

fs/netfs/misc.c +14 -0
+int netfs_invalidate_inode(struct inode *inode, bool flush)
+{
+	struct address_space *mapping = inode->i_mapping;
+
+	if (!mapping || !mapping->nrpages)
+		goto out;
+
+	/* Prevent folios from being faulted in. */
+	i_mmap_lock_write(mapping);
+
+	if (!mapping->nrpages)
+		goto unlock;
+
+	/* Assume there are probably PTEs only if there are mmaps.

*/

fs/netfs/misc.c +4 -0
+	if (unlikely(!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root))) {
+		struct zap_details details = { };
+
+		unmap_mapping_range_tree(&mapping->i_mmap, 0,

LLONG_MAX, &details);

fs/netfs/misc.c +15 -0
+	}
+
+	/* Write back the data if we're asked to. */
+	if (flush) {
+		struct writeback_control wbc = {
+			.sync_mode	= WB_SYNC_ALL,
+			.nr_to_write	= LONG_MAX,
+			.range_start	= 0,
+			.range_end	= LLONG_MAX,
+		};
+
+		filemap_fdatawrite_wbc(mapping, &wbc);
+	}
+
+	/* Wait for writeback to complete on all folios and discard.

*/

fs/netfs/misc.c +7 -0
+	truncate_inode_pages_range(mapping, 0, LLONG_MAX);
+
+unlock:
+	i_mmap_unlock_write(mapping);
+out:
+	return filemap_check_errors(mapping);
+}
include/linux/netfs.h +0 -0
index 298552f5122c..40dc34ee291d 100644
@@ -400,6 +400,9 @@ ssize_t netfs_buffered_write_iter_locked(struct

kiocb *iocb, struct iov_iter *fr
 ssize_t netfs_unbuffered_write_iter(struct kiocb *iocb, struct iov_iter *from);
 ssize_t netfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from);

include/linux/netfs.h +3 -0
+/* High-level invalidation API */
+int netfs_invalidate_inode(struct inode *inode, bool flush);
+

 /* Address operations API */
 struct readahead_control;
 void netfs_readahead(struct readahead_control *);

mm/memory.c +0 -0
index f2bc6dd15eb8..106f32c7d7fb 100644
@@ -3665,7 +3665,7 @@ static void unmap_mapping_range_vma(struct
 vm_area_struct *vma,

  zap_page_range_single(vma, start_addr, end_addr - start_addr, details);
 }

-static inline void unmap_mapping_range_tree(struct rb_root_cached *root,
+inline void unmap_mapping_range_tree(struct rb_root_cached *root, pgoff_t first_index, pgoff_t last_index, struct zap_details *details)

mm/memory.c +1 -0
@@ -3685,6 +3685,7 @@ static inline void
 unmap_mapping_range_tree(struct rb_root_cached *root,
  				details);
  	}
  }
+EXPORT_SYMBOL_GPL(unmap_mapping_range_tree);
Quoted 4 lines


 /**
  * unmap_mapping_folio() - Unmap single folio from processes.

This is hardly a drop-in replacement for launder_page. The whole point of using invalidate_inode_pages2() was that it only requires taking the page locks, allowing us to use it in contexts such as nfs_release_file().

The above use of truncate_inode_pages_range() will require any caller to grab several locks in order to prevent data loss through races with write system calls.

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
maintainer
#2 lore raw
Quoted 14 lines · Trond Myklebust
Quoted 4 lines · David Howells

Implement a replacement for launder_folio[1].  The key feature of invalidate_inode_pages2() is that it locks each folio individually, unmaps
it to prevent mmap'd accesses interfering and calls the -

Quoted 8 lines
Quoted 1 line

launder_folio()

address_space op to flush it.  This has problems: firstly, each folio is
written individually as one or more small writes; secondly, adjacent folios
cannot be added so easily into the laundry; thirdly, it's yet another op to
implement.

Quoted 8 lines

This is hardly a drop-in replacement for launder_page. The whole point of using invalidate_inode_pages2() was that it only requires taking the page locks, allowing us to use it in contexts such as nfs_release_file().

The above use of truncate_inode_pages_range() will require any caller to grab several locks in order to prevent data loss through races with write system calls.

I don't understand why you need launder_folio now
that you have a page_mkwrite implementation (your commit e3db7691e9f3dff3289f64e3d98583e28afe03db used this as justification). Other filesystems (except the network filesystems that copied the NFS implementation) don't implement launder_folio.

author maintainer
#3 lore raw

RFC PATCH v2 1/1 mm, netfs: Provide a means of invalidation without using launder_folio

In reply to David Howells

mm, netfs: Provide a means of invalidation without using launder_folio

Implement a replacement for launder_folio. The key feature of invalidate_inode_pages2() is that it locks each folio individually, unmaps it to prevent mmap'd accesses interfering and calls the ->launder_folio() address_space op to flush it. This has problems: firstly, each folio is written individually as one or more small writes; secondly, adjacent folios cannot be added so easily into the laundry; thirdly, it's yet another op to implement.

Instead, use the invalidate lock to cause anyone wanting to add a folio to the inode to wait, then unmap all the folios if we have mmaps, then, conditionally, use ->writepages() to flush any dirty data back and then discard all pages.

The invalidate lock prevents ->read_iter(), ->write_iter() and faulting through mmap all from adding pages for the duration.

---
 include/linux/pagemap.h |    1 +
 mm/filemap.c            |   48 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 49 insertions(+)
include/linux/pagemap.h +1 -0
index 2df35e65557d..4eb3d4177a53 100644
@@ -40,6 +40,7 @@ int filemap_fdatawait_keep_errors(struct address_space *mapping);
 int filemap_fdatawait_range(struct address_space *, loff_t lstart, loff_t lend);
 int filemap_fdatawait_range_keep_errors(struct address_space *mapping,
 		loff_t start_byte, loff_t end_byte);
+int filemap_invalidate_inode(struct inode *inode, bool flush);
 
 static inline int filemap_fdatawait(struct address_space *mapping)
 {
mm/filemap.c +48 -0

54 lines not on this page · raw message

maintainer
#4 lore raw

mm/filemap.c bool filemap_release_folio(struct folio *folio, gfp_t gfp) · line 4154

Quoted 9 lines · David Howells
mm/filemap.c +9 -0
+int filemap_invalidate_inode(struct inode *inode, bool flush)
+{
+	struct address_space *mapping = inode->i_mapping;
+
+	if (!mapping || !mapping->nrpages)
+		goto out;
+
+	/* Prevent new folios from being added to the inode. */
+	filemap_invalidate_lock(mapping);

I'm kind of surprised that the callers wouldn't want to hold that lock over a call to this function. I guess you're working on the callers, so you'd know better than I would, but I would have used lockdep to assert that invalidate_lock was held.

mm/filemap.c bool filemap_release_folio(struct folio *folio, gfp_t gfp) · line 4161

Quoted 6 lines
mm/filemap.c +6 -0
+	if (!mapping->nrpages)
+		goto unlock;
+
+	/* Assume there are probably PTEs only if there are mmaps. */
+	if (unlikely(!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root)))
+		unmap_mapping_pages(mapping, 0, ULONG_MAX, false);

Is this optimisation worth it? We're already doing some expensive operations here, does saving cycling the i_mmap_lock really help anything? You'll note that unmap_mapping_pages() already does this check inside the lock.

author maintainer
#5 lore raw
Quoted 7 lines · Matthew Wilcox

mm/filemap.c bool filemap_release_folio(struct folio *folio, gfp_t gfp) · line 4154

Quoted 2 lines
mm/filemap.c +2 -0
+	/* Prevent new folios from being added to the inode. */
+	filemap_invalidate_lock(mapping);

I'm kind of surprised that the callers wouldn't want to hold that lock over a call to this function. I guess you're working on the callers, so you'd know better than I would, but I would have used lockdep to assert that invalidate_lock was held.

I'm not sure. None of the places that look like they'd be calling this currently take that lock (though possibly they should).

Also, should I provide it with explicit range, I wonder?

Quoted 4 lines

mm/filemap.c bool filemap_release_folio(struct folio *folio, gfp_t gfp) · line 4161

Quoted 2 lines
mm/filemap.c +2 -0
+	if (unlikely(!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root)))
+		unmap_mapping_pages(mapping, 0, ULONG_MAX, false);

Is this optimisation worth it?

Perhaps not.

David