mm, netfs: Provide a means of invalidation without using launder_folio
Subsystems (10)
David Howells
maintainerRFC PATCH 1/1 mm, netfs: Provide a means of invalidation without using launder_folio
Implement a replacement for launder_folio[1]. The key feature of invalidate_inode_pages2() is that it locks each folio individually, unmaps it to prevent mmap'd accesses interfering and calls the ->launder_folio() address_space op to flush it. This has problems: firstly, each folio is written individually as one or more small writes; secondly, adjacent folios cannot be added so easily into the laundry; thirdly, it's yet another op to implement.
Here's a bit of a hacked together solution which should probably be moved to mm/:
Use the mmap lock to cause future faulting to wait, then unmap all the folios if we have mmaps, then, conditionally, use ->writepages() to flush any dirty data back and then discard all pages. The caller needs to hold a lock to prevent ->write_iter() getting underfoot.
Note that this does not prevent ->read_iter() from accessing the file whilst we do this since that may operate without locking.
We also have the writeback_control available and so have the opportunity to set a flag in it to tell the filesystem that we're doing an invalidation.
--- fs/netfs/misc.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++ include/linux/netfs.h | 3 ++ mm/memory.c | 3 +- 3 files changed, 61 insertions(+), 1 deletion(-)
fs/netfs/misc.c +56 -0
@@ -250,3 +250,59 @@ bool netfs_release_folio(struct folio *folio, gfp_t gfp)return true; }EXPORT_SYMBOL (netfs_release_folio); + +extern void unmap_mapping_range_tree (struct rb_root_cached *root, +pgoff_t first_index, +pgoff_t last_index, +struct zap_details *details); + +/** +* netfs_invalidate_inode - Invalidate/forcibly write back an inode's pagecache +* @inode: The inode to flush +* @flush: Set to write back rather than simply invalidate. +* +* Invalidate all the folios on an inode, possibly writing them back first. +* Whilst the operation is undertaken, the mmap lock is held to prevent +* ->fault() from reinstalling the folios. The caller must hold a lock on the +* inode sufficient to prevent ->write_iter() from dirtying more folios. +*/ +int netfs_invalidate_inode (struct inode *inode,bool flush) +{ +struct address_space *mapping = inode->i_mapping; + +if (!mapping || !mapping->nrpages) + goto out; + +/* Prevent folios from being faulted in. */ +i_mmap_lock_write (mapping); + +if (!mapping->nrpages) + goto unlock; + +/* Assume there are probably PTEs only if there are mmaps. */ +if (unlikely (!RB_EMPTY_ROOT (&mapping->i_mmap.rb_root))) { +struct zap_details details = { }; + +unmap_mapping_range_tree (&mapping->i_mmap,0 ,LLONG_MAX , &details); + } + +/* Write back the data if we're asked to. */ +if (flush) { +struct writeback_control wbc = { + .sync_mode =WB_SYNC_ALL , + .nr_to_write =LONG_MAX , + .range_start =0 , + .range_end =LLONG_MAX , + }; + +filemap_fdatawrite_wbc (mapping, &wbc); + } + +/* Wait for writeback to complete on all folios and discard. */ +truncate_inode_pages_range (mapping,0 ,LLONG_MAX ); + +unlock: +i_mmap_unlock_write (mapping); +out: +return filemap_check_errors (mapping); +}
include/linux/netfs.h +3 -0
@@ -400,6 +400,9 @@ ssize_t netfs_buffered_write_iter_locked(struct kiocb *iocb, struct iov_iter *frssize_t netfs_unbuffered_write_iter (struct kiocb *iocb,struct iov_iter *from);ssize_t netfs_file_write_iter (struct kiocb *iocb,struct iov_iter *from); +/* High-level invalidation API */ +int netfs_invalidate_inode (struct inode *inode,bool flush); +/* Address operations API */ struct readahead_control ;void netfs_readahead (struct readahead_control *);
mm/memory.c +2 -1
@@ -3665,7 +3665,7 @@ static void unmap_mapping_range_vma(struct vm_area_struct *vma,zap_page_range_single (vma, start_addr, end_addr - start_addr, details); } -static inline void unmap_mapping_range_tree (struct rb_root_cached *root, +inline void unmap_mapping_range_tree (struct rb_root_cached *root,pgoff_t first_index,pgoff_t last_index,struct zap_details *details)
@@ -3685,6 +3685,7 @@ static inline void unmap_mapping_range_tree(struct rb_root_cached *root, details); } } +EXPORT_SYMBOL_GPL (unmap_mapping_range_tree);/** * unmap_mapping_folio() - Unmap single folio from processes.
Quoted 174 lines · David Howells
Implement a replacement for launder_folio[1]. The key feature of
invalidate_inode_pages2()is that it locks each folio individually, unmaps
it to prevent mmap'd accesses interfering and calls the -Quoted 1 line
launder_folio()
address_spaceop to flush it. This has problems: firstly, each folio is
written individually as one or more small writes; secondly, adjacent folios
cannot be added so easily into the laundry; thirdly, it's yet another op to
implement.Here's a bit of a hacked together solution which should probably be moved
tomm/:Use the mmap lock to cause future faulting to wait, then unmap all the
folios if we have mmaps, then, conditionally, use ->writepages() to flush
any dirty data back and then discard all pages. The caller needs to hold a
lock to prevent ->write_iter() getting underfoot.Note that this does not prevent ->read_iter() from accessing the file whilst we do this since that may operate without locking.
We also have the
writeback_controlavailable and so have the opportunity to
set a flag in it to tell the filesystem that we're doing an invalidation.Signed-off-by David Howells <[email protected]>cc Matthew Wilcox <[email protected]>cc Miklos Szeredi <[email protected]>cc Trond Myklebust <[email protected]>cc Christoph Hellwig <[email protected]>cc Andrew Morton <[email protected]>cc Alexander Viro <[email protected]>cc Christian Brauner <[email protected]>cc Jeff Layton <[email protected]>Link:
https://lore.kernel.org/r/[email protected]/
[1]
---
fs/netfs/misc.c | 56
++++++++++++++++++++++++++++++++++++++++++++++++++
include/linux/netfs.h | 3 ++
mm/memory.c | 3 +-3 files changed, 61 insertions(+), 1 deletion(-)fs/netfs/misc.c +0
-0@@ -250,3 +250,59 @@ bool netfs_release_folio(struct folio *folio, gfp_t gfp) return true; }EXPORT_SYMBOL(netfs_release_folio);
fs/netfs/misc.c +8
-0+ +extern void unmap_mapping_range_tree (struct rb_root_cached *root, + pgoff_t first_index, + pgoff_t last_index, + struct zap_details *details); + +/** +* netfs_invalidate_inode - Invalidate/forcibly write back an inode's pagecache
+ * @inode: The inode to flush
+ * @flush: Set to write back rather than simply invalidate.
+ *
+ * Invalidate all the folios on an inode, possibly writing them back first.
+ * Whilst the operation is undertaken, the mmap lock is held to prevent
+ * ->fault() from reinstalling the folios. The caller must hold a lock on the
+ * inode sufficient to prevent ->write_iter() from dirtying more folios.
+ */fs/netfs/misc.c +14
-0+int netfs_invalidate_inode (struct inode *inode,bool flush) +{ +struct address_space *mapping = inode->i_mapping; + +if (!mapping || !mapping->nrpages) + goto out; + +/* Prevent folios from being faulted in. */ +i_mmap_lock_write (mapping); + +if (!mapping->nrpages) + goto unlock; + +/* Assume there are probably PTEs only if there are mmaps. */
fs/netfs/misc.c +4
-0+if (unlikely (!RB_EMPTY_ROOT (&mapping->i_mmap.rb_root))) { +struct zap_details details = { }; + + unmap_mapping_range_tree(&mapping->i_mmap, 0,
LLONG_MAX, &details);fs/netfs/misc.c +15
-0+ } + +/* Write back the data if we're asked to. */ +if (flush) { +struct writeback_control wbc = { + .sync_mode =WB_SYNC_ALL , + .nr_to_write =LONG_MAX , + .range_start =0 , + .range_end =LLONG_MAX , + }; + +filemap_fdatawrite_wbc (mapping, &wbc); + } + +/* Wait for writeback to complete on all folios and discard. */
fs/netfs/misc.c +7
-0+truncate_inode_pages_range (mapping,0 ,LLONG_MAX ); + +unlock: +i_mmap_unlock_write (mapping); +out: +return filemap_check_errors (mapping); +}include/linux/netfs.h +0
-0@@ -400,6 +400,9 @@ ssize_t netfs_buffered_write_iter_locked(structkiocb *iocb, struct
iov_iter*fr
ssize_tnetfs_unbuffered_write_iter(struct kiocb *iocb, struct iov_iter *from);
ssize_tnetfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from);include/linux/netfs.h +3
-0+/* High-level invalidation API */ +int netfs_invalidate_inode (struct inode *inode,bool flush); +/* Address operations API */
structreadahead_control;
voidnetfs_readahead(struct readahead_control *);mm/memory.c +0
-0@@ -3665,7 +3665,7 @@ static void unmap_mapping_range_vma(struct vm_area_struct *vma,
zap_page_range_single(vma, start_addr, end_addr - start_addr, details);
}
-static inline void unmap_mapping_range_tree(structrb_root_cached*root,
+inline voidunmap_mapping_range_tree(struct rb_root_cached *root, pgoff_t first_index, pgoff_t last_index, struct zap_details *details)mm/memory.c +1
-0@@ -3685,6 +3685,7 @@ static inline void unmap_mapping_range_tree(struct rb_root_cached *root, details); } } +EXPORT_SYMBOL_GPL (unmap_mapping_range_tree);
Quoted 4 lines
/**
*unmap_mapping_folio()- Unmap single folio from processes.
This is hardly a drop-in replacement for launder_page. The whole point of using invalidate_inode_pages2() was that it only requires taking the page locks, allowing us to use it in contexts such as nfs_release_file().
The above use of truncate_inode_pages_range() will require any caller to grab several locks in order to prevent data loss through races with write system calls.
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Matthew Wilcox
maintainerQuoted 14 lines · Trond Myklebust
Quoted 4 lines · David Howells
Implement a replacement for launder_folio[1]. The key feature of
invalidate_inode_pages2()is that it locks each folio individually, unmaps
it to prevent mmap'd accesses interfering and calls the -Quoted 8 lines
Quoted 1 line
launder_folio()
address_spaceop to flush it. This has problems: firstly, each folio is
written individually as one or more small writes; secondly, adjacent folios
cannot be added so easily into the laundry; thirdly, it's yet another op to
implement.
Quoted 8 lines
This is hardly a drop-in replacement for
launder_page. The whole point of usinginvalidate_inode_pages2()was that it only requires taking the page locks, allowing us to use it in contexts such asnfs_release_file().The above use of
truncate_inode_pages_range()will require any caller to grab several locks in order to prevent data loss through races with write system calls.
I don't understand why you need launder_folio now
that you have a page_mkwrite implementation (your commit e3db7691e9f3dff3289f64e3d98583e28afe03db used this as justification). Other filesystems (except the network filesystems that copied the NFS implementation) don't implement launder_folio.
David Howells
maintainerRFC PATCH v2 1/1 mm, netfs: Provide a means of invalidation without using launder_folio
In reply to David Howellsmm, netfs: Provide a means of invalidation without using launder_folio
Implement a replacement for launder_folio. The key feature of invalidate_inode_pages2() is that it locks each folio individually, unmaps it to prevent mmap'd accesses interfering and calls the ->launder_folio() address_space op to flush it. This has problems: firstly, each folio is written individually as one or more small writes; secondly, adjacent folios cannot be added so easily into the laundry; thirdly, it's yet another op to implement.
Instead, use the invalidate lock to cause anyone wanting to add a folio to the inode to wait, then unmap all the folios if we have mmaps, then, conditionally, use ->writepages() to flush any dirty data back and then discard all pages.
The invalidate lock prevents ->read_iter(), ->write_iter() and faulting through mmap all from adding pages for the duration.
--- include/linux/pagemap.h | 1 + mm/filemap.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+)
include/linux/pagemap.h +1 -0
@@ -40,6 +40,7 @@ int filemap_fdatawait_keep_errors(struct address_space *mapping);int filemap_fdatawait_range (struct address_space *,loff_t lstart,loff_t lend);int filemap_fdatawait_range_keep_errors (struct address_space *mapping,loff_t start_byte,loff_t end_byte); +int filemap_invalidate_inode (struct inode *inode,bool flush);static inline int filemap_fdatawait (struct address_space *mapping) {
mm/filemap.c +48 -0
54 lines not on this page · raw message
Matthew Wilcox
maintainermm/filemap.c bool filemap_release_folio(struct folio *folio, gfp_t gfp) · line 4154
Quoted 9 lines · David Howells
mm/filemap.c +9
-0+int filemap_invalidate_inode (struct inode *inode,bool flush) +{ +struct address_space *mapping = inode->i_mapping; + +if (!mapping || !mapping->nrpages) + goto out; + +/* Prevent new folios from being added to the inode. */ +filemap_invalidate_lock (mapping);
I'm kind of surprised that the callers wouldn't want to hold that lock over a call to this function. I guess you're working on the callers, so you'd know better than I would, but I would have used lockdep to assert that invalidate_lock was held.
mm/filemap.c bool filemap_release_folio(struct folio *folio, gfp_t gfp) · line 4161
Quoted 6 lines
mm/filemap.c +6
-0+if (!mapping->nrpages) + goto unlock; + +/* Assume there are probably PTEs only if there are mmaps. */ +if (unlikely (!RB_EMPTY_ROOT (&mapping->i_mmap.rb_root))) +unmap_mapping_pages (mapping,0 ,ULONG_MAX , false);
Is this optimisation worth it? We're already doing some expensive operations here, does saving cycling the i_mmap_lock really help anything? You'll note that unmap_mapping_pages() already does this check inside the lock.
David Howells
maintainerQuoted 7 lines · Matthew Wilcox
mm/filemap.c bool filemap_release_folio(struct folio *folio, gfp_t gfp) · line 4154
Quoted 2 lines
mm/filemap.c +2
-0+/* Prevent new folios from being added to the inode. */ +filemap_invalidate_lock (mapping);I'm kind of surprised that the callers wouldn't want to hold that lock over a call to this function. I guess you're working on the callers, so you'd know better than I would, but I would have used lockdep to assert that
invalidate_lockwas held.
I'm not sure. None of the places that look like they'd be calling this currently take that lock (though possibly they should).
Also, should I provide it with explicit range, I wonder?
Quoted 4 lines
mm/filemap.c bool filemap_release_folio(struct folio *folio, gfp_t gfp) · line 4161
Quoted 2 lines
mm/filemap.c +2
-0+if (unlikely (!RB_EMPTY_ROOT (&mapping->i_mmap.rb_root))) +unmap_mapping_pages (mapping,0 ,ULONG_MAX , false);Is this optimisation worth it?
Perhaps not.
David