mm, netfs, fscache: Stop read optimisation when folio removed from pagecache
Subsystems (18)
David Howells
maintainerv6 cover mm, netfs, fscache: Stop read optimisation when folio removed from pagecache
Hi Willy,
Is this okay by you? You said you wanted to look at the remaining uses of page_has_private(), of which there are then three after these patches, not counting folio_has_private():
arch/s390/kernel/uv.c: if (page_has_private(page))
mm/khugepaged.c: 1 + page_mapcount(page) + page_has_private(page)) {
mm/migrate_device.c: extra += 1 + page_has_private(page);--
I've split the folio_has_private()/filemap_release_folio() call pair merging into its own patch, separate from the actual bugfix and pulled out the folio_needs_release() function into mm/internal.h and made filemap_release_folio() use it. I've also got rid of the bit clearances from the network filesystem evict_inode functions as they doesn't seem to be necessary.
Note that the last vestiges of try_to_release_page() got swept away, so I rebased and dealt with that. One comment remained, which is removed by the first patch.
David
Changes:
========
ver #6)
- Drop the third patch which removes a duplicate check in vmscan().
ver #5)
- Rebased on linus/master. try_to_release_page() has now been entirely replaced by filemap_release_folio(), barring one comment.
- Cleaned up some pairs in ext4.
ver #4)
- Split has_private/release call pairs into own patch.
- Moved folio_needs_release() to mm/internal.h and removed open-coded version from filemap_release_folio().
- Don't need to clear AS_RELEASE_ALWAYS in ->evict_inode().
- Added experimental patch to reduce shrink_folio_list().
ver #3)
- Fixed mapping_clear_release_always() to use clear_bit() not set_bit().
- Moved a '&&' to the correct line.
ver #2)
- Rewrote entirely according to Willy's suggestion[1].
---
%(shortlog)s
%(diffstat)s
David Howells (2):
mm: Merge folio_has_private()/filemap_release_folio() call pairs mm, netfs, fscache: Stop read optimisation when folio removed from
pagecache
fs/9p/cache.c | 2 ++ fs/afs/internal.h | 2 ++ fs/cachefiles/namei.c | 2 ++ fs/ceph/cache.c | 2 ++ fs/cifs/fscache.c | 2 ++ fs/ext4/move_extent.c | 12 ++++-------- fs/splice.c | 3 +-- include/linux/pagemap.h | 16 ++++++++++++++++ mm/filemap.c | 2 ++ mm/huge_memory.c | 3 +-- mm/internal.h | 11 +++++++++++ mm/khugepaged.c | 3 +-- mm/memory-failure.c | 8 +++----- mm/migrate.c | 3 +-- mm/truncate.c | 6 ++---- mm/vmscan.c | 8 ++++---- 16 files changed, 56 insertions(+), 29 deletions(-)
David Howells
maintainerPATCH v6 1/2 mm: Merge folio_has_private()/filemap_release_folio() call pairs
Make filemap_release_folio() check folio_has_private(). Then, in most cases, where a call to folio_has_private() is immediately followed by a call to filemap_release_folio(), we can get rid of the test in the pair.
There are a couple of sites in mm/vscan.c that this can't so easily be done. In shrink_folio_list(), there are actually three cases (something different is done for incompletely invalidated buffers), but filemap_release_folio() elides two of them.
In shrink_active_list(), we don't have have the folio lock yet, so the check allows us to avoid locking the page unnecessarily.
A wrapper function to check if a folio needs release is provided for those places that still need to do it in the mm/ directory. This will acquire additional parts to the condition in a future patch.
After this, the only remaining caller of folio_has_private() outside of mm/ is a check in fuse.
---
Notes:
ver #5)
- Rebased on linus/master. try_to_release_page() has now been entirely replaced by filemap_release_folio(), barring one comment.
- Cleaned up some pairs in ext4.
ver #4)
- Split from fscache fix.
- Moved folio_needs_release() to mm/internal.h and removed open-coded version from filemap_release_folio().
ver #3)
- Fixed mapping_clear_release_always() to use clear_bit() not set_bit().
- Moved a '&&' to the correct line.
ver #2)
- Rewrote entirely according to Willy's suggestion[1].
fs/ext4/move_extent.c | 12 ++++-------- fs/splice.c | 3 +-- mm/filemap.c | 2 ++ mm/huge_memory.c | 3 +-- mm/internal.h | 8 ++++++++ mm/khugepaged.c | 3 +-- mm/memory-failure.c | 8 +++----- mm/migrate.c | 3 +-- mm/truncate.c | 6 ++---- mm/vmscan.c | 8 ++++---- 10 files changed, 27 insertions(+), 29 deletions(-)
fs/ext4/move_extent.c +4 -8
@@ -339,10 +339,8 @@ move_extent_per_page(struct file *o_filp, struct inode *donor_inode,ext4_double_up_write_data_sem (orig_inode, donor_inode); goto data_copy; } -if ((folio_has_private (folio[0 ]) && - !filemap_release_folio (folio[0 ],0 )) || - (folio_has_private (folio[1 ]) && - !filemap_release_folio (folio[1 ],0 ))) { +if (!filemap_release_folio (folio[0 ],0 ) || + !filemap_release_folio (folio[1 ],0 )) { *err = -EBUSY ; goto drop_data_sem; }
@@ -361,10 +359,8 @@ move_extent_per_page(struct file *o_filp, struct inode *donor_inode,/* At this point all buffers in range are uptodate, old mapping layout * is no longer required, try to drop it now. */ -if ((folio_has_private (folio[0 ]) && - !filemap_release_folio (folio[0 ],0 )) || - (folio_has_private (folio[1 ]) && - !filemap_release_folio (folio[1 ],0 ))) { +if (!filemap_release_folio (folio[0 ],0 ) || + !filemap_release_folio (folio[1 ],0 )) { *err = -EBUSY ; goto unlock_folios; }
fs/splice.c +1 -2
@@ -65,8 +65,7 @@ static bool page_cache_pipe_buf_try_steal(struct pipe_inode_info *pipe,*/ folio_wait_writeback (folio); -if (folio_has_private (folio) && - !filemap_release_folio (folio,GFP_KERNEL )) +if (!filemap_release_folio (folio,GFP_KERNEL )) goto out_unlock;/*
mm/filemap.c +2 -0
@@ -3960,6 +3960,8 @@ bool filemap_release_folio(struct folio *folio, gfp_t gfp)struct address_space *const mapping = folio->mapping;BUG_ON (!folio_test_locked (folio)); +if (!folio_needs_release (folio)) +return true;if (folio_test_writeback (folio))return false;
mm/huge_memory.c +1 -2
@@ -2702,8 +2702,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) gfp =current_gfp_context (mapping_gfp_mask (mapping) &GFP_RECLAIM_MASK ); -if (folio_test_private (folio) && - !filemap_release_folio (folio, gfp)) { +if (!filemap_release_folio (folio, gfp)) { ret = -EBUSY ; goto out; }
mm/internal.h +8 -0
@@ -163,6 +163,14 @@ static inline void set_page_refcounted(struct page *page)set_page_count (page,1 ); } +/* +* Return true if a folio needs ->release_folio() calling upon it. +*/ +static inline bool folio_needs_release (struct folio *folio) +{ +return folio_has_private (folio); +} +extern unsigned long highest_memmap_pfn;/*
mm/khugepaged.c +1 -2
@@ -1944,8 +1944,7 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, goto out_unlock; } -if (folio_has_private (folio) && - !filemap_release_folio (folio,GFP_KERNEL )) { +if (!filemap_release_folio (folio,GFP_KERNEL )) { result =SCAN_PAGE_HAS_PRIVATE ;folio_putback_lru (folio); goto out_unlock;
mm/memory-failure.c +3 -5
@@ -843,14 +843,12 @@ static int truncate_error_page(struct page *p, unsigned long pfn,struct folio *folio =page_folio (p);int err = mapping->a_ops->error_remove_page (mapping, p); -if (err !=0 ) { +if (err !=0 )pr_info ("%#lx: Failed to punch page: %d\n" , pfn, err); - }else if (folio_has_private (folio) && - !filemap_release_folio (folio,GFP_NOIO )) { +else if (!filemap_release_folio (folio,GFP_NOIO ))pr_info ("%#lx: failed to release buffers\n" , pfn); - }else { +else ret =MF_RECOVERED ; - } }else {/* * If the file system doesn't support it just invalidate
mm/migrate.c +1 -2
@@ -915,8 +915,7 @@ static int fallback_migrate_folio(struct address_space *mapping,* Buffers may be managed in a filesystem specific way. * We must have no buffers or drop them. */ -if (folio_test_private (src) && - !filemap_release_folio (src,GFP_KERNEL )) +if (!filemap_release_folio (src,GFP_KERNEL ))return mode ==MIGRATE_SYNC ? -EAGAIN : -EBUSY ;return migrate_folio (mapping, dst, src, mode);
mm/truncate.c +2 -4
@@ -19,7 +19,6 @@#include <linux/highmem.h> #include <linux/pagevec.h> #include <linux/task_io_accounting_ops.h> -#include <linux/buffer_head.h> /* grr. try_to_release_page */ #include <linux/shmem_fs.h> #include <linux/rmap.h> #include "internal.h"
@@ -276,7 +275,7 @@ static long mapping_evict_folio(struct address_space *mapping,if (folio_ref_count (folio) >folio_nr_pages (folio) +folio_has_private (folio) +1 )return 0 ; -if (folio_has_private (folio) && !filemap_release_folio (folio,0 )) +if (!filemap_release_folio (folio,0 ))return 0 ;return remove_mapping (mapping, folio);
@@ -573,8 +572,7 @@ static int invalidate_complete_folio2(struct address_space *mapping,if (folio->mapping != mapping)return 0 ; -if (folio_has_private (folio) && - !filemap_release_folio (folio,GFP_KERNEL )) +if (!filemap_release_folio (folio,GFP_KERNEL ))return 0 ;spin_lock (&mapping->host->i_lock);
mm/vmscan.c +4 -4
@@ -1996,7 +1996,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,* (refcount == 1) it can be freed. Otherwise, leave * the folio on the LRU so it is swappable. */ -if (folio_has_private (folio)) { +if (folio_needs_release (folio)) {if (!filemap_release_folio (folio, sc->gfp_mask)) goto activate_locked;if (!mapping &&folio_ref_count (folio) ==1 ) {
@@ -2641,9 +2641,9 @@ static void shrink_active_list(unsigned long nr_to_scan, }if (unlikely (buffer_heads_over_limit)) { -if (folio_test_private (folio) &&folio_trylock (folio)) { -if (folio_test_private (folio)) -filemap_release_folio (folio,0 ); +if (folio_needs_release (folio) && +folio_trylock (folio)) { +filemap_release_folio (folio,0 );folio_unlock (folio); } }
David Howells
maintainerPATCH v6 2/2 mm, netfs, fscache: Stop read optimisation when folio removed from pagecache
In reply to David HowellsFscache has an optimisation by which reads from the cache are skipped until we know that (a) there's data there to be read and (b) that data isn't entirely covered by pages resident in the netfs pagecache. This is done with two flags manipulated by fscache_note_page_release():
if (... test_bit(FSCACHE_COOKIE_HAVE_DATA, &cookie->flags) && test_bit(FSCACHE_COOKIE_NO_DATA_TO_READ, &cookie->flags)) clear_bit(FSCACHE_COOKIE_NO_DATA_TO_READ, &cookie->flags);
where the NO_DATA_TO_READ flag causes cachefiles_prepare_read() to indicate that netfslib should download from the server or clear the page instead.
The fscache_note_page_release() function is intended to be called from ->releasepage() - but that only gets called if PG_private or PG_private_2 is set - and currently the former is at the discretion of the network filesystem and the latter is only set whilst a page is being written to the cache, so sometimes we miss clearing the optimisation.
Fix this by following Willy's suggestion[1] and adding an address_space flag, AS_RELEASE_ALWAYS, that causes filemap_release_folio() to always call ->release_folio() if it's set, even if PG_private or PG_private_2 aren't set.
Note that this would require folio_test_private() and page_has_private() to become more complicated. To avoid that, in the places[*] where these are used to conditionalise calls to filemap_release_folio() and try_to_release_page(), the tests are removed the those functions just jumped to unconditionally and the test is performed there.
[*] There are some exceptions in vmscan.c where the check guards more than just a call to the releaser. I've added a function, folio_needs_release() to wrap all the checks for that.
AS_RELEASE_ALWAYS should be set if a non-NULL cookie is obtained from fscache and cleared in ->evict_inode() before truncate_inode_pages_final() is called.
Additionally, the FSCACHE_COOKIE_NO_DATA_TO_READ flag needs to be cleared and the optimisation cancelled if a cachefiles object already contains data when we open it.
---
Notes:
ver #4)
- Split out merging of folio_has_private()/filemap_release_folio() call pairs into a preceding patch.
- Don't need to clear AS_RELEASE_ALWAYS in ->evict_inode().
ver #3)
- Fixed mapping_clear_release_always() to use clear_bit() not set_bit().
- Moved a '&&' to the correct line.
ver #2)
- Rewrote entirely according to Willy's suggestion[1].
fs/9p/cache.c | 2 ++ fs/afs/internal.h | 2 ++ fs/cachefiles/namei.c | 2 ++ fs/ceph/cache.c | 2 ++ fs/cifs/fscache.c | 2 ++ include/linux/pagemap.h | 16 ++++++++++++++++ mm/internal.h | 5 ++++- 7 files changed, 30 insertions(+), 1 deletion(-)
fs/9p/cache.c +2 -0
@@ -68,6 +68,8 @@ void v9fs_cache_inode_get_cookie(struct inode *inode) &path,sizeof (path), &version,sizeof (version), i_size_read(&v9inode->netfs.inode)); +if (v9inode->netfs.cache) +mapping_set_release_always (inode->i_mapping);p9_debug (P9_DEBUG_FSC ,"inode %p get cookie %p\n" , inode,v9fs_inode_cookie (v9inode));
fs/afs/internal.h +2 -0
@@ -680,6 +680,8 @@ static inline void afs_vnode_set_cache(struct afs_vnode *vnode, {#ifdef CONFIG_AFS_FSCACHE vnode->netfs.cache = cookie; +if (cookie) +mapping_set_release_always (vnode->netfs.inode.i_mapping);#endif }
fs/cachefiles/namei.c +2 -0
@@ -584,6 +584,8 @@ static bool cachefiles_open_file(struct cachefiles_object *object,if (ret <0 ) goto check_failed; +clear_bit (FSCACHE_COOKIE_NO_DATA_TO_READ , &object->cookie->flags); + object->file = file;/* Always update the atime on an object we've just looked up (this is
fs/ceph/cache.c +2 -0
@@ -36,6 +36,8 @@ void ceph_fscache_register_inode_cookie(struct inode *inode) &ci->i_vino,sizeof (ci->i_vino), &ci->i_version,sizeof (ci->i_version), i_size_read(inode)); +if (ci->netfs.cache) +mapping_set_release_always (inode->i_mapping); }void ceph_fscache_unregister_inode_cookie (struct ceph_inode_info *ci)
fs/cifs/fscache.c +2 -0
@@ -108,6 +108,8 @@ void cifs_fscache_get_inode_cookie(struct inode *inode) &cifsi->uniqueid,sizeof (cifsi->uniqueid), &cd,sizeof (cd), i_size_read(&cifsi->netfs.inode)); +if (cifsi->netfs.cache) +mapping_set_release_always (inode->i_mapping); }void cifs_fscache_unuse_inode_cookie (struct inode *inode,bool update)
include/linux/pagemap.h +16 -0
@@ -199,6 +199,7 @@ enum mapping_flags {/* writeback related tags are not used */ AS_NO_WRITEBACK_TAGS =5 ,AS_LARGE_FOLIO_SUPPORT =6 , + AS_RELEASE_ALWAYS, /* Call ->release_folio(), even if no private data */ };/**
@@ -269,6 +270,21 @@ static inline int mapping_use_writeback_tags(struct address_space *mapping)return !test_bit (AS_NO_WRITEBACK_TAGS , &mapping->flags); } +static inline bool mapping_release_always (const struct address_space *mapping) +{ +return test_bit (AS_RELEASE_ALWAYS , &mapping->flags); +} + +static inline void mapping_set_release_always (struct address_space *mapping) +{ +set_bit (AS_RELEASE_ALWAYS , &mapping->flags); +} + +static inline void mapping_clear_release_always (struct address_space *mapping) +{ +clear_bit (AS_RELEASE_ALWAYS , &mapping->flags); +} +static inline gfp_t mapping_gfp_mask (struct address_space * mapping) {return mapping->gfp_mask;
mm/internal.h +4 -1
@@ -168,7 +168,10 @@ static inline void set_page_refcounted(struct page *page)*/ static inline bool folio_needs_release (struct folio *folio) { -return folio_has_private (folio); +struct address_space *mapping = folio->mapping; + +return folio_has_private (folio) || + (mapping &&mapping_release_always (mapping)); }extern unsigned long highest_memmap_pfn;
Quoted 83 lines · David Howells
Hi Willy,
Is this okay by you? You said you wanted to look at the remaining uses of
page_has_private(), of which there are then three after these patches, not countingfolio_has_private():arch/s390/kernel/uv.c: if (page_has_private(page)) mm/khugepaged.c: 1 + page_mapcount(page) + page_has_private(page)) { mm/migrate_device.c: extra += 1 + page_has_private(page);--
I've split the folio_has_private()/filemap_release_folio() call pair merging into its own patch, separate from the actual bugfix and pulled out thefolio_needs_release()function intomm/internal.hand madefilemap_release_folio()use it. I've also got rid of the bit clearances from the network filesystemevict_inodefunctions as they doesn't seem to be necessary.Note that the last vestiges of
try_to_release_page()got swept away, so I rebased and dealt with that. One comment remained, which is removed by the first patch.David
Changes:
========
ver #6)
- Drop the third patch which removes a duplicate check invmscan().ver #5)
- Rebased on linus/master.try_to_release_page()has now been entirely replaced byfilemap_release_folio(), barring one comment.
- Cleaned up some pairs in ext4.ver #4)
- Split has_private/release call pairs into own patch.
- Movedfolio_needs_release()tomm/internal.hand removed open-coded version fromfilemap_release_folio().
- Don't need to clearAS_RELEASE_ALWAYSin ->evict_inode().
- Added experimental patch to reduceshrink_folio_list().ver #3)
- Fixedmapping_clear_release_always()to useclear_bit()notset_bit().
- Moved a '&&' to the correct line.ver #2)
- Rewrote entirely according to Willy's suggestion[1].Link https://lore.kernel.org/r/164928630577.457102.8519251179327601178.stgit@warthog.procyon.org.uk/ # v1 open hereLink https://lore.kernel.org/r/166844174069.1124521.10890506360974169994.stgit@warthog.procyon.org.uk/ # v2 open hereLink https://lore.kernel.org/r/166869495238.3720468.4878151409085146764.stgit@warthog.procyon.org.uk/ # v3 open hereLink https://lore.kernel.org/r/[email protected]/ # v3 also open here---
%(shortlog)s
%(diffstat)sDavid Howells (2):
mm: Merge folio_has_private()/filemap_release_folio() call pairs mm, netfs, fscache: Stop read optimisation when folio removed from
pagecachefs/9p/cache.c | 2 ++ fs/afs/internal.h | 2 ++ fs/cachefiles/namei.c | 2 ++ fs/ceph/cache.c | 2 ++ fs/cifs/fscache.c | 2 ++ fs/ext4/move_extent.c | 12 ++++-------- fs/splice.c | 3 +-- include/linux/pagemap.h | 16 ++++++++++++++++ mm/filemap.c | 2 ++ mm/huge_memory.c | 3 +-- mm/internal.h | 11 +++++++++++ mm/khugepaged.c | 3 +-- mm/memory-failure.c | 8 +++----- mm/migrate.c | 3 +-- mm/truncate.c | 6 ++---- mm/vmscan.c | 8 ++++---- 16 files changed, 56 insertions(+), 29 deletions(-)
Quoted 5 lines
--
Linux-cachefs mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-cachefs
Willy, and David,
Can this series move forward?
This just got mentioned again [1] after Chris tested the NFS netfs patches that were merged in 6.4-rc1
Quoted 91 lines · David Wysochanski
Quoted 83 lines · David Howells
Hi Willy,
Is this okay by you? You said you wanted to look at the remaining uses of
page_has_private(), of which there are then three after these patches, not countingfolio_has_private():arch/s390/kernel/uv.c: if (page_has_private(page)) mm/khugepaged.c: 1 + page_mapcount(page) + page_has_private(page)) { mm/migrate_device.c: extra += 1 + page_has_private(page);--
I've split the folio_has_private()/filemap_release_folio() call pair merging into its own patch, separate from the actual bugfix and pulled out thefolio_needs_release()function intomm/internal.hand madefilemap_release_folio()use it. I've also got rid of the bit clearances from the network filesystemevict_inodefunctions as they doesn't seem to be necessary.Note that the last vestiges of
try_to_release_page()got swept away, so I rebased and dealt with that. One comment remained, which is removed by the first patch.David
Changes:
========
ver #6)
- Drop the third patch which removes a duplicate check invmscan().ver #5)
- Rebased on linus/master.try_to_release_page()has now been entirely replaced byfilemap_release_folio(), barring one comment.
- Cleaned up some pairs in ext4.ver #4)
- Split has_private/release call pairs into own patch.
- Movedfolio_needs_release()tomm/internal.hand removed open-coded version fromfilemap_release_folio().
- Don't need to clearAS_RELEASE_ALWAYSin ->evict_inode().
- Added experimental patch to reduceshrink_folio_list().ver #3)
- Fixedmapping_clear_release_always()to useclear_bit()notset_bit().
- Moved a '&&' to the correct line.ver #2)
- Rewrote entirely according to Willy's suggestion[1].Link https://lore.kernel.org/r/164928630577.457102.8519251179327601178.stgit@warthog.procyon.org.uk/ # v1 open hereLink https://lore.kernel.org/r/166844174069.1124521.10890506360974169994.stgit@warthog.procyon.org.uk/ # v2 open hereLink https://lore.kernel.org/r/166869495238.3720468.4878151409085146764.stgit@warthog.procyon.org.uk/ # v3 open hereLink https://lore.kernel.org/r/[email protected]/ # v3 also open here---
%(shortlog)s
%(diffstat)sDavid Howells (2):
mm: Merge folio_has_private()/filemap_release_folio() call pairs mm, netfs, fscache: Stop read optimisation when folio removed from
pagecachefs/9p/cache.c | 2 ++ fs/afs/internal.h | 2 ++ fs/cachefiles/namei.c | 2 ++ fs/ceph/cache.c | 2 ++ fs/cifs/fscache.c | 2 ++ fs/ext4/move_extent.c | 12 ++++-------- fs/splice.c | 3 +-- include/linux/pagemap.h | 16 ++++++++++++++++ mm/filemap.c | 2 ++ mm/huge_memory.c | 3 +-- mm/internal.h | 11 +++++++++++ mm/khugepaged.c | 3 +-- mm/memory-failure.c | 8 +++----- mm/migrate.c | 3 +-- mm/truncate.c | 6 ++---- mm/vmscan.c | 8 ++++---- 16 files changed, 56 insertions(+), 29 deletions(-)Quoted 5 lines
--
Linux-cachefs mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-cachefs
Quoted 7 lines
Willy, and David,
Can this series move forward?
This just got mentioned again [1] after Chris tested the NFS netfs patches that were merged in 6.4-rc1
Sorry about the timing on the original email as I forgot it lined up with LSF/MM.
FYI, I tested with 6.4-rc1 plus these two patches, then I added the NFS hunk needed (see below). All my tests pass now[1], and it makes sense from all the ftraces I've seen on test runs that fail (cachefiles_prep_read trace event would show "DOWN no-data" even after data was written previously).
This small NFS hunk needs added to patch #2 in this series:
fs/nfs/fscache.c +4 -0
@@ -180,6 +180,10 @@ void nfs_fscache_init_inode(struct inode *inode) &auxdata,/* aux_data */ sizeof (auxdata), i_size_read(inode)); + +if (netfs_inode (inode)->cache) +mapping_set_release_always (inode->i_mapping); + }/*
Any update on this? I assume it's too late for these patches to make 6.4.0.