Remove nrexceptional tracking
Subsystems (9)
Matthew Wilcox (Oracle)
maintainerv2 cover Remove nrexceptional tracking
We actually use nrexceptional for very little these days. It's a minor pain to keep in sync with nrpages, but the pain becomes much bigger with the THP patches because we don't know how many indices a shadow entry occupies. It's easier to just remove it than keep it accurate.
Also, we save 8 bytes per inode which is nothing to sneeze at; on my laptop, it would improve shmem_inode_cache from 22 to 23 objects per 16kB, and inode_cache from 26 to 27 objects. Combined, that saves a megabyte of memory from a combined usage of 25MB for both caches. Unfortunately, ext4 doesn't cross a magic boundary, so it doesn't save any memory for ext4.
Matthew Wilcox (Oracle) (4):
mm: Introduce and use mapping_empty
mm: Stop accounting shadow entries
dax: Account DAX entries as nrpages
mm: Remove nrexceptional from inode
fs/block_dev.c | 2 +- fs/dax.c | 8 ++++---- fs/gfs2/glock.c | 3 +-- fs/inode.c | 2 +- include/linux/fs.h | 2 -- include/linux/pagemap.h | 5 +++++ mm/filemap.c | 16 ---------------- mm/swap_state.c | 4 ---- mm/truncate.c | 19 +++---------------- mm/workingset.c | 1 - 10 files changed, 15 insertions(+), 47 deletions(-)
Signature · 1 line
2.28.0
Matthew Wilcox (Oracle)
maintainerPATCH v2 1/4 mm: Introduce and use mapping_empty
Instead of checking the two counters (nrpages and nrexceptional), we can just check whether i_pages is empty.
--- fs/block_dev.c | 2 +- fs/dax.c | 2 +- fs/gfs2/glock.c | 3 +-- include/linux/pagemap.h | 5 +++++ mm/truncate.c | 18 +++--------------- 5 files changed, 11 insertions(+), 19 deletions(-)
fs/block_dev.c +1 -1
@@ -79,7 +79,7 @@ static void kill_bdev(struct block_device *bdev) {struct address_space *mapping = bdev->bd_inode->i_mapping; -if (mapping->nrpages ==0 && mapping->nrexceptional ==0 ) +if (mapping_empty (mapping))return ;invalidate_bh_lrus ();
fs/dax.c +1 -1
@@ -965,7 +965,7 @@ int dax_writeback_mapping_range(struct address_space *mapping,if (WARN_ON_ONCE (inode->i_blkbits !=PAGE_SHIFT ))return -EIO ; -if (!mapping->nrexceptional || wbc->sync_mode !=WB_SYNC_ALL ) +if (mapping_empty (mapping) || wbc->sync_mode !=WB_SYNC_ALL )return 0 ;trace_dax_writeback_range (inode, xas.xa_index, end_index);
fs/gfs2/glock.c +1 -2
@@ -273,8 +273,7 @@ static void __gfs2_glock_put(struct gfs2_glock *gl)if (mapping) {truncate_inode_pages_final (mapping);if (!gfs2_withdrawn (sdp)) -GLOCK_BUG_ON (gl, mapping->nrpages || - mapping->nrexceptional); +GLOCK_BUG_ON (gl, !mapping_empty (mapping)); }trace_gfs2_glock_put (gl); sdp->sd_lockstruct.ls_ops->lm_put_lock (gl);
include/linux/pagemap.h +5 -0
@@ -18,6 +18,11 @@struct pagevec ; +static inline bool mapping_empty (struct address_space *mapping) +{ +return xa_empty (&mapping->i_pages); +} +/* * Bits in mapping->flags. */
mm/truncate.c +3 -15
@@ -342,7 +342,7 @@ void truncate_inode_pages_range(struct address_space *mapping,struct page * page;bool partial_end; -if (mapping->nrpages ==0 && mapping->nrexceptional ==0 ) +if (mapping_empty (mapping)) goto out;/*
@@ -470,9 +470,6 @@ EXPORT_SYMBOL(truncate_inode_pages);*/ void truncate_inode_pages_final (struct address_space *mapping) { -unsigned long nrexceptional; -unsigned long nrpages; -/* * Page reclaim can not participate in regular inode lifetime * management (can't call iput()) and thus can race with the
@@ -482,16 +479,7 @@ void truncate_inode_pages_final(struct address_space *mapping)*/ mapping_set_exiting (mapping); -/* -* When reclaim installs eviction entries, it increases -* nrexceptional first, then decreases nrpages. Make sure we see -* this in the right order or we might miss an entry. -*/ - nrpages = mapping->nrpages; -smp_rmb (); - nrexceptional = mapping->nrexceptional; - -if (nrpages || nrexceptional) { +if (!mapping_empty (mapping)) {/* * As truncation uses a lockless tree lookup, cycle * the tree lock to make sure any ongoing tree
@@ -657,7 +645,7 @@ int invalidate_inode_pages2_range(struct address_space *mapping,int ret2 =0 ;int did_range_unmap =0 ; -if (mapping->nrpages ==0 && mapping->nrexceptional ==0 ) +if (mapping_empty (mapping)) goto out;pagevec_init (&pvec);
Signature · 1 line
2.28.0
Re: [PATCH v2 1/4] mm: Introduce and use mapping_empty
Quoted 5 lines · Matthew Wilcox (Oracle)
Instead of checking the two counters (nrpages and nrexceptional), we can just check whether
i_pagesis empty.Signed-off-by Matthew Wilcox (Oracle) <[email protected]>Tested-by Vishal Verma <[email protected]>
Heh, I was looking for the fs/inode.c hunk here, because I remember those BUG_ONs in the inode free path. Found it in the last patch - I guess they escaped grep but the compiler let you know? :-)
Matthew Wilcox
maintainerRe: [PATCH v2 1/4] mm: Introduce and use mapping_empty
Quoted 4 lines · Johannes Weiner
Quoted 3 lines · Matthew Wilcox (Oracle)
Instead of checking the two counters (nrpages and nrexceptional), we can just check whether
i_pagesis empty.
Quoted 8 lines
Quoted 2 lines
Signed-off-by Matthew Wilcox (Oracle) <[email protected]>Tested-by Vishal Verma <[email protected]>Acked-by Johannes Weiner <[email protected]>Heh, I was looking for the
fs/inode.chunk here, because I remember thoseBUG_ONsin the inode free path. Found it in the last patch - I guess they escaped grep but the compiler let you know? :-)
Heh, I forget now! I think I did it that way on purpose, but now I forget what that purpose was!
Matthew Wilcox (Oracle)
maintainerPATCH v2 2/4 mm: Stop accounting shadow entries
In reply to Matthew Wilcox (Oracle)We no longer need to keep track of how many shadow entries are present in a mapping. This saves a few writes to the inode and memory barriers.
--- mm/filemap.c | 13 ------------- mm/swap_state.c | 4 ---- mm/truncate.c | 1 - mm/workingset.c | 1 - 4 files changed, 19 deletions(-)
mm/filemap.c +0 -13
@@ -140,17 +140,6 @@ static void page_cache_delete(struct address_space *mapping, page->mapping =NULL ;/* Leave page->index set: truncation lookup relies upon it */ - -if (shadow) { - mapping->nrexceptional += nr; -/* -* Make sure the nrexceptional update is committed before -* the nrpages update so that final truncate racing -* with reclaim does not see both counters 0 at the -* same time and miss a shadow entry. -*/ -smp_wmb (); - } mapping->nrpages -= nr; }
@@ -883,8 +872,6 @@ noinline int __add_to_page_cache_locked(struct page *page,if (xas_error (&xas)) goto unlock; -if (old) - mapping->nrexceptional--; mapping->nrpages++;/* hugetlb pages do not participate in page cache accounting */
mm/swap_state.c +0 -4
@@ -160,7 +160,6 @@ int add_to_swap_cache(struct page *page, swp_entry_t entry,xas_store (&xas, page);xas_next (&xas); } - address_space->nrexceptional -= nr_shadows; address_space->nrpages += nr;__mod_node_page_state (page_pgdat (page),NR_FILE_PAGES , nr);ADD_CACHE_INFO (add_total, nr);
@@ -199,8 +198,6 @@ void __delete_from_swap_cache(struct page *page,xas_next (&xas); }ClearPageSwapCache (page); -if (shadow) - address_space->nrexceptional += nr; address_space->nrpages -= nr;__mod_node_page_state (page_pgdat (page),NR_FILE_PAGES , -nr);ADD_CACHE_INFO (del_total, nr);
@@ -301,7 +298,6 @@ void clear_shadow_from_swap_cache(int type, unsigned long begin,xas_store (&xas,NULL ); nr_shadows++; } - address_space->nrexceptional -= nr_shadows;xa_unlock_irq (&address_space->i_pages);/* search the next swapcache until we meet end */
mm/truncate.c +0 -1
@@ -40,7 +40,6 @@ static inline void __clear_shadow_entry(struct address_space *mapping,if (xas_load (&xas) != entry)return ;xas_store (&xas,NULL ); - mapping->nrexceptional--; } static void clear_shadow_entry(struct address_space *mapping, pgoff_t index,
mm/workingset.c +0 -1
@@ -557,7 +557,6 @@ static enum lru_status shadow_lru_isolate(struct list_head *item, goto out_invalid;if (WARN_ON_ONCE (node->count != node->nr_values)) goto out_invalid; - mapping->nrexceptional -= node->nr_values;xa_delete_node (node, workingset_update_node);__inc_lruvec_slab_state (node,WORKINGSET_NODERECLAIM );
Signature · 1 line
2.28.0
Re: [PATCH v2 2/4] mm: Stop accounting shadow entries
Quoted 6 lines · Matthew Wilcox (Oracle)
We no longer need to keep track of how many shadow entries are present in a mapping. This saves a few writes to the inode and memory barriers.
Signed-off-by Matthew Wilcox (Oracle) <[email protected]>Tested-by Vishal Verma <[email protected]>
Matthew Wilcox (Oracle)
maintainerPATCH v2 3/4 dax: Account DAX entries as nrpages
In reply to Matthew Wilcox (Oracle)Simplify mapping_needs_writeback() by accounting DAX entries as pages instead of exceptional entries.
--- fs/dax.c | 6 +++--- mm/filemap.c | 3 --- 2 files changed, 3 insertions(+), 6 deletions(-)
fs/dax.c +3 -3
@@ -525,7 +525,7 @@ static void *grab_mapping_entry(struct xa_state *xas,dax_disassociate_entry (entry, mapping, false);xas_store (xas,NULL );/* undo the PMD join */ dax_wake_entry (xas, entry, true); - mapping->nrexceptional--; + mapping->nrpages -=PG_PMD_NR ; entry =NULL ;xas_set (xas, index); }
@@ -541,7 +541,7 @@ static void *grab_mapping_entry(struct xa_state *xas,dax_lock_entry (xas, entry);if (xas_error (xas)) goto out_unlock; - mapping->nrexceptional++; + mapping->nrpages +=1UL << order; } out_unlock:
@@ -661,7 +661,7 @@ static int __dax_invalidate_entry(struct address_space *mapping, goto out;dax_disassociate_entry (entry, mapping, trunc);xas_store (&xas,NULL ); - mapping->nrexceptional--; + mapping->nrpages -=1UL <<dax_entry_order (entry); ret =1 ; out:put_unlocked_entry (&xas, entry);
mm/filemap.c +0 -3
@@ -616,9 +616,6 @@ EXPORT_SYMBOL(filemap_fdatawait_keep_errors);/* Returns true if writeback might be needed or already in progress. */ static bool mapping_needs_writeback (struct address_space *mapping) { -if (dax_mapping (mapping)) -return mapping->nrexceptional; -return mapping->nrpages; }
Signature · 1 line
2.28.0
Re: [PATCH v2 3/4] dax: Account DAX entries as nrpages
Quoted 5 lines · Matthew Wilcox (Oracle)
Simplify
mapping_needs_writeback()by accounting DAX entries as pages instead of exceptional entries.Signed-off-by Matthew Wilcox (Oracle) <[email protected]>Tested-by Vishal Verma <[email protected]>
Matthew Wilcox (Oracle)
maintainerPATCH v2 4/4 mm: Remove nrexceptional from inode
In reply to Matthew Wilcox (Oracle)We no longer track anything in nrexceptional, so remove it, saving 8 bytes per inode.
--- fs/inode.c | 2 +- include/linux/fs.h | 2 -- 2 files changed, 1 insertion(+), 3 deletions(-)
fs/inode.c +1 -1
@@ -530,7 +530,7 @@ void clear_inode(struct inode *inode)*/ xa_lock_irq (&inode->i_data.i_pages);BUG_ON (inode->i_data.nrpages); -BUG_ON (inode->i_data.nrexceptional); +BUG_ON (!mapping_empty (&inode->i_data));xa_unlock_irq (&inode->i_data.i_pages);BUG_ON (!list_empty (&inode->i_data.private_list));BUG_ON (!(inode->i_state &I_FREEING ));
include/linux/fs.h +0 -2
@@ -439,7 +439,6 @@ int pagecache_write_end(struct file *, struct address_space *mapping,* @i_mmap: Tree of private and shared mappings. * @i_mmap_rwsem: Protects @i_mmap and @i_mmap_writable. * @nrpages: Number of page entries, protected by the i_pages lock. -* @nrexceptional: Shadow or DAX entries, protected by the i_pages lock. * @writeback_index: Writeback starts here. * @a_ops: Methods. * @flags: Error bits and flags (AS_*).
@@ -460,7 +459,6 @@ struct address_space {struct rb_root_cached i_mmap;struct rw_semaphore i_mmap_rwsem;unsigned long nrpages; -unsigned long nrexceptional;pgoff_t writeback_index;const struct address_space_operations *a_ops;unsigned long flags;
Signature · 1 line
2.28.0
Re: [PATCH v2 4/4] mm: Remove nrexceptional from inode
Quoted 5 lines · Matthew Wilcox (Oracle)
We no longer track anything in nrexceptional, so remove it, saving 8 bytes per inode.
Signed-off-by Matthew Wilcox (Oracle) <[email protected]>Tested-by Vishal Verma <[email protected]>
Matthew Wilcox
maintainerPing? These patches still apply to next-20210121.
Quoted 33 lines · Matthew Wilcox (Oracle)
We actually use nrexceptional for very little these days. It's a minor pain to keep in sync with nrpages, but the pain becomes much bigger with the THP patches because we don't know how many indices a shadow entry occupies. It's easier to just remove it than keep it accurate.
Also, we save 8 bytes per inode which is nothing to sneeze at; on my laptop, it would improve
shmem_inode_cachefrom 22 to 23 objects per 16kB, andinode_cachefrom 26 to 27 objects. Combined, that saves a megabyte of memory from a combined usage of 25MB for both caches. Unfortunately, ext4 doesn't cross a magic boundary, so it doesn't save any memory for ext4.Matthew Wilcox (Oracle) (4):
mm: Introduce and usemapping_empty
mm: Stop accounting shadow entries
dax: Account DAX entries as nrpages
mm: Remove nrexceptional from inodefs/block_dev.c | 2 +- fs/dax.c | 8 ++++---- fs/gfs2/glock.c | 3 +-- fs/inode.c | 2 +- include/linux/fs.h | 2 -- include/linux/pagemap.h | 5 +++++ mm/filemap.c | 16 ---------------- mm/swap_state.c | 4 ---- mm/truncate.c | 19 +++---------------- mm/workingset.c | 1 - 10 files changed, 15 insertions(+), 47 deletions(-)Signature · 1 line
2.28.0
Matthew Wilcox
maintainerPing?
Quoted 37 lines · Matthew Wilcox
Ping? These patches still apply to next-20210121.
Quoted 33 lines · Matthew Wilcox (Oracle)
We actually use nrexceptional for very little these days. It's a minor pain to keep in sync with nrpages, but the pain becomes much bigger with the THP patches because we don't know how many indices a shadow entry occupies. It's easier to just remove it than keep it accurate.
Also, we save 8 bytes per inode which is nothing to sneeze at; on my laptop, it would improve
shmem_inode_cachefrom 22 to 23 objects per 16kB, andinode_cachefrom 26 to 27 objects. Combined, that saves a megabyte of memory from a combined usage of 25MB for both caches. Unfortunately, ext4 doesn't cross a magic boundary, so it doesn't save any memory for ext4.Matthew Wilcox (Oracle) (4):
mm: Introduce and usemapping_empty
mm: Stop accounting shadow entries
dax: Account DAX entries as nrpages
mm: Remove nrexceptional from inodefs/block_dev.c | 2 +- fs/dax.c | 8 ++++---- fs/gfs2/glock.c | 3 +-- fs/inode.c | 2 +- include/linux/fs.h | 2 -- include/linux/pagemap.h | 5 +++++ mm/filemap.c | 16 ---------------- mm/swap_state.c | 4 ---- mm/truncate.c | 19 +++---------------- mm/workingset.c | 1 - 10 files changed, 15 insertions(+), 47 deletions(-)Signature · 1 line
2.28.0
Quoted 22 lines · Matthew Wilcox (Oracle)
We actually use nrexceptional for very little these days. It's a minor pain to keep in sync with nrpages, but the pain becomes much bigger with the THP patches because we don't know how many indices a shadow entry occupies. It's easier to just remove it than keep it accurate.
Also, we save 8 bytes per inode which is nothing to sneeze at; on my laptop, it would improve
shmem_inode_cachefrom 22 to 23 objects per 16kB, andinode_cachefrom 26 to 27 objects. Combined, that saves a megabyte of memory from a combined usage of 25MB for both caches. Unfortunately, ext4 doesn't cross a magic boundary, so it doesn't save any memory for ext4.Matthew Wilcox (Oracle) (4):
mm: Introduce and usemapping_empty
mm: Stop accounting shadow entries
dax: Account DAX entries as nrpages
mm: Remove nrexceptional from inode
Quoted 14 lines
fs/block_dev.c | 2 +- fs/dax.c | 8 ++++---- fs/gfs2/glock.c | 3 +-- fs/inode.c | 2 +- include/linux/fs.h | 2 -- include/linux/pagemap.h | 5 +++++ mm/filemap.c | 16 ---------------- mm/swap_state.c | 4 ---- mm/truncate.c | 19 +++---------------- mm/workingset.c | 1 - 10 files changed, 15 insertions(+), 47 deletions(-)Signature · 1 line
2.28.0
Looks good to me.