mm: don't be stuck to rmap lock on reclaim path
Subsystems (8)
PATCH v4 1/1 mm: don't be stuck to rmap lock on reclaim path
The rmap locks(i_mmap_rwsem and anon_vma->root->rwsem) could be contended under memory pressure if processes keep working on their vmas(e.g., fork, mmap, munmap). It makes reclaim path stuck. In our real workload traces, we see kswapd is waiting the lock for 300ms+(worst case, a sec) and it makes other processes entering direct reclaim, which were also stuck on the lock.
This patch makes lru aging path try_lock mode like shink_page_list so the reclaim context will keep working with next lru pages without being stuck. if it found the rmap lock contended, it rotates the page back to head of lru in both active/inactive lrus to make them consistent behavior, which is basic starting point rather than adding more heristic.
Since this patch introduces a new "contended" field as out-param along with try_lock in-param in rmap_walk_control, it's not immutable any longer if the try_lock is set so remove const keywords on rmap related functions. Since rmap walking is already expensive operation, I doubt the const would help sizable benefit( And we didn't have it until 5.17).
In a heavy app workload in Android, trace shows following statistics. It almost removes rmap lock contention from reclaim path.
From Martin Liu
Before:
max_dur(ms) min_dur(ms) max-min(dur)ms avg_dur(ms) sum_dur(ms) count blocked_function
1632 0 1631 151.542173 31672 209 page_lock_anon_vma_read
601 0 601 145.544681 28817 198 rmap_walk_fileAfter:
max_dur(ms) min_dur(ms) max-min(dur)ms avg_dur(ms) sum_dur(ms) count blocked_function
NaN NaN NaN NaN NaN 0.0 NaN
0 0 0 0.127645 1 12 rmap_walk_file
--- include/linux/fs.h | 5 +++++ include/linux/ksm.h | 4 ++-- include/linux/rmap.h | 28 +++++++++++++++++------- mm/ksm.c | 10 +++++++-- mm/memory-failure.c | 2 +- mm/page_idle.c | 7 ++---- mm/rmap.c | 52 ++++++++++++++++++++++++++++++++++---------- mm/vmscan.c | 7 +++++- 8 files changed, 85 insertions(+), 30 deletions(-)
include/linux/fs.h +5 -0
@@ -477,6 +477,11 @@ static inline void i_mmap_unlock_write(struct address_space *mapping)up_write (&mapping->i_mmap_rwsem); } +static inline int i_mmap_trylock_read (struct address_space *mapping) +{ +return down_read_trylock (&mapping->i_mmap_rwsem); +} +static inline void i_mmap_lock_read (struct address_space *mapping) {down_read (&mapping->i_mmap_rwsem);
include/linux/ksm.h +2 -2
@@ -51,7 +51,7 @@ static inline void ksm_exit(struct mm_struct *mm)struct page *ksm_might_need_to_copy (struct page *page,struct vm_area_struct *vma,unsigned long address); -void rmap_walk_ksm (struct folio *folio,const struct rmap_walk_control *rwc); +void rmap_walk_ksm (struct folio *folio,struct rmap_walk_control *rwc);void folio_migrate_ksm (struct folio *newfolio,struct folio *folio);#else /* !CONFIG_KSM */
@@ -79,7 +79,7 @@ static inline struct page *ksm_might_need_to_copy(struct page *page, }static inline void rmap_walk_ksm (struct folio *folio, -const struct rmap_walk_control *rwc) +struct rmap_walk_control *rwc) { }
include/linux/rmap.h +20 -8
@@ -128,6 +128,11 @@ static inline void anon_vma_lock_read(struct anon_vma *anon_vma)down_read (&anon_vma->root->rwsem); } +static inline int anon_vma_trylock_read (struct anon_vma *anon_vma) +{ +return down_read_trylock (&anon_vma->root->rwsem); +} +static inline void anon_vma_unlock_read (struct anon_vma *anon_vma) {up_read (&anon_vma->root->rwsem);
@@ -366,17 +371,14 @@ int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff,void remove_migration_ptes (struct folio *src,struct folio *dst,bool locked); -/* -* Called by memory-failure.c to kill processes. -*/ -struct anon_vma *folio_lock_anon_vma_read (struct folio *folio); -void page_unlock_anon_vma_read (struct anon_vma *anon_vma);int page_mapped_in_vma (struct page *page,struct vm_area_struct *vma);/* * rmap_walk_control: To control rmap traversing for specific needs * * arg: passed to rmap_one() and invalid_vma() +* try_lock: bail out if the rmap lock is contended +* contended: indicate the rmap traversal bailed out due to lock contention * rmap_one: executed on each vma where page is mapped * done: for checking traversing termination condition * anon_lock: for getting anon_lock by optimized way rather than default
@@ -384,6 +386,8 @@ int page_mapped_in_vma(struct page *page, struct vm_area_struct *vma);*/ struct rmap_walk_control {void *arg; +bool try_lock; +bool contended;/* * Return false if page table scanning in rmap_walk should be stopped. * Otherwise, return true.
@@ -391,12 +395,20 @@ struct rmap_walk_control {bool (*rmap_one)(struct folio *folio,struct vm_area_struct *vma,unsigned long addr,void *arg);int (*done)(struct folio *folio); -struct anon_vma *(*anon_lock)(struct folio *folio); +struct anon_vma *(*anon_lock)(struct folio *folio, +struct rmap_walk_control *rwc); bool (*invalid_vma)(struct vm_area_struct *vma, void *arg); }; -void rmap_walk (struct folio *folio,const struct rmap_walk_control *rwc); -void rmap_walk_locked (struct folio *folio,const struct rmap_walk_control *rwc); +void rmap_walk (struct folio *folio,struct rmap_walk_control *rwc); +void rmap_walk_locked (struct folio *folio,struct rmap_walk_control *rwc); + +/* +* Called by memory-failure.c to kill processes. +*/ +struct anon_vma *folio_lock_anon_vma_read (struct folio *folio, +struct rmap_walk_control *rwc); +void page_unlock_anon_vma_read (struct anon_vma *anon_vma);#else /* !CONFIG_MMU */
mm/ksm.c +8 -2
@@ -2614,7 +2614,7 @@ struct page *ksm_might_need_to_copy(struct page *page,return new_page; } -void rmap_walk_ksm (struct folio *folio,const struct rmap_walk_control *rwc) +void rmap_walk_ksm (struct folio *folio,struct rmap_walk_control *rwc) {struct stable_node *stable_node;struct rmap_item *rmap_item;
@@ -2638,7 +2638,13 @@ void rmap_walk_ksm(struct folio *folio, const struct rmap_walk_control *rwc)struct vm_area_struct *vma;cond_resched (); -anon_vma_lock_read (anon_vma); +if (!anon_vma_trylock_read (anon_vma)) { +if (rwc->try_lock) { + rwc->contended = true; +return ; + } +anon_vma_lock_read (anon_vma); + }anon_vma_interval_tree_foreach (vmac, &anon_vma->rb_root, 0, ULONG_MAX) {unsigned long addr;
mm/memory-failure.c +1 -1
@@ -485,7 +485,7 @@ static void collect_procs_anon(struct page *page, struct list_head *to_kill,struct anon_vma *av;pgoff_t pgoff; - av =folio_lock_anon_vma_read (folio); + av =folio_lock_anon_vma_read (folio,NULL );if (av ==NULL )/* Not actually mapped anymore */ return ;
mm/page_idle.c +2 -5
@@ -86,11 +86,8 @@ static bool page_idle_clear_pte_refs_one(struct folio *folio,static void page_idle_clear_pte_refs (struct page *page) {struct folio *folio =page_folio (page); -/* -* Since rwc.arg is unused, rwc is effectively immutable, so we -* can make it static const to save some cycles and stack. -*/ -static const struct rmap_walk_control rwc = { + +static struct rmap_walk_control rwc = { .rmap_one = page_idle_clear_pte_refs_one, .anon_lock = folio_lock_anon_vma_read, };
mm/rmap.c +41 -11
131 lines not on this page · raw message
mm/vmscan.c +6 -1
@@ -1391,6 +1391,10 @@ static enum page_references folio_check_references(struct folio *folio,if (vm_flags &VM_LOCKED )return PAGEREF_ACTIVATE ; +/* rmap lock contention: rotate */ +if (referenced_ptes ==-1 ) +return PAGEREF_KEEP ; +if (referenced_ptes) {/* * All mapped folios start out with page table
@@ -2491,8 +2495,9 @@ static void shrink_active_list(unsigned long nr_to_scan, } } +/* Referenced or rmap lock contention: rotate */ if (folio_referenced (folio,0 , sc->target_mem_cgroup, - &vm_flags)) { + &vm_flags) !=0 ) {/* * Identify referenced, file-backed active pages and * give them one more trip around the active list. So
Signature · 1 line
2.36.0.512.ge40c2bad7a-goog
Quoted 32 lines · Minchan Kim
The rmap
locks(i_mmap_rwsem and anon_vma->root->rwsem)could be contended under memory pressure if processes keep working on theirvmas(e.g., fork, mmap, munmap). It makes reclaim path stuck. In our real workload traces, we see kswapd is waiting the lock for 300ms+(worst case, a sec) and it makes other processes entering direct reclaim, which were also stuck on the lock.This patch makes lru aging path
try_lockmode likeshink_page_listso the reclaim context will keep working with next lru pages without being stuck. if it found the rmap lock contended, it rotates the page back to head of lru in both active/inactive lrus to make them consistent behavior, which is basic starting point rather than adding more heristic.Since this patch introduces a new "contended" field as out-param along with
try_lockin-param inrmap_walk_control, it's not immutable any longer if thetry_lockis set so remove const keywords on rmap related functions. Since rmap walking is already expensive operation, I doubt the const would help sizable benefit( And we didn't have it until 5.17).In a heavy app workload in Android, trace shows following statistics. It almost removes rmap lock contention from reclaim path.
From Martin Liu
Before:
max_dur(ms) min_dur(ms) max-min(dur)ms avg_dur(ms) sum_dur(ms) count blocked_function 1632 0 1631 151.542173 31672 209 page_lock_anon_vma_read 601 0 601 145.544681 28817 198 rmap_walk_file
Quoted 7 lines
After:
max_dur(ms)min_dur(ms)max-min(dur)msavg_dur(ms)sum_dur(ms)countblocked_function
NaN NaN NaN NaN NaN 0.0 NaN
0 0 0 0.127645 1 12rmap_walk_fileSigned-off-by Minchan Kim <[email protected]>
Andrew Morton
maintainerQuoted 21 lines · On Tue, 10 May 2022 14:54:23 -0700 Minchan Kim <[email protected]> wrote:
The rmap
locks(i_mmap_rwsem and anon_vma->root->rwsem)could be contended under memory pressure if processes keep working on theirvmas(e.g., fork, mmap, munmap). It makes reclaim path stuck. In our real workload traces, we see kswapd is waiting the lock for 300ms+(worst case, a sec) and it makes other processes entering direct reclaim, which were also stuck on the lock.This patch makes lru aging path
try_lockmode likeshink_page_listso the reclaim context will keep working with next lru pages without being stuck. if it found the rmap lock contended, it rotates the page back to head of lru in both active/inactive lrus to make them consistent behavior, which is basic starting point rather than adding more heristic.Since this patch introduces a new "contended" field as out-param along with
try_lockin-param inrmap_walk_control, it's not immutable any longer if thetry_lockis set so remove const keywords on rmap related functions. Since rmap walking is already expensive operation, I doubt the const would help sizable benefit( And we didn't have it until 5.17).
Quoted 2 lines
In a heavy app workload in Android, trace shows following statistics. It almost removes rmap lock contention from reclaim path.
What might be the worst-case failure modes using this approach?
Could we burn much CPU time pointlessly churning though the LRU? Could it mess up aging decisions enough to be performance-affecting in any workload?
Something else?
Quoted 23 lines · Andrew Morton
Quoted 21 lines · On Tue, 10 May 2022 14:54:23 -0700 Minchan Kim <[email protected]> wrote:
The rmap
locks(i_mmap_rwsem and anon_vma->root->rwsem)could be contended under memory pressure if processes keep working on theirvmas(e.g., fork, mmap, munmap). It makes reclaim path stuck. In our real workload traces, we see kswapd is waiting the lock for 300ms+(worst case, a sec) and it makes other processes entering direct reclaim, which were also stuck on the lock.This patch makes lru aging path
try_lockmode likeshink_page_listso the reclaim context will keep working with next lru pages without being stuck. if it found the rmap lock contended, it rotates the page back to head of lru in both active/inactive lrus to make them consistent behavior, which is basic starting point rather than adding more heristic.Since this patch introduces a new "contended" field as out-param along with
try_lockin-param inrmap_walk_control, it's not immutable any longer if thetry_lockis set so remove const keywords on rmap related functions. Since rmap walking is already expensive operation, I doubt the const would help sizable benefit( And we didn't have it until 5.17).
Quoted 8 lines
Quoted 2 lines
In a heavy app workload in Android, trace shows following statistics. It almost removes rmap lock contention from reclaim path.
What might be the worst-case failure modes using this approach?
Could we burn much CPU time pointlessly churning though the LRU? Could it mess up aging decisions enough to be performance-affecting in any workload?
Yes, correct. However, we are already churning LRUs by several ways. For example, isolate and putback from LRU list for page migration from several sources(typical example is compaction) and trylock_page and sc->gfp_mask not allowing page to be reclaimed in shrink_page_list.
Quoted 2 lines
Something else?
One thing I am worry about was the granularity of the churning. Example above was page granuarity churning so might be execuse but this one is address space's churning, especically for file LRU (i_mmap_rwsem) which might cause too many rotating and live-lock in the end(keey rotating in small LRU with heavy memory pressure).
If it could be a problem, maybe we use sc->priority to stop the skipping on a certain level of memory pressure.
Any thought? Do we really need it?
Andrew Morton
maintainerQuoted 10 lines · On Wed, 11 May 2022 15:57:09 -0700 Minchan Kim <[email protected]> wrote:
Quoted 4 lines
Could we burn much CPU time pointlessly churning though the LRU? Could it mess up aging decisions enough to be performance-affecting in any workload?
Yes, correct. However, we are already churning LRUs by several ways. For example, isolate and putback from LRU list for page migration from several sources(typical example is compaction) and
trylock_pageandsc->gfp_masknot allowing page to be reclaimed inshrink_page_list.
Well. "we're already doing a risky thing so it's OK to do more of that thing"?
Quoted 9 lines
Quoted 2 lines
Something else?
One thing I am worry about was the granularity of the churning. Example above was page granuarity churning so might be execuse but this one is address space's churning, especically for file LRU (
i_mmap_rwsem) which might cause too many rotating and live-lock in the end(keey rotating in small LRU with heavy memory pressure).
Quoted 4 lines
If it could be a problem, maybe we use
sc->priorityto stop the skipping on a certain level of memory pressure.Any thought? Do we really need it?
Are we able to think of a test which might demonstrate any worst case? Whip that up and see what the numbers say?
It's a bit of a drag, but if we don't do it, our users surely will ;)
Quoted 7 lines · Andrew Morton
Quoted 5 lines · On Wed, 11 May 2022 15:57:09 -0700 Minchan Kim <[email protected]> wrote:
Quoted 4 lines
Could we burn much CPU time pointlessly churning though the LRU? Could it mess up aging decisions enough to be performance-affecting in any workload?
Quoted 8 lines
Quoted 5 lines
Yes, correct. However, we are already churning LRUs by several ways. For example, isolate and putback from LRU list for page migration from several sources(typical example is compaction) and
trylock_pageandsc->gfp_masknot allowing page to be reclaimed inshrink_page_list.Well. "we're already doing a risky thing so it's OK to do more of that thing"?
I meant the aging is not rocket science.
Quoted 10 lines
Quoted 10 lines
Quoted 3 lines
Something else?
One thing I am worry about was the granularity of the churning. Example above was page granuarity churning so might be execuse but this one is address space's churning, especically for file LRU (
i_mmap_rwsem) which might cause too many rotating and live-lock in the end(keey rotating in small LRU with heavy memory pressure).
Quoted 7 lines
Quoted 4 lines
If it could be a problem, maybe we use
sc->priorityto stop the skipping on a certain level of memory pressure.Any thought? Do we really need it?
Are we able to think of a test which might demonstrate any worst case? Whip that up and see what the numbers say?
Yeah, let me create a worst test case to see how it goes.
A thread keep reading a file-backed vma with 2xRAM file but other threads keep changing other vmas mapped at the same file so heavy i_mmap_rwsem contention in aging path.
Quoted 2 lines
It's a bit of a drag, but if we don't do it, our users surely will ;)