per memcg lru lock
Subsystems (15)
v21 cover per memcg lru lock
This version rebase on next/master 20201104, with much of Johannes's Acks and some changes according to Johannes comments. And add a new patch v21-0006-mm-rmap-stop-store-reordering-issue-on-page-mapp.patch to support v21-0007.
This patchset followed 2 memcg VM_WARN_ON_ONCE_PAGE patches which were added to -mm tree yesterday.
Many thanks for line by line review by Hugh Dickins, Alexander Duyck and Johannes Weiner.
So now this patchset includes 3 parts:
1, some code cleanup and minimum optimization as a preparation. 2, use TestCleanPageLRU as page isolation's precondition. 3, replace per node lru_lock with per memcg per node lru_lock.
Current lru_lock is one for each of node, pgdat->lru_lock, that guard for lru lists, but now we had moved the lru lists into memcg for long time. Still using per node lru_lock is clearly unscalable, pages on each of memcgs have to compete each others for a whole lru_lock. This patchset try to use per lruvec/memcg lru_lock to repleace per node lru lock to guard lru lists, make it scalable for memcgs and get performance gain.
Currently lru_lock still guards both lru list and page's lru bit, that's ok. but if we want to use specific lruvec lock on the page, we need to pin down the page's lruvec/memcg during locking. Just taking lruvec lock first may be undermined by the page's memcg charge/migration. To fix this problem, we could take out the page's lru bit clear and use it as pin down action to block the memcg changes. That's the reason for new atomic func TestClearPageLRU. So now isolating a page need both actions: TestClearPageLRU and hold the lru_lock.
The typical usage of this is isolate_migratepages_block() in compaction.c we have to take lru bit before lru lock, that serialized the page isolation in memcg page charge/migration which will change page's lruvec and new lru_lock in it.
The above solution suggested by Johannes Weiner, and based on his new memcg charge path, then have this patchset. (Hugh Dickins tested and contributed much code from compaction fix to general code polish, thanks a lot!).
Daniel Jordan's testing show 62% improvement on modified readtwice case on his 2P * 10 core * 2 HT broadwell box on v18, which has no much different with this v20.
https://lore.kernel.org/lkml/20200915165807.kpp7uhiw7l3loofu-S51bK0XF4qpuJJETbFA3a0B3C2bhBk7L0E9HWUfgJXw@public.gmane.org/
Thanks Hugh Dickins and Konstantin Khlebnikov, they both brought this idea 8 years ago, and others who give comments as well: Daniel Jordan, Mel Gorman, Shakeel Butt, Matthew Wilcox, Alexander Duyck etc.
Thanks for Testing support from Intel 0day and Rong Chen, Fengguang Wu, and Yun Wang. Hugh Dickins also shared his kbuild-swap case. Thanks!
Alex Shi (16):
mm/thp: move lru_add_page_tail func to huge_memory.c mm/thp: use head for head page in lru_add_page_tail mm/thp: Simplify lru_add_page_tail()
mm/thp: narrow lru locking
mm/vmscan: remove unnecessary lruvec adding mm/rmap: stop store reordering issue on page->mapping mm/memcg: add debug checking in lock_page_memcg mm/swap.c: fold vm event PGROTATED into pagevec_move_tail_fn mm/lru: move lock into lru_note_cost
mm/vmscan: remove lruvec reget in move_pages_to_lru mm/mlock: remove lru_lock on TestClearPageMlocked mm/mlock: remove __munlock_isolate_lru_page mm/lru: introduce TestClearPageLRU
mm/compaction: do page isolation first in compaction mm/swap.c: serialize memcg changes in pagevec_lru_move_fn mm/lru: replace pgdat lru_lock with lruvec lock
Alexander Duyck (1):
mm/lru: introduce the relock_page_lruvec function
Hugh Dickins (2):
mm: page_idle_get_page() does not need lru_lock
mm/lru: revise the comments of lru_lock
Documentation/admin-guide/cgroup-v1/memcg_test.rst | 15 +- Documentation/admin-guide/cgroup-v1/memory.rst | 21 +-- Documentation/trace/events-kmem.rst | 2 +- Documentation/vm/unevictable-lru.rst | 22 +-- include/linux/memcontrol.h | 110 +++++++++++ include/linux/mm_types.h | 2 +- include/linux/mmzone.h | 6 +- include/linux/page-flags.h | 1 + include/linux/swap.h | 4 +- mm/compaction.c | 94 +++++++--- mm/filemap.c | 4 +- mm/huge_memory.c | 45 +++-- mm/memcontrol.c | 79 +++++++- mm/mlock.c | 63 ++----- mm/mmzone.c | 1 + mm/page_alloc.c | 1 - mm/page_idle.c | 4 - mm/rmap.c | 11 +- mm/swap.c | 208 ++++++++------------- mm/vmscan.c | 207 ++++++++++---------- mm/workingset.c | 2 - 21 files changed, 530 insertions(+), 372 deletions(-)
Signature · 1 line
1.8.3.1
PATCH v21 1/19 mm/thp: move lru_add_page_tail func to huge_memory.c
The func is only used in huge_memory.c, defining it in other file with a CONFIG_TRANSPARENT_HUGEPAGE macro restrict just looks weird.
Let's move it THP. And make it static as Hugh Dickin suggested.
--- include/linux/swap.h | 2 -- mm/huge_memory.c | 30 ++++++++++++++++++++++++++++++ mm/swap.c | 33 --------------------------------- 3 files changed, 30 insertions(+), 35 deletions(-)
include/linux/swap.h +0 -2
@@ -338,8 +338,6 @@ extern void lru_note_cost(struct lruvec *lruvec, bool file, unsigned int nr_pages);extern void lru_note_cost_page (struct page *);extern void lru_cache_add (struct page *); -extern void lru_add_page_tail (struct page *page,struct page *page_tail, -struct lruvec *lruvec,struct list_head *head);extern void mark_page_accessed (struct page *);extern void lru_add_drain (void );extern void lru_add_drain_cpu (int cpu);
mm/huge_memory.c +30 -0
@@ -2348,6 +2348,36 @@ static void remap_page(struct page *page, unsigned int nr) } } +static void lru_add_page_tail (struct page *page,struct page *page_tail, +struct lruvec *lruvec,struct list_head *list) +{ +VM_BUG_ON_PAGE (!PageHead (page), page); +VM_BUG_ON_PAGE (PageCompound (page_tail), page); +VM_BUG_ON_PAGE (PageLRU (page_tail), page); +lockdep_assert_held (&lruvec_pgdat (lruvec)->lru_lock); + +if (!list) +SetPageLRU (page_tail); + +if (likely (PageLRU (page))) +list_add_tail (&page_tail->lru, &page->lru); +else if (list) { +/* page reclaim is reclaiming a huge page */ +get_page (page_tail); +list_add_tail (&page_tail->lru, list); + }else { +/* +* Head page has not yet been counted, as an hpage, +* so we must account for each subpage individually. +* +* Put page_tail on the list at the correct position +* so they all end up in order. +*/ +add_page_to_lru_list_tail (page_tail, lruvec, +page_lru (page_tail)); + } +} +static void __split_huge_page_tail (struct page *head,int tail,struct lruvec *lruvec,struct list_head *list) {
mm/swap.c +0 -33
@@ -977,39 +977,6 @@ void __pagevec_release(struct pagevec *pvec) }EXPORT_SYMBOL (__pagevec_release); -#ifdef CONFIG_TRANSPARENT_HUGEPAGE -/* used by __split_huge_page_refcount() */ -void lru_add_page_tail (struct page *page,struct page *page_tail, -struct lruvec *lruvec,struct list_head *list) -{ -VM_BUG_ON_PAGE (!PageHead (page), page); -VM_BUG_ON_PAGE (PageCompound (page_tail), page); -VM_BUG_ON_PAGE (PageLRU (page_tail), page); -lockdep_assert_held (&lruvec_pgdat (lruvec)->lru_lock); - -if (!list) -SetPageLRU (page_tail); - -if (likely (PageLRU (page))) -list_add_tail (&page_tail->lru, &page->lru); -else if (list) { -/* page reclaim is reclaiming a huge page */ -get_page (page_tail); -list_add_tail (&page_tail->lru, list); - }else { -/* -* Head page has not yet been counted, as an hpage, -* so we must account for each subpage individually. -* -* Put page_tail on the list at the correct position -* so they all end up in order. -*/ -add_page_to_lru_list_tail (page_tail, lruvec, -page_lru (page_tail)); - } -} -#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static void __pagevec_lru_add_fn (struct page *page,struct lruvec *lruvec,void *arg) {
Signature · 1 line
1.8.3.1
PATCH v21 2/19 mm/thp: use head for head page in lru_add_page_tail
In reply to Alex ShiSince the first parameter is only used by head page, it's better to make it explicit.
--- mm/huge_memory.c | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-)
mm/huge_memory.c +11 -12
@@ -2348,33 +2348,32 @@ static void remap_page(struct page *page, unsigned int nr) } } -static void lru_add_page_tail (struct page *page,struct page *page_tail, +static void lru_add_page_tail (struct page *head,struct page *tail,struct lruvec *lruvec,struct list_head *list) { -VM_BUG_ON_PAGE (!PageHead (page), page); -VM_BUG_ON_PAGE (PageCompound (page_tail), page); -VM_BUG_ON_PAGE (PageLRU (page_tail), page); +VM_BUG_ON_PAGE (!PageHead (head), head); +VM_BUG_ON_PAGE (PageCompound (tail), head); +VM_BUG_ON_PAGE (PageLRU (tail), head);lockdep_assert_held (&lruvec_pgdat (lruvec)->lru_lock);if (!list) -SetPageLRU (page_tail); +SetPageLRU (tail); -if (likely (PageLRU (page))) -list_add_tail (&page_tail->lru, &page->lru); +if (likely (PageLRU (head))) +list_add_tail (&tail->lru, &head->lru);else if (list) {/* page reclaim is reclaiming a huge page */ -get_page (page_tail); -list_add_tail (&page_tail->lru, list); +get_page (tail); +list_add_tail (&tail->lru, list); }else {/* * Head page has not yet been counted, as an hpage, * so we must account for each subpage individually. * -* Put page_tail on the list at the correct position +* Put tail on the list at the correct position * so they all end up in order. */ -add_page_to_lru_list_tail (page_tail, lruvec, -page_lru (page_tail)); +add_page_to_lru_list_tail (tail, lruvec,page_lru (tail)); } }
Signature · 1 line
1.8.3.1
PATCH v21 3/19 mm/thp: Simplify lru_add_page_tail()
In reply to Alex ShiSimplify lru_add_page_tail(), there are actually only two cases possible:split_huge_page_to_list(), with list supplied and head isolated from lru by its caller; or split_huge_page(), with NULL list and head on lru - because when head is racily isolated from lru, the isolator's reference will stop the split from getting any further than its page_ref_freeze().
So decide between the two cases by "list", but add VM_WARN_ON()s to verify that they match our lru expectations.
--- mm/huge_memory.c | 20 ++++++-------------- 1 file changed, 6 insertions(+), 14 deletions(-)
mm/huge_memory.c +6 -14
@@ -2356,24 +2356,16 @@ static void lru_add_page_tail(struct page *head, struct page *tail,VM_BUG_ON_PAGE (PageLRU (tail), head);lockdep_assert_held (&lruvec_pgdat (lruvec)->lru_lock); -if (!list) -SetPageLRU (tail); - -if (likely (PageLRU (head))) -list_add_tail (&tail->lru, &head->lru); -else if (list) { +if (list) {/* page reclaim is reclaiming a huge page */ +VM_WARN_ON (PageLRU (head));get_page (tail);list_add_tail (&tail->lru, list); }else { -/* -* Head page has not yet been counted, as an hpage, -* so we must account for each subpage individually. -* -* Put tail on the list at the correct position -* so they all end up in order. -*/ -add_page_to_lru_list_tail (tail, lruvec,page_lru (tail)); +/* head is still on lru (and we have it frozen) */ +VM_WARN_ON (!PageLRU (head)); +SetPageLRU (tail); +list_add_tail (&tail->lru, &head->lru); } }
Signature · 1 line
1.8.3.1
PATCH v21 4/19 mm/thp: narrow lru locking
In reply to Alex Shilru_lock and page cache xa_lock have no obvious reason to be taken one way round or the other: until now, lru_lock has been taken before page cache xa_lock, when splitting a THP; but nothing else takes them together. Reverse that ordering: let's narrow the lru locking - but leave local_irq_disable to block interrupts throughout, like before.
Hugh Dickins point: split_huge_page_to_list() was already silly, to be using the _irqsave variant: it's just been taking sleeping locks, so would already be broken if entered with interrupts enabled. So we can save passing flags argument down to __split_huge_page().
Why change the lock ordering here? That was hard to decide. One reason:
when this series reaches per-memcg lru locking, it relies on the THP's memcg to be stable when taking the lru_lock: that is now done after the THP's refcount has been frozen, which ensures page memcg cannot change.
Another reason: previously, lock_page_memcg()'s move_lock was presumed to nest inside lru_lock; but now lru_lock must nest inside (page cache lock inside) move_lock, so it becomes possible to use lock_page_memcg() to stabilize page memcg before taking its lru_lock. That is not the mechanism used in this series, but it is an option we want to keep open.
--- mm/huge_memory.c | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-)
mm/huge_memory.c +13 -12
@@ -2435,7 +2435,7 @@ static void __split_huge_page_tail(struct page *head, int tail, }static void __split_huge_page (struct page *page,struct list_head *list, -pgoff_t end,unsigned long flags) +pgoff_t end) {struct page *head =compound_head (page);pg_data_t *pgdat =page_pgdat (head);
@@ -2445,8 +2445,6 @@ static void __split_huge_page(struct page *page, struct list_head *list,unsigned int nr =thp_nr_pages (head);int i; - lruvec =mem_cgroup_page_lruvec (head, pgdat); -/* complete memcg works before add pages to LRU */ mem_cgroup_split_huge_fixup (head);
@@ -2458,6 +2456,11 @@ static void __split_huge_page(struct page *page, struct list_head *list,xa_lock (&swap_cache->i_pages); } +/* prevent PageLRU to go away from under us, and freeze lru stats */ +spin_lock (&pgdat->lru_lock); + + lruvec =mem_cgroup_page_lruvec (head, pgdat); +for (i = nr -1 ; i >=1 ; i--) {__split_huge_page_tail (head, i, lruvec, list);/* Some pages can be beyond i_size: drop them from page cache */
@@ -2477,6 +2480,8 @@ static void __split_huge_page(struct page *page, struct list_head *list, }ClearPageCompound (head); +spin_unlock (&pgdat->lru_lock); +/* Caller disabled irqs, so they are still disabled here */ split_page_owner (head, nr);
@@ -2494,8 +2499,7 @@ static void __split_huge_page(struct page *page, struct list_head *list,page_ref_add (head,2 );xa_unlock (&head->mapping->i_pages); } - -spin_unlock_irqrestore (&pgdat->lru_lock, flags); +local_irq_enable ();remap_page (head, nr);
@@ -2641,12 +2645,10 @@ bool can_split_huge_page(struct page *page, int *pextra_pins)int split_huge_page_to_list (struct page *page,struct list_head *list) {struct page *head =compound_head (page); -struct pglist_data *pgdata =NODE_DATA (page_to_nid (head));struct deferred_split *ds_queue =get_deferred_split_queue (head);struct anon_vma *anon_vma =NULL ;struct address_space *mapping =NULL ;int count, mapcount, extra_pins, ret; -unsigned long flags;pgoff_t end;VM_BUG_ON_PAGE (is_huge_zero_page (head), head);
@@ -2707,9 +2709,8 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)unmap_page (head);VM_BUG_ON_PAGE (compound_mapcount (head), head); -/* prevent PageLRU to go away from under us, and freeze lru stats */ -spin_lock_irqsave (&pgdata->lru_lock, flags); - +/* block interrupt reentry in xa_lock and spinlock */ +local_irq_disable ();if (mapping) {XA_STATE (xas, &mapping->i_pages,page_index (head));
@@ -2739,7 +2740,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)__dec_lruvec_page_state (head,NR_FILE_THPS ); } -__split_huge_page (page, list, end, flags); +__split_huge_page (page, list, end); ret =0 ; }else {if (IS_ENABLED (CONFIG_DEBUG_VM ) && mapcount) {
@@ -2753,7 +2754,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)spin_unlock (&ds_queue->split_queue_lock); fail:if (mapping)xa_unlock (&mapping->i_pages); -spin_unlock_irqrestore (&pgdata->lru_lock, flags); +local_irq_enable ();remap_page (head,thp_nr_pages (head)); ret = -EBUSY ; }
Signature · 1 line
1.8.3.1
PATCH v21 5/19 mm/vmscan: remove unnecessary lruvec adding
In reply to Alex ShiWe don't have to add a freeable page into lru and then remove from it. This change saves a couple of actions and makes the moving more clear.
The SetPageLRU needs to be kept before put_page_testzero for list integrity, otherwise:
#0 move_pages_to_lru #1 release_pages
if !put_page_testzero
if (put_page_testzero())
!PageLRU //skip lru_lock
SetPageLRU()
list_add(&page->lru,)
list_add(&page->lru,)--- mm/vmscan.c | 38 +++++++++++++++++++++++++------------- 1 file changed, 25 insertions(+), 13 deletions(-)
mm/vmscan.c +25 -13
@@ -1852,26 +1852,30 @@ static unsigned noinline_for_stack move_pages_to_lru(struct lruvec *lruvec,while (!list_empty (list)) { page =lru_to_page (list);VM_BUG_ON_PAGE (PageLRU (page), page); +list_del (&page->lru);if (unlikely (!page_evictable (page))) { -list_del (&page->lru);spin_unlock_irq (&pgdat->lru_lock);putback_lru_page (page);spin_lock_irq (&pgdat->lru_lock);continue ; } - lruvec =mem_cgroup_page_lruvec (page, pgdat); +/* +* The SetPageLRU needs to be kept here for list integrity. +* Otherwise: +* #0 move_pages_to_lru #1 release_pages +* if !put_page_testzero +* if (put_page_testzero()) +* !PageLRU //skip lru_lock +* SetPageLRU() +* list_add(&page->lru,) +* list_add(&page->lru,) +*/ SetPageLRU (page); - lru =page_lru (page); - nr_pages =thp_nr_pages (page); -update_lru_size (lruvec, lru,page_zonenum (page), nr_pages); -list_move (&page->lru, &lruvec->lists[lru]); - -if (put_page_testzero (page)) { +if (unlikely (put_page_testzero (page))) {__ClearPageLRU (page);__ClearPageActive (page); -del_page_from_lru_list (page, lruvec, lru);if (unlikely (PageCompound (page))) {spin_unlock_irq (&pgdat->lru_lock);
@@ -1879,11 +1883,19 @@ static unsigned noinline_for_stack move_pages_to_lru(struct lruvec *lruvec,spin_lock_irq (&pgdat->lru_lock); }else list_add (&page->lru, &pages_to_free); - }else { - nr_moved += nr_pages; -if (PageActive (page)) -workingset_age_nonresident (lruvec, nr_pages); + +continue ; } + + lruvec =mem_cgroup_page_lruvec (page, pgdat); + lru =page_lru (page); + nr_pages =thp_nr_pages (page); + +update_lru_size (lruvec, lru,page_zonenum (page), nr_pages); +list_add (&page->lru, &lruvec->lists[lru]); + nr_moved += nr_pages; +if (PageActive (page)) +workingset_age_nonresident (lruvec, nr_pages); }/*
Signature · 1 line
1.8.3.1
Re: [PATCH v21 05/19] mm/vmscan: remove unnecessary lruvec adding
Quoted 14 lines · Alex Shi
We don't have to add a freeable page into lru and then remove from it. This change saves a couple of actions and makes the moving more clear.
The SetPageLRU needs to be kept before
put_page_testzerofor list integrity, otherwise:#0
move_pages_to_lru#1release_pages
if !put_page_testzeroif (put_page_testzero()) !PageLRU //skip lru_lock SetPageLRU() list_add(&page->lru,) list_add(&page->lru,)
Quoted 11 lines
[akpm-de/[email protected]: coding style fixes]Acked-by Hugh Dickins <[email protected]>Acked-by Johannes Weiner <[email protected]>Cc Andrew Morton <akpm-de/[email protected]>Cc Johannes Weiner <[email protected]>Cc Tejun Heo <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Hugh Dickins <[email protected]>
Nice cleanup!
PATCH v21 6/19 mm/rmap: stop store reordering issue on page->mapping
In reply to Alex ShiHugh Dickins and Minchan Kim observed a long time issue which discussed here, but actully the mentioned fix missed.
https://lore.kernel.org/lkml/20150504031722.GA2768@blaptop/
The store reordering may cause problem in the scenario:
CPU 0 CPU1 do_anonymous_page page_add_new_anon_rmap() page->mapping = anon_vma + PAGE_MAPPING_ANON lru_cache_add_inactive_or_unevictable() spin_lock(lruvec->lock) SetPageLRU() spin_unlock(lruvec->lock) /* idletacking judged it as LRU * page so pass the page in * page_idle_clear_pte_refs */ page_idle_clear_pte_refs rmap_walk if PageAnon(page)
Johannes give detailed examples how the store reordering could cause a trouble:
The concern is the SetPageLRU may get reorder before 'page->mapping' setting, That would make CPU 1 will observe at page->mapping after observing PageLRU set on the page.
1. anon_vma + PAGE_MAPPING_ANON
That's the in-order scenario and is fine.
2. NULL
That's possible if the page->mapping store gets reordered to occur after SetPageLRU. That's fine too because we check for it.
3. anon_vma without the PAGE_MAPPING_ANON bit
That would be a problem and could lead to all kinds of undesirable behavior including crashes and data corruption.
Is it possible? AFAICT the compiler is allowed to tear the store to page->mapping and I don't see anything that would prevent it.
That said, I also don't see how the reader testing PageLRU under the lru_lock would prevent that in the first place. AFAICT we need that WRITE_ONCE() around the page->mapping assignment.
--- mm/rmap.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
mm/rmap.c +6 -1
@@ -1054,8 +1054,13 @@ static void __page_set_anon_rmap(struct page *page,if (!exclusive) anon_vma = anon_vma->root; +/* +* Prevent page->mapping from pointing to an anon_vma without +* the PAGE_MAPPING_ANON bit set. This could happen if the +* compiler stores anon_vma and then adds PAGE_MAPPING_ANON to it. +*/ anon_vma = (void *) anon_vma +PAGE_MAPPING_ANON ; - page->mapping = (struct address_space *) anon_vma; +WRITE_ONCE (page->mapping, (struct address_space *) anon_vma); page->index =linear_page_index (vma, address); }
Signature · 1 line
1.8.3.1
Re: [PATCH v21 06/19] mm/rmap: stop store reordering issue on page->mapping
updated for comments change from Johannes
From 2fd278b1ca6c3e260ad249808b62f671d8db5a7b Mon Sep 17 00:00:00 2001
From: Alex Shi <alex.shi-KPsoFbNs7GizrGE5bRqYAgC/[email protected]>
Date: Thu, 5 Nov 2020 11:38:24 +0800
Subject: [PATCH v21 06/19] mm/rmap: stop store reordering issue on
page->mapping
Hugh Dickins and Minchan Kim observed a long time issue which discussed here, but actully the mentioned fix missed.
https://lore.kernel.org/lkml/20150504031722.GA2768@blaptop/
The store reordering may cause problem in the scenario:
CPU 0 CPU1 do_anonymous_page page_add_new_anon_rmap() page->mapping = anon_vma + PAGE_MAPPING_ANON lru_cache_add_inactive_or_unevictable() spin_lock(lruvec->lock) SetPageLRU() spin_unlock(lruvec->lock) /* idletacking judged it as LRU * page so pass the page in * page_idle_clear_pte_refs */ page_idle_clear_pte_refs rmap_walk if PageAnon(page)
Johannes give detailed examples how the store reordering could cause a trouble:
"The concern is the SetPageLRU may get reorder before 'page->mapping' setting, That would make CPU 1 will observe at page->mapping after observing PageLRU set on the page.
1. anon_vma + PAGE_MAPPING_ANON
That's the in-order scenario and is fine.
2. NULL
That's possible if the page->mapping store gets reordered to occur after SetPageLRU. That's fine too because we check for it.
3. anon_vma without the PAGE_MAPPING_ANON bit
That would be a problem and could lead to all kinds of undesirable behavior including crashes and data corruption.
Is it possible? AFAICT the compiler is allowed to tear the store to page->mapping and I don't see anything that would prevent it.
That said, I also don't see how the reader testing PageLRU under the lru_lock would prevent that in the first place. AFAICT we need that WRITE_ONCE() around the page->mapping assignment."
--- mm/rmap.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
mm/rmap.c +7 -1
@@ -1054,8 +1054,14 @@ static void __page_set_anon_rmap(struct page *page,if (!exclusive) anon_vma = anon_vma->root; +/* +* page_idle does a lockless/optimistic rmap scan on page->mapping. +* Make sure the compiler doesn't split the stores of anon_vma and +* the PAGE_MAPPING_ANON type identifier, otherwise the rmap code +* could mistake the mapping for a struct address_space and crash. +*/ anon_vma = (void *) anon_vma +PAGE_MAPPING_ANON ; - page->mapping = (struct address_space *) anon_vma; +WRITE_ONCE (page->mapping, (struct address_space *) anon_vma); page->index =linear_page_index (vma, address); }
Signature · 1 line
1.8.3.1
Re: [PATCH v21 06/19] mm/rmap: stop store reordering issue on page->mapping
Quoted 54 lines · Alex Shi
From
2fd278b1ca6c3e260ad249808b62f671d8db5a7bMon Sep 17 00:00:00 2001
From: Alex Shi <[email protected]>
Date: Thu, 5 Nov 2020 11:38:24 +0800
Subject: [PATCH v21 06/19] mm/rmap: stop store reordering issue on
page->mappingHugh Dickins and Minchan Kim observed a long time issue which discussed here, but actully the mentioned fix missed.
https://lore.kernel.org/lkml/20150504031722.GA2768@blaptop/
The store reordering may cause problem in the scenario:CPU 0 CPU1 do_anonymous_page page_add_new_anon_rmap() page->mapping = anon_vma + PAGE_MAPPING_ANON lru_cache_add_inactive_or_unevictable() spin_lock(lruvec->lock) SetPageLRU() spin_unlock(lruvec->lock) /* idletacking judged it as LRU * page so pass the page in * page_idle_clear_pte_refs */ page_idle_clear_pte_refs rmap_walk if PageAnon(page)Johannes give detailed examples how the store reordering could cause a trouble:
"The concern is the SetPageLRU may get reorder before 'page->mapping' setting, That would make CPU 1 will observe atpage->mappingafter observing PageLRU set on the page.1.
anon_vma+PAGE_MAPPING_ANONThat's the in-order scenario and is fine.
2. NULL
That's possible if the
page->mappingstore gets reordered to occur after SetPageLRU. That's fine too because we check for it.3.
anon_vmawithout thePAGE_MAPPING_ANONbitThat would be a problem and could lead to all kinds of undesirable behavior including crashes and data corruption.
Is it possible? AFAICT the compiler is allowed to tear the store to
page->mappingand I don't see anything that would prevent it.That said, I also don't see how the reader testing PageLRU under the
lru_lockwould prevent that in the first place. AFAICT we need thatWRITE_ONCE()around thepage->mappingassignment."
Quoted 9 lines
Signed-off-by Alex Shi <[email protected]>Cc Johannes Weiner <[email protected]>Cc Andrew Morton <[email protected]>Cc Hugh Dickins <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Minchan Kim <[email protected]>Cc Vladimir Davydov <[email protected]>
Thanks Alex!
Hugh Dickins
Acked-byRe: [PATCH v21 06/19] mm/rmap: stop store reordering issue on page->mapping
In reply to Alex ShiQuoted 54 lines · Alex Shi
updated for comments change from Johannes
From
2fd278b1ca6c3e260ad249808b62f671d8db5a7bMon Sep 17 00:00:00 2001
From: Alex Shi <[email protected]>
Date: Thu, 5 Nov 2020 11:38:24 +0800
Subject: [PATCH v21 06/19] mm/rmap: stop store reordering issue on
page->mappingHugh Dickins and Minchan Kim observed a long time issue which discussed here, but actully the mentioned fix missed.
https://lore.kernel.org/lkml/20150504031722.GA2768@blaptop/
The store reordering may cause problem in the scenario:CPU 0 CPU1 do_anonymous_page page_add_new_anon_rmap() page->mapping = anon_vma + PAGE_MAPPING_ANON lru_cache_add_inactive_or_unevictable() spin_lock(lruvec->lock) SetPageLRU() spin_unlock(lruvec->lock) /* idletacking judged it as LRU * page so pass the page in * page_idle_clear_pte_refs */ page_idle_clear_pte_refs rmap_walk if PageAnon(page)Johannes give detailed examples how the store reordering could cause a trouble:
"The concern is the SetPageLRU may get reorder before 'page->mapping' setting, That would make CPU 1 will observe atpage->mappingafter observing PageLRU set on the page.1.
anon_vma+PAGE_MAPPING_ANONThat's the in-order scenario and is fine.
2. NULL
That's possible if the
page->mappingstore gets reordered to occur after SetPageLRU. That's fine too because we check for it.3.
anon_vmawithout thePAGE_MAPPING_ANONbitThat would be a problem and could lead to all kinds of undesirable behavior including crashes and data corruption.
Is it possible? AFAICT the compiler is allowed to tear the store to
page->mappingand I don't see anything that would prevent it.
Quoted 8 lines
That said, I also don't see how the reader testing PageLRU under the
lru_lockwould prevent that in the first place. AFAICT we need thatWRITE_ONCE()around thepage->mappingassignment."Signed-off-by Alex Shi <[email protected]>Cc Johannes Weiner <[email protected]>Cc Andrew Morton <[email protected]>Cc Hugh Dickins <[email protected]>
Many thanks to Johannes for spotting my falsehood in the next patch, and to Alex for making it true with this patch. As I just remarked against the v20, I do have some more of these WRITE_ONCEs, but consider them merely theoretical: so please don't let me hold this series up.
Andrew, I am hoping that Alex's v21 will appear in the next mmotm?
Thanks,
Hugh
Quoted 31 lines
Cc Matthew Wilcox <[email protected]>Cc Minchan Kim <[email protected]>Cc Vladimir Davydov <[email protected]>--- mm/rmap.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)mm/rmap.c +7
-1@@ -1054,8 +1054,14 @@ static void __page_set_anon_rmap(struct page *page,if (!exclusive) anon_vma = anon_vma->root; +/* +* page_idle does a lockless/optimistic rmap scan on page->mapping. +* Make sure the compiler doesn't split the stores of anon_vma and +* the PAGE_MAPPING_ANON type identifier, otherwise the rmap code +* could mistake the mapping for a struct address_space and crash. +*/ anon_vma = (void *) anon_vma +PAGE_MAPPING_ANON ; - page->mapping = (struct address_space *) anon_vma; +WRITE_ONCE (page->mapping, (struct address_space *) anon_vma); page->index =linear_page_index (vma, address); }Signature · 1 line
1.8.3.1
PATCH v21 7/19 mm: page_idle_get_page() does not need lru_lock
In reply to Alex ShiFrom: Hugh Dickins <[email protected]>
It is necessary for page_idle_get_page() to recheck PageLRU() after get_page_unless_zero(), but holding lru_lock around that serves no useful purpose, and adds to lru_lock contention: delete it.
See https://lore.kernel.org/lkml/20150504031722.GA2768@blaptop for the discussion that led to lru_lock there; but __page_set_anon_rmap() now uses WRITE_ONCE(), and I see no other risk in page_idle_clear_pte_refs() using rmap_walk() (beyond the risk of racing PageAnon->PageKsm, mostly but not entirely prevented by page_count() check in ksm.c's write_protect_page(): that risk being shared with page_referenced() and not helped by lru_lock).
--- mm/page_idle.c | 4 ---- 1 file changed, 4 deletions(-)
mm/page_idle.c +0 -4
@@ -32,19 +32,15 @@static struct page *page_idle_get_page (unsigned long pfn) {struct page *page =pfn_to_online_page (pfn); -pg_data_t *pgdat;if (!page || !PageLRU (page) || !get_page_unless_zero (page))return NULL ; - pgdat =page_pgdat (page); -spin_lock_irq (&pgdat->lru_lock);if (unlikely (!PageLRU (page))) {put_page (page); page =NULL ; } -spin_unlock_irq (&pgdat->lru_lock);return page; }
Signature · 1 line
1.8.3.1
Re: [PATCH v21 07/19] mm: page_idle_get_page() does not need lru_lock
Quoted 14 lines · Alex Shi
From: Hugh Dickins <[email protected]>
It is necessary for
page_idle_get_page()to recheckPageLRU()afterget_page_unless_zero(), but holdinglru_lockaround that serves no useful purpose, and adds tolru_lockcontention: delete it.See https://lore.kernel.org/lkml/20150504031722.GA2768@blaptop for the discussion that led to
lru_lockthere; but__page_set_anon_rmap()now usesWRITE_ONCE(), and I see no other risk inpage_idle_clear_pte_refs()usingrmap_walk()(beyond the risk of racingPageAnon->PageKsm, mostly but not entirely prevented bypage_count()check in ksm.c'swrite_protect_page(): that risk being shared withpage_referenced()and not helped bylru_lock).
Quoted 9 lines
Signed-off-by Hugh Dickins <[email protected]>Signed-off-by Alex Shi <[email protected]>Cc Andrew Morton <[email protected]>Cc Vladimir Davydov <[email protected]>Cc Vlastimil Babka <[email protected]>Cc Minchan Kim <[email protected]>Cc Alex Shi <[email protected]>
Re: [PATCH v21 07/19] mm: page_idle_get_page() does not need lru_lock
In reply to Alex ShiQuoted 34 lines · Alex Shi
From: Hugh Dickins <[email protected]>
It is necessary for
page_idle_get_page()to recheckPageLRU()afterget_page_unless_zero(), but holdinglru_lockaround that serves no useful purpose, and adds tolru_lockcontention: delete it.See https://lore.kernel.org/lkml/20150504031722.GA2768@blaptop for the discussion that led to
lru_lockthere; but__page_set_anon_rmap()now usesWRITE_ONCE(), and I see no other risk inpage_idle_clear_pte_refs()usingrmap_walk()(beyond the risk of racingPageAnon->PageKsm, mostly but not entirely prevented bypage_count()check in ksm.c'swrite_protect_page(): that risk being shared withpage_referenced()and not helped bylru_lock).Signed-off-by Hugh Dickins <[email protected]>Cc Andrew Morton <akpm-de/[email protected]>Cc Vladimir Davydov <[email protected]>Cc Vlastimil Babka <[email protected]>Cc Minchan Kim <[email protected]>Cc Alex Shi <alex.shi-KPsoFbNs7GizrGE5bRqYAgC/[email protected]>--- mm/page_idle.c | 4 ---- 1 file changed, 4 deletions(-)mm/page_idle.c +0
-0@@ -32,19 +32,15 @@static struct page *page_idle_get_page (unsigned long pfn)
mm/page_idle.c line 42
Quoted 10 lines
mm/page_idle.c +0
-3{struct page *page =pfn_to_online_page (pfn); -pg_data_t *pgdat;if (!page || !PageLRU (page) || !get_page_unless_zero (page))return NULL ; - pgdat =page_pgdat (page); -spin_lock_irq (&pgdat->lru_lock);
get_page_unless_zero() is a full memory barrier. But do we need a compiler barrier here to prevent the compiler to cache PageLRU() results here? Otherwise looks OK to me,
Best Regards,
Huang, Ying
Quoted 12 lines
mm/page_idle.c +0
-2if (unlikely (!PageLRU (page))) {put_page (page); page =NULL ; } -spin_unlock_irq (&pgdat->lru_lock);return page; } --1.8.3.1
Vlastimil Babka
Acked-byRe: [PATCH v21 07/19] mm: page_idle_get_page() does not need lru_lock
Quoted 43 lines · huang ying
Quoted 42 lines · Alex Shi
From: Hugh Dickins <[email protected]>
It is necessary for
page_idle_get_page()to recheckPageLRU()afterget_page_unless_zero(), but holdinglru_lockaround that serves no useful purpose, and adds tolru_lockcontention: delete it.See https://lore.kernel.org/lkml/20150504031722.GA2768@blaptop for the discussion that led to
lru_lockthere; but__page_set_anon_rmap()now usesWRITE_ONCE(), and I see no other risk inpage_idle_clear_pte_refs()usingrmap_walk()(beyond the risk of racingPageAnon->PageKsm, mostly but not entirely prevented bypage_count()check in ksm.c'swrite_protect_page(): that risk being shared withpage_referenced()and not helped bylru_lock).Signed-off-by Hugh Dickins <[email protected]>Cc Andrew Morton <akpm-de/[email protected]>Cc Vladimir Davydov <[email protected]>Cc Vlastimil Babka <[email protected]>Cc Minchan Kim <[email protected]>Cc Alex Shi <alex.shi-KPsoFbNs7GizrGE5bRqYAgC/[email protected]>--- mm/page_idle.c | 4 ---- 1 file changed, 4 deletions(-)mm/page_idle.c +0
-1@@ -32,19 +32,15 @@static struct page *page_idle_get_page (unsigned long pfn) {struct page *page =pfn_to_online_page (pfn); -pg_data_t *pgdat;if (!page || !PageLRU (page) || !get_page_unless_zero (page))return NULL ;
Quoted 6 lines
mm/page_idle.c line 42
Quoted 2 lines
mm/page_idle.c +0
-2- pgdat =page_pgdat (page); -spin_lock_irq (&pgdat->lru_lock);
get_page_unless_zero()is a full memory barrier. But do we need a compiler barrier here to prevent the compiler to cachePageLRU()results here? Otherwise looks OK to me,
I think the compiler barrier is also implied by the full memory barrier and prevents the caching.
Quoted 18 lines
Acked-by "Huang, Ying" <[email protected]>Best Regards,
Huang, YingQuoted 12 lines
mm/page_idle.c +0
-2if (unlikely (!PageLRU (page))) {put_page (page); page =NULL ; } -spin_unlock_irq (&pgdat->lru_lock);return page; } --1.8.3.1
PATCH v21 8/19 mm/memcg: add debug checking in lock_page_memcg
In reply to Alex ShiAdd a debug checking in lock_page_memcg, then we could get alarm if anything wrong here.
--- mm/memcontrol.c | 6 ++++++ 1 file changed, 6 insertions(+)
mm/memcontrol.c +6 -0
@@ -2121,6 +2121,12 @@ struct mem_cgroup *lock_page_memcg(struct page *page)if (unlikely (!memcg))return NULL ; +#ifdef CONFIG_PROVE_LOCKING +local_irq_save (flags); +might_lock (&memcg->move_lock); +local_irq_restore (flags); +#endif +if (atomic_read (&memcg->moving_account) <=0 )return memcg;
Signature · 1 line
1.8.3.1
PATCH v21 9/19 mm/swap.c: fold vm event PGROTATED into pagevec_move_tail_fn
In reply to Alex ShiFold the PGROTATED event collection into pagevec_move_tail_fn call back func like other funcs does in pagevec_lru_move_fn. Thus we could save func call pagevec_move_tail().
Now all usage of pagevec_lru_move_fn are same and no needs of its 3rd parameter.
It's just simply the calling. No functional change.
--- mm/swap.c | 65 ++++++++++++++++++++++----------------------------------------- 1 file changed, 23 insertions(+), 42 deletions(-)
mm/swap.c +23 -42
@@ -204,8 +204,7 @@ int get_kernel_page(unsigned long start, int write, struct page **pages)EXPORT_SYMBOL_GPL (get_kernel_page);static void pagevec_lru_move_fn (struct pagevec *pvec, -void (*move_fn)(struct page *page,struct lruvec *lruvec,void *arg), -void *arg) +void (*move_fn)(struct page *page,struct lruvec *lruvec)) {int i;struct pglist_data *pgdat =NULL ;
@@ -224,7 +223,7 @@ static void pagevec_lru_move_fn(struct pagevec *pvec, } lruvec =mem_cgroup_page_lruvec (page, pgdat); - (*move_fn)(page, lruvec, arg); + (*move_fn)(page, lruvec); }if (pgdat)spin_unlock_irqrestore (&pgdat->lru_lock, flags);
@@ -232,35 +231,22 @@ static void pagevec_lru_move_fn(struct pagevec *pvec,pagevec_reinit (pvec); } -static void pagevec_move_tail_fn (struct page *page,struct lruvec *lruvec, -void *arg) +static void pagevec_move_tail_fn (struct page *page,struct lruvec *lruvec) { -int *pgmoved = arg; -if (PageLRU (page) && !PageUnevictable (page)) {del_page_from_lru_list (page, lruvec,page_lru (page));ClearPageActive (page);add_page_to_lru_list_tail (page, lruvec,page_lru (page)); - (*pgmoved) +=thp_nr_pages (page); +__count_vm_events (PGROTATED ,thp_nr_pages (page)); } }/* -* pagevec_move_tail() must be called with IRQ disabled. -* Otherwise this may cause nasty races. -*/ -static void pagevec_move_tail (struct pagevec *pvec) -{ -int pgmoved =0 ; - -pagevec_lru_move_fn (pvec, pagevec_move_tail_fn, &pgmoved); -__count_vm_events (PGROTATED , pgmoved); -} - -/* * Writeback is about to end against a page which has been marked for immediate * reclaim. If it still appears to be reclaimable, move it to the tail of the * inactive list. +* +* rotate_reclaimable_page() must disable IRQs, to prevent nasty races. */ void rotate_reclaimable_page (struct page *page) {
@@ -273,7 +259,7 @@ void rotate_reclaimable_page(struct page *page)local_lock_irqsave (&lru_rotate.lock, flags); pvec =this_cpu_ptr (&lru_rotate.pvec);if (!pagevec_add (pvec, page) ||PageCompound (page)) -pagevec_move_tail (pvec); +pagevec_lru_move_fn (pvec, pagevec_move_tail_fn);local_unlock_irqrestore (&lru_rotate.lock, flags); } }
@@ -315,8 +301,7 @@ void lru_note_cost_page(struct page *page) page_is_file_lru(page), thp_nr_pages(page)); } -static void __activate_page (struct page *page,struct lruvec *lruvec, -void *arg) +static void __activate_page (struct page *page,struct lruvec *lruvec) {if (PageLRU (page) && !PageActive (page) && !PageUnevictable (page)) {int lru =page_lru_base_type (page);
@@ -340,7 +325,7 @@ static void activate_page_drain(int cpu)struct pagevec *pvec = &per_cpu (lru_pvecs.activate_page, cpu);if (pagevec_count (pvec)) -pagevec_lru_move_fn (pvec, __activate_page,NULL ); +pagevec_lru_move_fn (pvec, __activate_page); }static bool need_activate_page_drain (int cpu)
@@ -358,7 +343,7 @@ static void activate_page(struct page *page) pvec =this_cpu_ptr (&lru_pvecs.activate_page);get_page (page);if (!pagevec_add (pvec, page) ||PageCompound (page)) -pagevec_lru_move_fn (pvec, __activate_page,NULL ); +pagevec_lru_move_fn (pvec, __activate_page);local_unlock (&lru_pvecs.lock); } }
@@ -374,7 +359,7 @@ static void activate_page(struct page *page) page =compound_head (page);spin_lock_irq (&pgdat->lru_lock); -__activate_page (page,mem_cgroup_page_lruvec (page, pgdat),NULL ); +__activate_page (page,mem_cgroup_page_lruvec (page, pgdat)); spin_unlock_irq(&pgdat->lru_lock); }#endif
@@ -525,8 +510,7 @@ void lru_cache_add_inactive_or_unevictable(struct page *page,* be write it out by flusher threads as this is much more effective * than the single-page writeout from reclaim. */ -static void lru_deactivate_file_fn (struct page *page,struct lruvec *lruvec, -void *arg) +static void lru_deactivate_file_fn (struct page *page,struct lruvec *lruvec) {int lru;bool active;
@@ -573,8 +557,7 @@ static void lru_deactivate_file_fn(struct page *page, struct lruvec *lruvec, } } -static void lru_deactivate_fn (struct page *page,struct lruvec *lruvec, -void *arg) +static void lru_deactivate_fn (struct page *page,struct lruvec *lruvec) {if (PageLRU (page) &&PageActive (page) && !PageUnevictable (page)) {int lru =page_lru_base_type (page);
@@ -591,8 +574,7 @@ static void lru_deactivate_fn(struct page *page, struct lruvec *lruvec, } } -static void lru_lazyfree_fn (struct page *page,struct lruvec *lruvec, -void *arg) +static void lru_lazyfree_fn (struct page *page,struct lruvec *lruvec) {if (PageLRU (page) &&PageAnon (page) &&PageSwapBacked (page) && !PageSwapCache (page) && !PageUnevictable (page)) {
@@ -636,21 +618,21 @@ void lru_add_drain_cpu(int cpu)/* No harm done if a racing interrupt already did this */ local_lock_irqsave (&lru_rotate.lock, flags); -pagevec_move_tail (pvec); +pagevec_lru_move_fn (pvec, pagevec_move_tail_fn);local_unlock_irqrestore (&lru_rotate.lock, flags); } pvec = &per_cpu (lru_pvecs.lru_deactivate_file, cpu);if (pagevec_count (pvec)) -pagevec_lru_move_fn (pvec, lru_deactivate_file_fn,NULL ); +pagevec_lru_move_fn (pvec, lru_deactivate_file_fn); pvec = &per_cpu (lru_pvecs.lru_deactivate, cpu);if (pagevec_count (pvec)) -pagevec_lru_move_fn (pvec, lru_deactivate_fn,NULL ); +pagevec_lru_move_fn (pvec, lru_deactivate_fn); pvec = &per_cpu (lru_pvecs.lru_lazyfree, cpu);if (pagevec_count (pvec)) -pagevec_lru_move_fn (pvec, lru_lazyfree_fn,NULL ); +pagevec_lru_move_fn (pvec, lru_lazyfree_fn);activate_page_drain (cpu); }
@@ -679,7 +661,7 @@ void deactivate_file_page(struct page *page) pvec =this_cpu_ptr (&lru_pvecs.lru_deactivate_file);if (!pagevec_add (pvec, page) ||PageCompound (page)) -pagevec_lru_move_fn (pvec, lru_deactivate_file_fn,NULL ); +pagevec_lru_move_fn (pvec, lru_deactivate_file_fn);local_unlock (&lru_pvecs.lock); } }
@@ -701,7 +683,7 @@ void deactivate_page(struct page *page) pvec =this_cpu_ptr (&lru_pvecs.lru_deactivate);get_page (page);if (!pagevec_add (pvec, page) ||PageCompound (page)) -pagevec_lru_move_fn (pvec, lru_deactivate_fn,NULL ); +pagevec_lru_move_fn (pvec, lru_deactivate_fn);local_unlock (&lru_pvecs.lock); } }
@@ -723,7 +705,7 @@ void mark_page_lazyfree(struct page *page) pvec =this_cpu_ptr (&lru_pvecs.lru_lazyfree);get_page (page);if (!pagevec_add (pvec, page) ||PageCompound (page)) -pagevec_lru_move_fn (pvec, lru_lazyfree_fn,NULL ); +pagevec_lru_move_fn (pvec, lru_lazyfree_fn);local_unlock (&lru_pvecs.lock); } }
@@ -977,8 +959,7 @@ void __pagevec_release(struct pagevec *pvec) }EXPORT_SYMBOL (__pagevec_release); -static void __pagevec_lru_add_fn (struct page *page,struct lruvec *lruvec, -void *arg) +static void __pagevec_lru_add_fn (struct page *page,struct lruvec *lruvec) {enum lru_list lru;int was_unevictable =TestClearPageUnevictable (page);
@@ -1037,7 +1018,7 @@ static void __pagevec_lru_add_fn(struct page *page, struct lruvec *lruvec,*/ void __pagevec_lru_add (struct pagevec *pvec) { -pagevec_lru_move_fn (pvec, __pagevec_lru_add_fn,NULL ); +pagevec_lru_move_fn (pvec, __pagevec_lru_add_fn); }/**
Signature · 1 line
1.8.3.1
PATCH v21 10/19 mm/lru: move lock into lru_note_cost
In reply to Alex ShiWe have to move lru_lock into lru_note_cost, since it cycle up on memcg tree, for future per lruvec lru_lock replace. It's a bit ugly and may cost a bit more locking, but benefit from multiple memcg locking could cover the lost.
--- mm/swap.c | 3 +++ mm/vmscan.c | 4 +--- mm/workingset.c | 2 -- 3 files changed, 4 insertions(+), 5 deletions(-)
mm/swap.c +3 -0
@@ -268,7 +268,9 @@ void lru_note_cost(struct lruvec *lruvec, bool file, unsigned int nr_pages) { do { unsigned long lrusize; + struct pglist_data *pgdat = lruvec_pgdat(lruvec); + spin_lock_irq(&pgdat->lru_lock); /* Record cost event */ if (file) lruvec->file_cost += nr_pages;
@@ -292,6 +294,7 @@ void lru_note_cost(struct lruvec *lruvec, bool file, unsigned int nr_pages) lruvec->file_cost /=2 ; lruvec->anon_cost /=2 ; } +spin_unlock_irq (&pgdat->lru_lock); }while ((lruvec =parent_lruvec (lruvec))); }
mm/vmscan.c +1 -3
@@ -1973,19 +1973,17 @@ static int current_may_throttle(void) &stat, false);spin_lock_irq (&pgdat->lru_lock); -move_pages_to_lru (lruvec, &page_list);__mod_node_page_state (pgdat,NR_ISOLATED_ANON + file, -nr_taken); -lru_note_cost (lruvec, file, stat.nr_pageout); item =current_is_kswapd () ?PGSTEAL_KSWAPD :PGSTEAL_DIRECT ;if (!cgroup_reclaim (sc))__count_vm_events (item, nr_reclaimed);__count_memcg_events (lruvec_memcg (lruvec), item, nr_reclaimed);__count_vm_events (PGSTEAL_ANON + file, nr_reclaimed); -spin_unlock_irq (&pgdat->lru_lock); +lru_note_cost (lruvec, file, stat.nr_pageout);mem_cgroup_uncharge_list (&page_list);free_unref_page_list (&page_list);
mm/workingset.c +0 -2
@@ -381,9 +381,7 @@ void workingset_refault(struct page *page, void *shadow)if (workingset) {SetPageWorkingset (page);/* XXX: Move to lru_cache_add() when it supports new vs putback */ -spin_lock_irq (&page_pgdat (page)->lru_lock);lru_note_cost_page (page); -spin_unlock_irq (&page_pgdat (page)->lru_lock);inc_lruvec_state (lruvec,WORKINGSET_RESTORE_BASE + file); } out:
Signature · 1 line
1.8.3.1
PATCH v21 11/19 mm/vmscan: remove lruvec reget in move_pages_to_lru
In reply to Alex ShiIsolated page shouldn't be recharged by memcg since the memcg migration isn't possible at the time. All pages were isolated from the same lruvec (and isolation inhibits memcg migration). So remove unnecessary regetting.
Thanks to Alexander Duyck for pointing this out.
--- mm/vmscan.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
mm/vmscan.c +6 -1
@@ -1887,7 +1887,12 @@ static unsigned noinline_for_stack move_pages_to_lru(struct lruvec *lruvec,continue ; } - lruvec =mem_cgroup_page_lruvec (page, pgdat); +/* +* All pages were isolated from the same lruvec (and isolation +* inhibits memcg migration). +*/ +VM_BUG_ON_PAGE (mem_cgroup_page_lruvec (page,page_pgdat (page)) + != lruvec, page); lru =page_lru (page); nr_pages =thp_nr_pages (page);
Signature · 1 line
1.8.3.1
PATCH v21 12/19 mm/mlock: remove lru_lock on TestClearPageMlocked
In reply to Alex ShiIn the func munlock_vma_page, comments mentained lru_lock needed for serialization with split_huge_pages. But the page must be PageLocked as well as pages in split_huge_page series funcs. Thus the PageLocked is enough to serialize both funcs.
Further more, Hugh Dickins pointed: before splitting in split_huge_page_to_list, the page was unmap_page() to remove pmd/ptes which protect the page from munlock. Thus, no needs to guard __split_huge_page_tail for mlock clean, just keep the lru_lock there for isolation purpose.
LKP found a preempt issue on __mod_zone_page_state which need change to mod_zone_page_state. Thanks!
--- mm/mlock.c | 26 +++++--------------------- 1 file changed, 5 insertions(+), 21 deletions(-)
mm/mlock.c +5 -21
@@ -187,40 +187,24 @@ static void __munlock_isolation_failed(struct page *page)unsigned int munlock_vma_page (struct page *page) {int nr_pages; -pg_data_t *pgdat =page_pgdat (page);/* For try_to_munlock() and to serialize with page migration */ BUG_ON (!PageLocked (page)); -VM_BUG_ON_PAGE (PageTail (page), page); -/* -* Serialize with any parallel __split_huge_page_refcount() which -* might otherwise copy PageMlocked to part of the tail pages before -* we clear it in the head page. It also stabilizes thp_nr_pages(). -*/ -spin_lock_irq (&pgdat->lru_lock); -if (!TestClearPageMlocked (page)) {/* Potentially, PTE-mapped THP: do not skip the rest PTEs */ - nr_pages =1 ; - goto unlock_out; +return 0 ; } nr_pages =thp_nr_pages (page); -__mod_zone_page_state (page_zone (page),NR_MLOCK , -nr_pages); +mod_zone_page_state (page_zone (page),NR_MLOCK , -nr_pages); -if (__munlock_isolate_lru_page (page, true)) { -spin_unlock_irq (&pgdat->lru_lock); +if (!isolate_lru_page (page))__munlock_isolated_page (page); - goto out; - } -__munlock_isolation_failed (page); - -unlock_out: -spin_unlock_irq (&pgdat->lru_lock); +else +__munlock_isolation_failed (page); -out:return nr_pages -1 ; }
Signature · 1 line
1.8.3.1
Re: [PATCH v21 12/19] mm/mlock: remove lru_lock on TestClearPageMlocked
Quoted 14 lines · Alex Shi
In the func
munlock_vma_page, comments mentainedlru_lockneeded for serialization withsplit_huge_pages. But the page must be PageLocked as well as pages insplit_huge_pageseries funcs. Thus the PageLocked is enough to serialize both funcs.Further more, Hugh Dickins pointed: before splitting in
split_huge_page_to_list, the page wasunmap_page()to remove pmd/ptes which protect the page from munlock. Thus, no needs to guard__split_huge_page_tailfor mlock clean, just keep thelru_lockthere for isolation purpose.LKP found a preempt issue on
__mod_zone_page_statewhich need change tomod_zone_page_state. Thanks!
Quoted 8 lines
Acked-by Hugh Dickins <[email protected]>Acked-by Johannes Weiner <[email protected]>Cc Kirill A. Shutemov <kirill.shutemov-VuQAYsv1563Yd54FQh9/[email protected]>Cc Vlastimil Babka <[email protected]>Cc Andrew Morton <akpm-de/[email protected]>
Nit below:
Quoted 9 lines
--- mm/mlock.c | 26 +++++--------------------- 1 file changed, 5 insertions(+), 21 deletions(-)mm/mlock.c +0
-0
mm/mlock.c static void __munlock_isolation_failed(struct page *page) · line 191
Quoted 6 lines
mm/mlock.c +0
-1@@ -187,40 +187,24 @@ static void __munlock_isolation_failed(struct page *page)unsigned int munlock_vma_page (struct page *page) {int nr_pages; -pg_data_t *pgdat =page_pgdat (page);/* For try_to_munlock() and to serialize with page migration */
Now the reasons for locking are expanded?
Quoted 40 lines
mm/mlock.c +5
-20BUG_ON (!PageLocked (page)); -VM_BUG_ON_PAGE (PageTail (page), page); -/* -* Serialize with any parallel __split_huge_page_refcount() which -* might otherwise copy PageMlocked to part of the tail pages before -* we clear it in the head page. It also stabilizes thp_nr_pages(). -*/ -spin_lock_irq (&pgdat->lru_lock); -if (!TestClearPageMlocked (page)) {/* Potentially, PTE-mapped THP: do not skip the rest PTEs */ - nr_pages =1 ; - goto unlock_out; +return 0 ; } nr_pages =thp_nr_pages (page); -__mod_zone_page_state (page_zone (page),NR_MLOCK , -nr_pages); +mod_zone_page_state (page_zone (page),NR_MLOCK , -nr_pages); -if (__munlock_isolate_lru_page (page, true)) { -spin_unlock_irq (&pgdat->lru_lock); +if (!isolate_lru_page (page))__munlock_isolated_page (page); - goto out; - } -__munlock_isolation_failed (page); - -unlock_out: -spin_unlock_irq (&pgdat->lru_lock); +else +__munlock_isolation_failed (page); -out:return nr_pages -1 ; }
PATCH v21 13/19 mm/mlock: remove __munlock_isolate_lru_page
In reply to Alex ShiThe func only has one caller, remove it to clean up code and simplify code.
--- mm/mlock.c | 31 +++++++++---------------------- 1 file changed, 9 insertions(+), 22 deletions(-)
mm/mlock.c +9 -22
@@ -106,26 +106,6 @@ void mlock_vma_page(struct page *page) }/* -* Isolate a page from LRU with optional get_page() pin. -* Assumes lru_lock already held and page already pinned. -*/ -static bool __munlock_isolate_lru_page (struct page *page,bool getpage) -{ -if (PageLRU (page)) { -struct lruvec *lruvec; - - lruvec =mem_cgroup_page_lruvec (page,page_pgdat (page)); -if (getpage) -get_page (page); -ClearPageLRU (page); -del_page_from_lru_list (page, lruvec,page_lru (page)); -return true; - } - -return false; -} - -/* * Finish munlock after successful page isolation * * Page must be locked. This is a wrapper for try_to_munlock()
@@ -296,9 +276,16 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone)* We already have pin from follow_page_mask() * so we can spare the get_page() here. */ -if (__munlock_isolate_lru_page (page, false)) +if (PageLRU (page)) { +struct lruvec *lruvec; + +ClearPageLRU (page); + lruvec =mem_cgroup_page_lruvec (page, +page_pgdat (page)); +del_page_from_lru_list (page, lruvec, +page_lru (page));continue ; -else + }else __munlock_isolation_failed (page); }else { delta_munlocked++;
Signature · 1 line
1.8.3.1
Re: [PATCH v21 13/19] mm/mlock: remove __munlock_isolate_lru_page
Quoted 12 lines · Alex Shi
The func only has one caller, remove it to clean up code and simplify code.
Signed-off-by Alex Shi <[email protected]>Acked-by Hugh Dickins <[email protected]>Acked-by Johannes Weiner <[email protected]>Cc Hugh Dickins <[email protected]>Cc Kirill A. Shutemov <[email protected]>Cc Vlastimil Babka <[email protected]>Cc Andrew Morton <[email protected]>
Nit below:
Quoted 42 lines
--- mm/mlock.c | 31 +++++++++---------------------- 1 file changed, 9 insertions(+), 22 deletions(-)mm/mlock.c +2
-21@@ -106,26 +106,6 @@ void mlock_vma_page(struct page *page) }/* -* Isolate a page from LRU with optional get_page() pin. -* Assumes lru_lock already held and page already pinned. -*/ -static bool __munlock_isolate_lru_page (struct page *page,bool getpage) -{ -if (PageLRU (page)) { -struct lruvec *lruvec; - - lruvec =mem_cgroup_page_lruvec (page,page_pgdat (page)); -if (getpage) -get_page (page); -ClearPageLRU (page); -del_page_from_lru_list (page, lruvec,page_lru (page)); -return true; - } - -return false; -} - -/* * Finish munlock after successful page isolation * * Page must be locked. This is a wrapper for try_to_munlock() @@ -296,9 +276,16 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone)* We already have pin from follow_page_mask() * so we can spare the get_page() here. */ -if (__munlock_isolate_lru_page (page, false)) +if (PageLRU (page)) { +struct lruvec *lruvec;
mm/mlock.c static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone) · line 289
Quoted 10 lines
mm/mlock.c +7
-1+ +ClearPageLRU (page); + lruvec =mem_cgroup_page_lruvec (page, +page_pgdat (page)); +del_page_from_lru_list (page, lruvec, +page_lru (page));continue ; - else + }else __munlock_isolation_failed (page);
IIRC coding styles says that once the if () part uses brackets, the else part should too, even if it's single line.
Quoted 3 lines
mm/mlock.c +0
-0}else { delta_munlocked++;
PATCH v21 14/19 mm/lru: introduce TestClearPageLRU
In reply to Alex ShiCurrently lru_lock still guards both lru list and page's lru bit, that's ok. but if we want to use specific lruvec lock on the page, we need to pin down the page's lruvec/memcg during locking. Just taking lruvec lock first may be undermined by the page's memcg charge/migration. To fix this problem, we will clear the lru bit out of locking and use it as pin down action to block the page isolation in memcg changing.
So now a standard steps of page isolation is following:
1, get_page(); #pin the page avoid to be free 2, TestClearPageLRU(); #block other isolation like memcg change 3, spin_lock on lru_lock; #serialize lru list access 4, delete page from lru list;
This patch start with the first part: TestClearPageLRU, which combines PageLRU check and ClearPageLRU into a macro func TestClearPageLRU. This function will be used as page isolation precondition to prevent other isolations some where else. Then there are may !PageLRU page on lru list, need to remove BUG() checking accordingly.
There 2 rules for lru bit now:
1, the lru bit still indicate if a page on lru list, just in some
temporary moment(isolating), the page may have no lru bit when it's on lru list. but the page still must be on lru list when the lru bit set.
2, have to remove lru bit before delete it from lru list.
As Andrew Morton mentioned this change would dirty cacheline for page isn't on LRU. But the lost would be acceptable in Rong Chen <[email protected]> report:
https://lore.kernel.org/lkml/20200304090301.GB5972@shao2-debian/
--- include/linux/page-flags.h | 1 + mm/mlock.c | 3 +-- mm/vmscan.c | 39 +++++++++++++++++++-------------------- 3 files changed, 21 insertions(+), 22 deletions(-)
include/linux/page-flags.h +1 -0
@@ -335,6 +335,7 @@ static inline void page_init_poison(struct page *page, size_t size) PAGEFLAG(Dirty, dirty, PF_HEAD) TESTSCFLAG(Dirty, dirty, PF_HEAD) __CLEARPAGEFLAG(Dirty, dirty, PF_HEAD) PAGEFLAG(LRU, lru, PF_HEAD) __CLEARPAGEFLAG(LRU, lru, PF_HEAD) + TESTCLEARFLAG(LRU, lru, PF_HEAD) PAGEFLAG(Active, active, PF_HEAD) __CLEARPAGEFLAG(Active, active, PF_HEAD) TESTCLEARFLAG(Active, active, PF_HEAD)PAGEFLAG (Workingset, workingset,PF_HEAD )
mm/mlock.c +1 -2
@@ -276,10 +276,9 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone)* We already have pin from follow_page_mask() * so we can spare the get_page() here. */ -if (PageLRU (page)) { +if (TestClearPageLRU (page)) {struct lruvec *lruvec; -ClearPageLRU (page); lruvec =mem_cgroup_page_lruvec (page,page_pgdat (page)); del_page_from_lru_list(page, lruvec,
mm/vmscan.c +19 -20
@@ -1542,7 +1542,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,*/ int __isolate_lru_page (struct page *page,isolate_mode_t mode) { -int ret = -EINVAL ; +int ret = -EBUSY ;/* Only take pages on the LRU. */ if (!PageLRU (page))
@@ -1552,8 +1552,6 @@ int __isolate_lru_page(struct page *page, isolate_mode_t mode)if (PageUnevictable (page) && !(mode &ISOLATE_UNEVICTABLE ))return ret; - ret = -EBUSY ; -/* * To minimise LRU disruption, the caller can indicate that it only * wants to isolate pages it will be able to operate on without
@@ -1600,8 +1598,10 @@ int __isolate_lru_page(struct page *page, isolate_mode_t mode)* sure the page is not being freed elsewhere -- the * page release code relies on it. */ -ClearPageLRU (page); - ret =0 ; +if (TestClearPageLRU (page)) + ret =0 ; +else +put_page (page); }return ret;
@@ -1667,8 +1667,6 @@ static unsigned long isolate_lru_pages(unsigned long nr_to_scan, page =lru_to_page (src);prefetchw_prev_lru_page (page, src, flags); -VM_BUG_ON_PAGE (!PageLRU (page), page); - nr_pages =compound_nr (page); total_scan += nr_pages;
@@ -1765,21 +1763,18 @@ int isolate_lru_page(struct page *page)VM_BUG_ON_PAGE (!page_count (page), page);WARN_RATELIMIT (PageTail (page),"trying to isolate tail page" ); -if (PageLRU (page)) { +if (TestClearPageLRU (page)) {pg_data_t *pgdat =page_pgdat (page);struct lruvec *lruvec; -spin_lock_irq (&pgdat->lru_lock); +get_page (page); lruvec =mem_cgroup_page_lruvec (page, pgdat); -if (PageLRU (page)) { -int lru =page_lru (page); -get_page (page); -ClearPageLRU (page); -del_page_from_lru_list (page, lruvec, lru); - ret =0 ; - } +spin_lock_irq (&pgdat->lru_lock); +del_page_from_lru_list (page, lruvec,page_lru (page));spin_unlock_irq (&pgdat->lru_lock); + ret =0 ; } +return ret; }
@@ -4293,6 +4288,10 @@ void check_move_unevictable_pages(struct pagevec *pvec) nr_pages =thp_nr_pages (page); pgscanned += nr_pages; +/* block memcg migration during page moving between lru */ +if (!TestClearPageLRU (page)) +continue ; +if (pagepgdat != pgdat) {if (pgdat)spin_unlock_irq (&pgdat->lru_lock);
@@ -4301,10 +4300,7 @@ void check_move_unevictable_pages(struct pagevec *pvec) } lruvec =mem_cgroup_page_lruvec (page, pgdat); -if (!PageLRU (page) || !PageUnevictable (page)) -continue ; - -if (page_evictable (page)) { +if (page_evictable (page) &&PageUnevictable (page)) {enum lru_list lru =page_lru_base_type (page);VM_BUG_ON_PAGE (PageActive (page), page);
@@ -4313,12 +4309,15 @@ void check_move_unevictable_pages(struct pagevec *pvec)add_page_to_lru_list (page, lruvec, lru); pgrescued += nr_pages; } +SetPageLRU (page); }if (pgdat) {__count_vm_events (UNEVICTABLE_PGRESCUED , pgrescued);__count_vm_events (UNEVICTABLE_PGSCANNED , pgscanned);spin_unlock_irq (&pgdat->lru_lock); + }else if (pgscanned) { +count_vm_events (UNEVICTABLE_PGSCANNED , pgscanned); } }EXPORT_SYMBOL_GPL (check_move_unevictable_pages);
Signature · 1 line
1.8.3.1
Re: [PATCH v21 14/19] mm/lru: introduce TestClearPageLRU
Quoted 13 lines · Alex Shi
Currently
lru_lockstill guards both lru list and page's lru bit, that's ok. but if we want to use specific lruvec lock on the page, we need to pin down the page's lruvec/memcg during locking. Just taking lruvec lock first may be undermined by the page's memcg charge/migration. To fix this problem, we will clear the lru bit out of locking and use it as pin down action to block the page isolation in memcg changing.So now a standard steps of page isolation is following:
1, get_page(); #pin the page avoid to be free 2, TestClearPageLRU(); #block other isolation like memcg change 3, spin_lock on lru_lock; #serialize lru list access 4, delete page from lru list;
Quoted 5 lines
This patch start with the first part: TestClearPageLRU, which combines PageLRU check and ClearPageLRU into a macro func TestClearPageLRU. This function will be used as page isolation precondition to prevent other isolations some where else. Then there are may !PageLRU page on lru list, need to remove
BUG()checking accordingly.
As there now may be !PageLRU pages on lru list, we need to ...
Quoted 7 lines
There 2 rules for lru bit now:
1, the lru bit still indicate if a page on lru list, just in some
temporary moment(isolating), the page may have no lru bit when it's on lru list. but the page still must be on lru list when the lru bit set.
2, have to remove lru bit before delete it from lru list.
2. we have to remove the lru bit before deleting page from lru list
Quoted 5 lines
As Andrew Morton mentioned this change would dirty cacheline for page isn't on LRU. But the lost would be acceptable in Rong Chen <[email protected]> report:
https://lore.kernel.org/lkml/20200304090301.GB5972@shao2-debian/
AFAIK these places generally expect PageLRU to be true, and if it's false, it's because of a race, so that effect should be negligible?
Quoted 13 lines
Suggested-by Johannes Weiner <[email protected]>Signed-off-by Alex Shi <[email protected]>Acked-by Hugh Dickins <[email protected]>Acked-by Johannes Weiner <[email protected]>Cc Hugh Dickins <[email protected]>Cc Johannes Weiner <[email protected]>Cc Michal Hocko <[email protected]>Cc Vladimir Davydov <[email protected]>Cc Andrew Morton <[email protected]>---
...
Quoted 12 lines
mm/vmscan.c +1
-1@@ -1542,7 +1542,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,*/ int __isolate_lru_page (struct page *page,isolate_mode_t mode) { -int ret = -EINVAL ; +int ret = -EBUSY ;/* Only take pages on the LRU. */ if (!PageLRU (page))
mm/vmscan.c int __isolate_lru_page(struct page *page, isolate_mode_t mode) · line 1555
Quoted 4 lines
mm/vmscan.c +0
-1@@ -1552,8 +1552,6 @@ int __isolate_lru_page(struct page *page, isolate_mode_t mode)if (PageUnevictable (page) && !(mode &ISOLATE_UNEVICTABLE ))return ret; - ret = -EBUSY ;
I'm not sure why this change is here, looks unrelated to the patch?
Oh I see, you want to prevent the BUG() in isolate_lru_pages().
But due to that, the PageUnevictable check was also affected unintentionally. But I don't think it's that important to BUG() when we run into PageUnevictable unexpectedly, so that's probably ok.
But with that, we can just make __isolate_lru_page() a bool function and remove the ugly switch in isolate_lru_pages()?
Quoted 66 lines
mm/vmscan.c +14
-14-/* * To minimise LRU disruption, the caller can indicate that it only * wants to isolate pages it will be able to operate on without @@ -1600,8 +1598,10 @@ int __isolate_lru_page(struct page *page, isolate_mode_t mode)* sure the page is not being freed elsewhere -- the * page release code relies on it. */ -ClearPageLRU (page); - ret =0 ; +if (TestClearPageLRU (page)) + ret =0 ; +else +put_page (page); }return ret;@@ -1667,8 +1667,6 @@ static unsigned long isolate_lru_pages(unsigned long nr_to_scan, page =lru_to_page (src);prefetchw_prev_lru_page (page, src, flags); -VM_BUG_ON_PAGE (!PageLRU (page), page); - nr_pages =compound_nr (page); total_scan += nr_pages;@@ -1765,21 +1763,18 @@ int isolate_lru_page(struct page *page)VM_BUG_ON_PAGE (!page_count (page), page);WARN_RATELIMIT (PageTail (page),"trying to isolate tail page" ); -if (PageLRU (page)) { +if (TestClearPageLRU (page)) {pg_data_t *pgdat =page_pgdat (page);struct lruvec *lruvec; -spin_lock_irq (&pgdat->lru_lock); +get_page (page); lruvec =mem_cgroup_page_lruvec (page, pgdat); -if (PageLRU (page)) { -int lru =page_lru (page); -get_page (page); -ClearPageLRU (page); -del_page_from_lru_list (page, lruvec, lru); - ret =0 ; - } +spin_lock_irq (&pgdat->lru_lock); +del_page_from_lru_list (page, lruvec,page_lru (page));spin_unlock_irq (&pgdat->lru_lock); + ret =0 ; } +return ret; }@@ -4293,6 +4288,10 @@ void check_move_unevictable_pages(struct pagevec *pvec) nr_pages =thp_nr_pages (page); pgscanned += nr_pages; +/* block memcg migration during page moving between lru */ +if (!TestClearPageLRU (page)) +continue ; +if (pagepgdat != pgdat) {if (pgdat)spin_unlock_irq (&pgdat->lru_lock);
mm/vmscan.c void check_move_unevictable_pages(struct pagevec *pvec) · line 4303
Quoted 8 lines
mm/vmscan.c +1
-4@@ -4301,10 +4300,7 @@ void check_move_unevictable_pages(struct pagevec *pvec) } lruvec =mem_cgroup_page_lruvec (page, pgdat); -if (!PageLRU (page) || !PageUnevictable (page)) -continue ; - -if (page_evictable (page)) { +if (page_evictable (page) &&PageUnevictable (page)) {
Doing PageUnevictable() test first should be cheaper?
Quoted 20 lines
mm/vmscan.c +3
-0enum lru_list lru =page_lru_base_type (page);VM_BUG_ON_PAGE (PageActive (page), page);@@ -4313,12 +4309,15 @@ void check_move_unevictable_pages(struct pagevec *pvec)add_page_to_lru_list (page, lruvec, lru); pgrescued += nr_pages; } +SetPageLRU (page); }if (pgdat) {__count_vm_events (UNEVICTABLE_PGRESCUED , pgrescued);__count_vm_events (UNEVICTABLE_PGSCANNED , pgscanned);spin_unlock_irq (&pgdat->lru_lock); + }else if (pgscanned) { +count_vm_events (UNEVICTABLE_PGSCANNED , pgscanned); } }EXPORT_SYMBOL_GPL (check_move_unevictable_pages);
Re: [PATCH v21 14/19] mm/lru: introduce TestClearPageLRU
Quoted 21 lines · Vlastimil Babka
Quoted 12 lines · Alex Shi
mm/vmscan.c +1
-1@@ -1542,7 +1542,7 @@ unsigned int reclaim_clean_pages_from_list(structzone *zone, */ int __isolate_lru_page (struct page *page,isolate_mode_t mode) { -int ret = -EINVAL ; +int ret = -EBUSY ;/* Only take pages on the LRU. */ if (!PageLRU (page))mm/vmscan.c int __isolate_lru_page(struct page *page, · line 1555
Quoted 4 lines
mm/vmscan.c +0
-0@@ -1552,8 +1552,6 @@ int __isolate_lru_page(struct page *page, isolate_mode_t mode)if (PageUnevictable (page) && !(mode &ISOLATE_UNEVICTABLE ))return ret; - ret = -EBUSY ;I'm not sure why this change is here, looks unrelated to the patch?
Quoted 1 line
Oh I see, you want to prevent the
BUG()inisolate_lru_pages().
Yes, I suggested this part of the patch to Alex, when I hit that BUG().
Quoted 4 lines
But due to that, the PageUnevictable check was also affected unintentionally. But I don't think it's that important to
BUG()when we run into PageUnevictable unexpectedly, so that's probably ok.
Not unintentional. __isolate_lru_page(), or __isolate_lru_page_prepare(), is a silly function, used by two callers whose requirements are almost entirely disjoint. The ISOLATE_UNEVICTABLE case is only for compaction.c, which takes no interest in -EINVAL versus -EBUSY, and has no such BUG().
I think it dates back to lumpy reclaim days, and it probably made more sense back then.
Quoted 3 lines
But with that, we can just make
__isolate_lru_page()a bool function and remove the ugly switch inisolate_lru_pages()?
I agree that the switch statement in isolate_lru_pages() seems pointless now, and can be turned into an if{}else{}. But that cleanup is a diversion from this particular TestClearPageLRU patch, and I think from the whole series (checking final state of the patchset, yes, the switch is still there - though I think there have been variant series which removed it).
Can we please leave that cleanup until after the series has gone in?
I think several of us have cleanups or optimization that we want to follow (I had one that inlines what isolate_migratepages_block() wanted of __isolate_lru_page() into that function, so simplifying what vmscan.c needs; perhaps that can now eliminate it completely, I've not tried recently). But there was a point at which the series was growing ten patches per release as we all added our bits and pieces on top, it got harder and harder to review the whole, and further from getting the basics in: I do push back against that tendency.
Hugh
Vlastimil Babka
Acked-byRe: [PATCH v21 14/19] mm/lru: introduce TestClearPageLRU
Quoted 31 lines · Hugh Dickins
Quoted 21 lines · Vlastimil Babka
Quoted 12 lines · Alex Shi
mm/vmscan.c +1
-1@@ -1542,7 +1542,7 @@ unsigned int reclaim_clean_pages_from_list(structzone *zone, */ int __isolate_lru_page (struct page *page,isolate_mode_t mode) { -int ret = -EINVAL ; +int ret = -EBUSY ;/* Only take pages on the LRU. */ if (!PageLRU (page))mm/vmscan.c int __isolate_lru_page(struct page *page, · line 1555
Quoted 4 lines
mm/vmscan.c +0
-0@@ -1552,8 +1552,6 @@ int __isolate_lru_page(struct page *page, isolate_mode_t mode)if (PageUnevictable (page) && !(mode &ISOLATE_UNEVICTABLE ))return ret; - ret = -EBUSY ;I'm not sure why this change is here, looks unrelated to the patch?
Quoted 1 line
Oh I see, you want to prevent the
BUG()inisolate_lru_pages().Yes, I suggested this part of the patch to Alex, when I hit that
BUG().Quoted 4 lines
But due to that, the PageUnevictable check was also affected unintentionally. But I don't think it's that important to
BUG()when we run into PageUnevictable unexpectedly, so that's probably ok.
Quoted 7 lines
Not unintentional.
__isolate_lru_page(), or__isolate_lru_page_prepare(), is a silly function, used by two callers whose requirements are almost entirely disjoint. TheISOLATE_UNEVICTABLEcase is only forcompaction.c, which takes no interest in-EINVALversus-EBUSY, and has no suchBUG().I think it dates back to lumpy reclaim days, and it probably made more sense back then.
Ah, thanks for explaining.
Quoted 4 lines
Quoted 3 lines
But with that, we can just make
__isolate_lru_page()a bool function and remove the ugly switch inisolate_lru_pages()?
Quoted 8 lines
I agree that the switch statement in
isolate_lru_pages()seems pointless now, and can be turned into an if{}else{}. But that cleanup is a diversion from this particular TestClearPageLRU patch, and I think from the whole series (checking final state of the patchset, yes, the switch is still there - though I think there have been variant series which removed it).Can we please leave that cleanup until after the series has gone in?
Sure thing!
The patch seems functionally fine, so
Quoted 11 lines
I think several of us have cleanups or optimization that we want to follow (I had one that inlines what
isolate_migratepages_block()wanted of__isolate_lru_page()into that function, so simplifying whatvmscan.cneeds; perhaps that can now eliminate it completely, I've not tried recently). But there was a point at which the series was growing ten patches per release as we all added our bits and pieces on top, it got harder and harder to review the whole, and further from getting the basics in: I do push back against that tendency.Hugh
PATCH v21 15/19 mm/compaction: do page isolation first in compaction
In reply to Alex ShiCurrently, compaction would get the lru_lock and then do page isolation which works fine with pgdat->lru_lock, since any page isoltion would compete for the lru_lock. If we want to change to memcg lru_lock, we have to isolate the page before getting lru_lock, thus isoltion would block page's memcg change which relay on page isoltion too. Then we could safely use per memcg lru_lock later.
The new page isolation use previous introduced TestClearPageLRU() + pgdat lru locking which will be changed to memcg lru lock later.
Hugh Dickins <[email protected]> fixed following bugs in this patch's early version:
Fix lots of crashes under compaction load: isolate_migratepages_block() must clean up appropriately when rejecting a page, setting PageLRU again if it had been cleared; and a put_page() after get_page_unless_zero() cannot safely be done while holding locked_lruvec - it may turn out to be the final put_page(), which will take an lruvec lock when PageLRU. And move __isolate_lru_page_prepare back after get_page_unless_zero to make trylock_page() safe:trylock_page() is not safe to use at this time: its setting PG_locked can race with the page being freed or allocated ("Bad page"), and can also erase flags being set by one of those "sole owners" of a freshly allocated page who use non-atomic __SetPageFlag().
--- include/linux/swap.h | 2 +- mm/compaction.c | 42 +++++++++++++++++++++++++++++++++--------- mm/vmscan.c | 43 ++++++++++++++++++++++--------------------- 3 files changed, 56 insertions(+), 31 deletions(-)
include/linux/swap.h +1 -1
@@ -356,7 +356,7 @@ extern void lru_cache_add_inactive_or_unevictable(struct page *page,extern unsigned long zone_reclaimable_pages (struct zone *zone);extern unsigned long try_to_free_pages (struct zonelist *zonelist,int order,gfp_t gfp_mask,nodemask_t *mask); -extern int __isolate_lru_page (struct page *page,isolate_mode_t mode); +extern int __isolate_lru_page_prepare (struct page *page,isolate_mode_t mode); extern unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, unsigned long nr_pages, gfp_t gfp_mask,
mm/compaction.c +33 -9
86 lines not on this page · raw message
mm/vmscan.c +22 -21
71 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Re: [PATCH v21 15/19] mm/compaction: do page isolation first in compaction
Quoted 25 lines · Alex Shi
Currently, compaction would get the
lru_lockand then do page isolation which works fine withpgdat->lru_lock, since any page isoltion would compete for thelru_lock. If we want to change to memcglru_lock, we have to isolate the page before gettinglru_lock, thus isoltion would block page's memcg change which relay on page isoltion too. Then we could safely use per memcglru_locklater.The new page isolation use previous introduced
TestClearPageLRU()+ pgdat lru locking which will be changed to memcg lru lock later.Hugh Dickins <[email protected]> fixed following bugs in this patch's early version:
Fix lots of crashes under compaction load:
isolate_migratepages_block()must clean up appropriately when rejecting a page, setting PageLRU again if it had been cleared; and aput_page()afterget_page_unless_zero()cannot safely be done while holdinglocked_lruvec- it may turn out to be the finalput_page(), which will take an lruvec lock when PageLRU. And move__isolate_lru_page_prepareback afterget_page_unless_zeroto maketrylock_page()safe:trylock_page()is not safe to use at this time: its settingPG_lockedcan race with the page being freed or allocated ("Bad page"), and can also erase flags being set by one of those "sole owners" of a freshly allocated page who use non-atomic__SetPageFlag().
Quoted 8 lines
Suggested-by Johannes Weiner <[email protected]>Acked-by Hugh Dickins <[email protected]>Acked-by Johannes Weiner <[email protected]>Cc Andrew Morton <akpm-de/[email protected]>Cc Matthew Wilcox <[email protected]>
A question below:
Quoted 12 lines
mm/compaction.c +0
-4@@ -979,10 +995,6 @@ static bool too_many_isolated(pg_data_t *pgdat) goto isolate_abort; } -/* Recheck PageLRU and PageCompound under lock */ -if (!PageLRU (page)) - goto isolate_fail; -/* * Page become compound since the non-locked check, * and it's on LRU. It can only be a THP so the order
mm/compaction.c static bool too_many_isolated(pg_data_t *pgdat) · line 1007
Quoted 7 lines
mm/compaction.c +2
-1@@ -990,16 +1002,13 @@ static bool too_many_isolated(pg_data_t *pgdat)*/ if (unlikely (PageCompound (page) && !cc->alloc_contig)) { low_pfn +=compound_nr (page) -1 ; - goto isolate_fail; +SetPageLRU (page); + goto isolate_fail_put; }
IIUC the danger here is khugepaged will collapse a THP. For that, __collapse_huge_page_isolate() has to succeed isolate_lru_page(). Under the new scheme, it shouldn't be possible, right? If that's correct, we can remove this part?
Re: [PATCH v21 15/19] mm/compaction: do page isolation first in compaction
Quoted 13 lines · Vlastimil Babka
mm/compaction.c static bool too_many_isolated(pg_data_t *pgdat)
Quoted 11 lines · Alex Shi
mm/compaction.c +0
-3@@ -979,10 +995,6 @@ static bool too_many_isolated(pg_data_t *pgdat) goto isolate_abort; } - /* Recheck PageLRU and PageCompound under lock */ - if (!PageLRU(page)) - goto isolate_fail; -/* * Page become compound since the non-locked check, * and it's on LRU. It can only be a THP so the order @@ -990,16 +1002,13 @@ static bool too_many_isolated(pg_data_t *pgdat)
Completely off-topic, and won't matter at all when Andrew rediffs into
mmotm: but isn't it weird that this is showing "too_many_isolated(", when actually the function is isolate_migratepages_block()?
Quoted 5 lines
Quoted 5 lines
mm/compaction.c +0
-1*/ if (unlikely (PageCompound (page) && !cc->alloc_contig)) { low_pfn +=compound_nr (page) -1 ; - goto isolate_fail;
Quoted 8 lines
mm/compaction.c static bool too_many_isolated(pg_data_t *pgdat) · line 1007
Quoted 3 lines
mm/compaction.c +2
-0+SetPageLRU (page); + goto isolate_fail_put; }IIUC the danger here is khugepaged will collapse a THP. For that,
__collapse_huge_page_isolate()has to succeedisolate_lru_page(). Under the new scheme, it shouldn't be possible, right? If that's correct, we can remove this part?
I don't think so. A preliminary check for PageCompound was made much higher up, before taking a reference on the page, but it can easily have become PageCompound since then (when racing prep_new_page() calls prep_compound_page()).
And __collapse_huge_page_isolate() does not turn a non-compound page into a compound page: it isolates small pages before copying them into the compound page (in the usual case: I can see there's also allowance for PageCompound there too, which will do something different).
Hugh
Re: [PATCH v21 15/19] mm/compaction: do page isolation first in compaction
ÔÚ 2020/11/12 ÉÏÎç10:28, Hugh Dickins дµÀ:
Quoted 7 lines
Quoted 3 lines
Review static bool too_many_isolated(pg_data_t *pgdat)
Quoted 3 lines
* Page become compound since the non-locked check,
* and it's on LRU. It can only be a THP so the order(quoted diff) +0
-0@@ -990,16 +1002,13 @@ static bool too_many_isolated(pg_data_t *pgdat)Completely off-topic, and won't matter at all when Andrew rediffs into
mmotm: but isn't it weird that this is showing "too_many_isolated(", when actually the function isisolate_migratepages_block()?
My git version is too low for this. Thanks for reminder. the latest git work fine on correct function name.
Thanks
Re: [PATCH v21 15/19] mm/compaction: do page isolation first in compaction
In reply to Hugh DickinsQuoted 38 lines · Hugh Dickins
Quoted 13 lines · Vlastimil Babka
mm/compaction.c static bool too_many_isolated(pg_data_t *pgdat)
Quoted 11 lines · Alex Shi
mm/compaction.c +0
-3@@ -979,10 +995,6 @@ static bool too_many_isolated(pg_data_t *pgdat) goto isolate_abort; } - /* Recheck PageLRU and PageCompound under lock */ - if (!PageLRU(page)) - goto isolate_fail; -/* * Page become compound since the non-locked check, * and it's on LRU. It can only be a THP so the order @@ -990,16 +1002,13 @@ static bool too_many_isolated(pg_data_t *pgdat)Completely off-topic, and won't matter at all when Andrew rediffs into
mmotm: but isn't it weird that this is showing "too_many_isolated(", when actually the function isisolate_migratepages_block()?Quoted 5 lines
Quoted 5 lines
mm/compaction.c +0
-1*/ if (unlikely (PageCompound (page) && !cc->alloc_contig)) { low_pfn +=compound_nr (page) -1 ; - goto isolate_fail;Quoted 8 lines
mm/compaction.c static bool too_many_isolated(pg_data_t *pgdat) · line 1007
Quoted 3 lines
mm/compaction.c +2
-0+SetPageLRU (page); + goto isolate_fail_put; }IIUC the danger here is khugepaged will collapse a THP. For that,
__collapse_huge_page_isolate()has to succeedisolate_lru_page(). Under the new scheme, it shouldn't be possible, right? If that's correct, we can remove this part?I don't think so. A preliminary check for PageCompound was made much higher up, before taking a reference on the page, but it can easily have become PageCompound since then (when racing
prep_new_page()callsprep_compound_page()).
Quoted 4 lines
And
__collapse_huge_page_isolate()does not turn a non-compound page into a compound page: it isolates small pages before copying them into the compound page (in the usual case: I can see there's also allowance for PageCompound there too, which will do something different).
Right, on both points, got too confused.
Quoted 2 lines
Hugh
PATCH v21 16/19 mm/swap.c: serialize memcg changes in pagevec_lru_move_fn
In reply to Alex ShiHugh Dickins' found a memcg change bug on original version:
If we want to change the pgdat->lru_lock to memcg's lruvec lock, we have to serialize mem_cgroup_move_account during pagevec_lru_move_fn. The possible bad scenario would like:
cpu 0 cpu 1 lruvec = mem_cgroup_page_lruvec() if (!isolate_lru_page()) mem_cgroup_move_account
spin_lock_irqsave(&lruvec->lru_lock <== wrong lock.
So we need TestClearPageLRU to block isolate_lru_page(), that serializes the memcg change. and then removing the PageLRU check in move_fn callee as the consequence.
__pagevec_lru_add_fn() is different from the others, because the pages it deals with are, by definition, not yet on the lru. TestClearPageLRU is not needed and would not work, so __pagevec_lru_add() goes its own way.
--- mm/swap.c | 44 +++++++++++++++++++++++++++++++++++--------- 1 file changed, 35 insertions(+), 9 deletions(-)
mm/swap.c +35 -9
@@ -222,8 +222,14 @@ static void pagevec_lru_move_fn(struct pagevec *pvec,spin_lock_irqsave (&pgdat->lru_lock, flags); } +/* block memcg migration during page moving between lru */ +if (!TestClearPageLRU (page)) +continue ; + lruvec =mem_cgroup_page_lruvec (page, pgdat); (*move_fn)(page, lruvec); + +SetPageLRU (page); }if (pgdat)spin_unlock_irqrestore (&pgdat->lru_lock, flags);
@@ -233,7 +239,7 @@ static void pagevec_lru_move_fn(struct pagevec *pvec,static void pagevec_move_tail_fn (struct page *page,struct lruvec *lruvec) { -if (PageLRU (page) && !PageUnevictable (page)) { +if (!PageUnevictable (page)) {del_page_from_lru_list (page, lruvec,page_lru (page));ClearPageActive (page);add_page_to_lru_list_tail (page, lruvec,page_lru (page));
@@ -306,7 +312,7 @@ void lru_note_cost_page(struct page *page)static void __activate_page (struct page *page,struct lruvec *lruvec) { -if (PageLRU (page) && !PageActive (page) && !PageUnevictable (page)) { +if (!PageActive (page) && !PageUnevictable (page)) {int lru =page_lru_base_type (page);int nr_pages =thp_nr_pages (page);
@@ -362,7 +368,8 @@ static void activate_page(struct page *page) page =compound_head (page);spin_lock_irq (&pgdat->lru_lock); -__activate_page (page,mem_cgroup_page_lruvec (page, pgdat)); +if (PageLRU (page)) +__activate_page (page,mem_cgroup_page_lruvec (page, pgdat)); spin_unlock_irq(&pgdat->lru_lock); }#endif
@@ -519,9 +526,6 @@ static void lru_deactivate_file_fn(struct page *page, struct lruvec *lruvec)bool active;int nr_pages =thp_nr_pages (page); -if (!PageLRU (page)) -return ; -if (PageUnevictable (page))return ;
@@ -562,7 +566,7 @@ static void lru_deactivate_file_fn(struct page *page, struct lruvec *lruvec)static void lru_deactivate_fn (struct page *page,struct lruvec *lruvec) { -if (PageLRU (page) &&PageActive (page) && !PageUnevictable (page)) { +if (PageActive (page) && !PageUnevictable (page)) {int lru =page_lru_base_type (page);int nr_pages =thp_nr_pages (page);
@@ -579,7 +583,7 @@ static void lru_deactivate_fn(struct page *page, struct lruvec *lruvec)static void lru_lazyfree_fn (struct page *page,struct lruvec *lruvec) { -if (PageLRU (page) &&PageAnon (page) &&PageSwapBacked (page) && +if (PageAnon (page) &&PageSwapBacked (page) && !PageSwapCache (page) && !PageUnevictable (page)) {bool active =PageActive (page);int nr_pages =thp_nr_pages (page);
@@ -1021,7 +1025,29 @@ static void __pagevec_lru_add_fn(struct page *page, struct lruvec *lruvec)*/ void __pagevec_lru_add (struct pagevec *pvec) { -pagevec_lru_move_fn (pvec, __pagevec_lru_add_fn); +int i; +struct pglist_data *pgdat =NULL ; +struct lruvec *lruvec; +unsigned long flags =0 ; + +for (i =0 ; i <pagevec_count (pvec); i++) { +struct page *page = pvec->pages[i]; +struct pglist_data *pagepgdat =page_pgdat (page); + +if (pagepgdat != pgdat) { +if (pgdat) +spin_unlock_irqrestore (&pgdat->lru_lock, flags); + pgdat = pagepgdat; +spin_lock_irqsave (&pgdat->lru_lock, flags); + } + + lruvec =mem_cgroup_page_lruvec (page, pgdat); +__pagevec_lru_add_fn (page, lruvec); + } +if (pgdat) +spin_unlock_irqrestore (&pgdat->lru_lock, flags); +release_pages (pvec->pages, pvec->nr); +pagevec_reinit (pvec); }/**
Signature · 1 line
1.8.3.1
Re: [PATCH v21 16/19] mm/swap.c: serialize memcg changes in pagevec_lru_move_fn
Quoted 21 lines · Alex Shi
Hugh Dickins' found a memcg change bug on original version:
If we want to change thepgdat->lru_lockto memcg's lruvec lock, we have to serializemem_cgroup_move_accountduringpagevec_lru_move_fn. The possible bad scenario would like:cpu 0 cpu 1 lruvec = mem_cgroup_page_lruvec() if (!isolate_lru_page()) mem_cgroup_move_accountspin_lock_irqsave(&lruvec->lru_lock <== wrong lock.
So we need TestClearPageLRU to block
isolate_lru_page(), that serializes the memcg change. and then removing the PageLRU check inmove_fncallee as the consequence.
__pagevec_lru_add_fn()is different from the others, because the pages it deals with are, by definition, not yet on the lru. TestClearPageLRU is not needed and would not work, so__pagevec_lru_add()goes its own way.
Quoted 7 lines
Reported-by Hugh Dickins <[email protected]>Acked-by Hugh Dickins <[email protected]>Acked-by Johannes Weiner <[email protected]>Cc Andrew Morton <akpm-de/[email protected]>
Quoted 111 lines
--- mm/swap.c | 44 +++++++++++++++++++++++++++++++++++--------- 1 file changed, 35 insertions(+), 9 deletions(-)mm/swap.c +35
-9@@ -222,8 +222,14 @@ static void pagevec_lru_move_fn(struct pagevec *pvec,spin_lock_irqsave (&pgdat->lru_lock, flags); } +/* block memcg migration during page moving between lru */ +if (!TestClearPageLRU (page)) +continue ; + lruvec =mem_cgroup_page_lruvec (page, pgdat); (*move_fn)(page, lruvec); + +SetPageLRU (page); }if (pgdat)spin_unlock_irqrestore (&pgdat->lru_lock, flags);@@ -233,7 +239,7 @@ static void pagevec_lru_move_fn(struct pagevec *pvec,static void pagevec_move_tail_fn (struct page *page,struct lruvec *lruvec) { -if (PageLRU (page) && !PageUnevictable (page)) { +if (!PageUnevictable (page)) {del_page_from_lru_list (page, lruvec,page_lru (page));ClearPageActive (page);add_page_to_lru_list_tail (page, lruvec,page_lru (page));@@ -306,7 +312,7 @@ void lru_note_cost_page(struct page *page)static void __activate_page (struct page *page,struct lruvec *lruvec) { -if (PageLRU (page) && !PageActive (page) && !PageUnevictable (page)) { +if (!PageActive (page) && !PageUnevictable (page)) {int lru =page_lru_base_type (page);int nr_pages =thp_nr_pages (page);@@ -362,7 +368,8 @@ static void activate_page(struct page *page) page =compound_head (page);spin_lock_irq (&pgdat->lru_lock); -__activate_page (page,mem_cgroup_page_lruvec (page, pgdat)); +if (PageLRU (page)) +__activate_page (page,mem_cgroup_page_lruvec (page, pgdat)); spin_unlock_irq(&pgdat->lru_lock); }#endif @@ -519,9 +526,6 @@ static void lru_deactivate_file_fn(struct page *page, struct lruvec *lruvec)bool active;int nr_pages =thp_nr_pages (page); -if (!PageLRU (page)) -return ; -if (PageUnevictable (page))return ;@@ -562,7 +566,7 @@ static void lru_deactivate_file_fn(struct page *page, struct lruvec *lruvec)static void lru_deactivate_fn (struct page *page,struct lruvec *lruvec) { -if (PageLRU (page) &&PageActive (page) && !PageUnevictable (page)) { +if (PageActive (page) && !PageUnevictable (page)) {int lru =page_lru_base_type (page);int nr_pages =thp_nr_pages (page);@@ -579,7 +583,7 @@ static void lru_deactivate_fn(struct page *page, struct lruvec *lruvec)static void lru_lazyfree_fn (struct page *page,struct lruvec *lruvec) { -if (PageLRU (page) &&PageAnon (page) &&PageSwapBacked (page) && +if (PageAnon (page) &&PageSwapBacked (page) && !PageSwapCache (page) && !PageUnevictable (page)) {bool active =PageActive (page);int nr_pages =thp_nr_pages (page);@@ -1021,7 +1025,29 @@ static void __pagevec_lru_add_fn(struct page *page, struct lruvec *lruvec)*/ void __pagevec_lru_add (struct pagevec *pvec) { -pagevec_lru_move_fn (pvec, __pagevec_lru_add_fn); +int i; +struct pglist_data *pgdat =NULL ; +struct lruvec *lruvec; +unsigned long flags =0 ; + +for (i =0 ; i <pagevec_count (pvec); i++) { +struct page *page = pvec->pages[i]; +struct pglist_data *pagepgdat =page_pgdat (page); + +if (pagepgdat != pgdat) { +if (pgdat) +spin_unlock_irqrestore (&pgdat->lru_lock, flags); + pgdat = pagepgdat; +spin_lock_irqsave (&pgdat->lru_lock, flags); + } + + lruvec =mem_cgroup_page_lruvec (page, pgdat); +__pagevec_lru_add_fn (page, lruvec); + } +if (pgdat) +spin_unlock_irqrestore (&pgdat->lru_lock, flags); +release_pages (pvec->pages, pvec->nr); +pagevec_reinit (pvec); }/**
PATCH v21 17/19 mm/lru: replace pgdat lru_lock with lruvec lock
In reply to Alex ShiThis patch moves per node lru_lock into lruvec, thus bring a lru_lock for each of memcg per node. So on a large machine, each of memcg don't have to suffer from per node pgdat->lru_lock competition. They could go fast with their self lru_lock.
After move memcg charge before lru inserting, page isolation could serialize page's memcg, then per memcg lruvec lock is stable and could replace per node lru lock.
In func isolate_migratepages_block, compact_unlock_should_abort and lock_page_lruvec_irqsave are open coded to work with compact_control. Also add a debug func in locking which may give some clues if there are sth out of hands.
Daniel Jordan's testing show 62% improvement on modified readtwice case on his 2P * 10 core * 2 HT broadwell box.
https://lore.kernel.org/lkml/20200915165807.kpp7uhiw7l3loofu-S51bK0XF4qpuJJETbFA3a0B3C2bhBk7L0E9HWUfgJXw@public.gmane.org/
On a large machine with memcg enabled but not used, the page's lruvec seeking pass a few pointers, that may lead to lru_lock holding time increase and a bit regression.
Hugh Dickins helped on the patch polish, thanks!
--- include/linux/memcontrol.h | 58 +++++++++++++++++++++++ include/linux/mmzone.h | 3 +- mm/compaction.c | 56 ++++++++++++++-------- mm/huge_memory.c | 11 ++--- mm/memcontrol.c | 73 ++++++++++++++++++++++++++-- mm/mlock.c | 22 ++++++--- mm/mmzone.c | 1 + mm/page_alloc.c | 1 - mm/swap.c | 116 ++++++++++++++++++++++----------------------- mm/vmscan.c | 55 ++++++++++----------- 10 files changed, 270 insertions(+), 126 deletions(-)
include/linux/memcontrol.h +58 -0
@@ -666,6 +666,19 @@ static inline struct lruvec *mem_cgroup_lruvec(struct mem_cgroup *memcg,struct mem_cgroup *get_mem_cgroup_from_page (struct page *page); +struct lruvec *lock_page_lruvec (struct page *page); +struct lruvec *lock_page_lruvec_irq (struct page *page); +struct lruvec *lock_page_lruvec_irqsave (struct page *page, +unsigned long *flags); + +#ifdef CONFIG_DEBUG_VM +void lruvec_memcg_debug (struct lruvec *lruvec,struct page *page); +#else +static inline void lruvec_memcg_debug (struct lruvec *lruvec,struct page *page) +{ +} +#endif +static inline struct mem_cgroup *mem_cgroup_from_css (struct cgroup_subsys_state *css){ return css ? container_of(css, struct mem_cgroup, css) : NULL;
@@ -1233,6 +1246,31 @@ static inline void mem_cgroup_put(struct mem_cgroup *memcg) { } +static inline struct lruvec *lock_page_lruvec (struct page *page) +{ +struct pglist_data *pgdat =page_pgdat (page); + +spin_lock (&pgdat->__lruvec.lru_lock); +return &pgdat->__lruvec; +} + +static inline struct lruvec *lock_page_lruvec_irq (struct page *page) +{ +struct pglist_data *pgdat =page_pgdat (page); + +spin_lock_irq (&pgdat->__lruvec.lru_lock); +return &pgdat->__lruvec; +} + +static inline struct lruvec *lock_page_lruvec_irqsave (struct page *page, +unsigned long *flagsp) +{ +struct pglist_data *pgdat =page_pgdat (page); + +spin_lock_irqsave (&pgdat->__lruvec.lru_lock, *flagsp); +return &pgdat->__lruvec; +} +static inline struct mem_cgroup * mem_cgroup_iter(struct mem_cgroup *root, struct mem_cgroup *prev,
@@ -1476,6 +1514,10 @@ static inline void count_memcg_page_event(struct page *page,void count_memcg_event_mm (struct mm_struct *mm,enum vm_event_item idx) { } + +static inline void lruvec_memcg_debug (struct lruvec *lruvec,struct page *page) +{ +}#endif /* CONFIG_MEMCG */ /* idx can be of type enum memcg_stat_item or node_stat_item */
@@ -1605,6 +1647,22 @@ static inline struct lruvec *parent_lruvec(struct lruvec *lruvec)return mem_cgroup_lruvec (memcg,lruvec_pgdat (lruvec)); } +static inline void unlock_page_lruvec (struct lruvec *lruvec) +{ +spin_unlock (&lruvec->lru_lock); +} + +static inline void unlock_page_lruvec_irq (struct lruvec *lruvec) +{ +spin_unlock_irq (&lruvec->lru_lock); +} + +static inline void unlock_page_lruvec_irqrestore (struct lruvec *lruvec, +unsigned long flags) +{ +spin_unlock_irqrestore (&lruvec->lru_lock, flags); +} +#ifdef CONFIG_CGROUP_WRITEBACK struct wb_domain *mem_cgroup_wb_domain (struct bdi_writeback *wb);
include/linux/mmzone.h +2 -1
@@ -276,6 +276,8 @@ enum lruvec_flags {struct lruvec {struct list_head lists[NR_LRU_LISTS ]; +/* per lruvec lru_lock for memcg */ +spinlock_t lru_lock;/* * These track the cost of reclaiming one LRU - file or anon - * over the other. As the observed cost of reclaiming one LRU
@@ -796,7 +798,6 @@ struct deferred_split {/* Write-intensive fields used by page reclaim */ ZONE_PADDING (_pad1_) - spinlock_t lru_lock; #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT/*
mm/compaction.c +36 -20
105 lines not on this page · raw message
mm/huge_memory.c +4 -7
@@ -2354,7 +2354,7 @@ static void lru_add_page_tail(struct page *head, struct page *tail,VM_BUG_ON_PAGE (!PageHead (head), head);VM_BUG_ON_PAGE (PageCompound (tail), head);VM_BUG_ON_PAGE (PageLRU (tail), head); -lockdep_assert_held (&lruvec_pgdat (lruvec)->lru_lock); +lockdep_assert_held (&lruvec->lru_lock);if (list) {/* page reclaim is reclaiming a huge page */
@@ -2438,7 +2438,6 @@ static void __split_huge_page(struct page *page, struct list_head *list, pgoff_t end) {struct page *head =compound_head (page); -pg_data_t *pgdat =page_pgdat (head);struct lruvec *lruvec;struct address_space *swap_cache =NULL ;unsigned long offset =0 ;
@@ -2456,10 +2455,8 @@ static void __split_huge_page(struct page *page, struct list_head *list,xa_lock (&swap_cache->i_pages); } -/* prevent PageLRU to go away from under us, and freeze lru stats */ -spin_lock (&pgdat->lru_lock); - - lruvec =mem_cgroup_page_lruvec (head, pgdat); +/* lock lru list/PageCompound, ref freezed by page_ref_freeze */ + lruvec =lock_page_lruvec (head);for (i = nr -1 ; i >=1 ; i--) {__split_huge_page_tail (head, i, lruvec, list);
@@ -2480,7 +2477,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, }ClearPageCompound (head); -spin_unlock (&pgdat->lru_lock); +unlock_page_lruvec (lruvec);/* Caller disabled irqs, so they are still disabled here */ split_page_owner (head, nr);
mm/memcontrol.c +70 -3
98 lines not on this page · raw message
mm/mlock.c +16 -6
@@ -262,12 +262,12 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone)int nr =pagevec_count (pvec);int delta_munlocked = -nr;struct pagevec pvec_putback; +struct lruvec *lruvec =NULL ;int pgrescued =0 ;pagevec_init (&pvec_putback);/* Phase 1: page isolation */ -spin_lock_irq (&zone->zone_pgdat->lru_lock);for (i =0 ; i < nr; i++) {struct page *page = pvec->pages[i];
@@ -277,10 +277,16 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone)* so we can spare the get_page() here. */ if (TestClearPageLRU (page)) { -struct lruvec *lruvec; +struct lruvec *new_lruvec; + + new_lruvec =mem_cgroup_page_lruvec (page, +page_pgdat (page)); +if (new_lruvec != lruvec) { +if (lruvec) +unlock_page_lruvec_irq (lruvec); + lruvec =lock_page_lruvec_irq (page); + } - lruvec =mem_cgroup_page_lruvec (page, -page_pgdat (page));del_page_from_lru_list (page, lruvec,page_lru (page));continue ;
@@ -299,8 +305,12 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone)pagevec_add (&pvec_putback, pvec->pages[i]); pvec->pages[i] =NULL ; } -__mod_zone_page_state (zone,NR_MLOCK , delta_munlocked); -spin_unlock_irq (&zone->zone_pgdat->lru_lock); +if (lruvec) { +__mod_zone_page_state (zone,NR_MLOCK , delta_munlocked); +unlock_page_lruvec_irq (lruvec); + }else if (delta_munlocked) { +mod_zone_page_state (zone,NR_MLOCK , delta_munlocked); + }/* Now we can release pins of pages that we are not munlocking */ pagevec_release (&pvec_putback);
mm/mmzone.c +1 -0
@@ -91,6 +91,7 @@ void lruvec_init(struct lruvec *lruvec)enum lru_list lru;memset (lruvec,0 ,sizeof (struct lruvec )); +spin_lock_init (&lruvec->lru_lock);for_each_lru (lru)INIT_LIST_HEAD (&lruvec->lists[lru]);
mm/page_alloc.c +0 -1
@@ -6808,7 +6808,6 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat)init_waitqueue_head (&pgdat->pfmemalloc_wait);pgdat_page_ext_init (pgdat); -spin_lock_init (&pgdat->lru_lock);lruvec_init (&pgdat->__lruvec); }
mm/swap.c +58 -58
225 lines not on this page · raw message
mm/vmscan.c +25 -30
169 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Re: [PATCH v21 17/19] mm/lru: replace pgdat lru_lock with lruvec lock
ÔÚ 2020/11/5 ÏÂÎç4:55, Alex Shi дµÀ:
Quoted 7 lines
/**
+ *lock_page_lruvec- lock and return lruvec for a given page.
+ * @page: the page
+ *
+ * This series functions should be used in either conditions:
+ * PageLRU is cleared or unset
+ * or page is locked.
or page->_refcount is zero.
Ops, here is a typo, we need to add back above line. so the patch updated here:
From 9f187d04c7ba62bb7e07c07733b2848f155961f6 Mon Sep 17 00:00:00 2001
From: Alex Shi <alex.shi-KPsoFbNs7GizrGE5bRqYAgC/[email protected]>
Date: Tue, 18 Aug 2020 16:44:21 +0800
Subject: [PATCH v21 17/19] mm/lru: replace pgdat lru_lock with lruvec lock
This patch moves per node lru_lock into lruvec, thus bring a lru_lock for each of memcg per node. So on a large machine, each of memcg don't have to suffer from per node pgdat->lru_lock competition. They could go fast with their self lru_lock.
After move memcg charge before lru inserting, page isolation could serialize page's memcg, then per memcg lruvec lock is stable and could replace per node lru lock.
In func isolate_migratepages_block, compact_unlock_should_abort and lock_page_lruvec_irqsave are open coded to work with compact_control. Also add a debug func in locking which may give some clues if there are sth out of hands.
Daniel Jordan's testing show 62% improvement on modified readtwice case on his 2P * 10 core * 2 HT broadwell box.
https://lore.kernel.org/lkml/20200915165807.kpp7uhiw7l3loofu-S51bK0XF4qpuJJETbFA3a0B3C2bhBk7L0E9HWUfgJXw@public.gmane.org/
On a large machine with memcg enabled but not used, the page's lruvec seeking pass a few pointers, that may lead to lru_lock holding time increase and a bit regression.
Hugh Dickins helped on the patch polish, thanks!
--- include/linux/memcontrol.h | 58 +++++++++++++++++++++++ include/linux/mmzone.h | 3 +- mm/compaction.c | 56 ++++++++++++++-------- mm/huge_memory.c | 11 ++--- mm/memcontrol.c | 74 +++++++++++++++++++++++++++-- mm/mlock.c | 22 ++++++--- mm/mmzone.c | 1 + mm/page_alloc.c | 1 - mm/swap.c | 116 ++++++++++++++++++++++----------------------- mm/vmscan.c | 55 ++++++++++----------- 10 files changed, 271 insertions(+), 126 deletions(-)
include/linux/memcontrol.h +58 -0
@@ -666,6 +666,19 @@ static inline struct lruvec *mem_cgroup_lruvec(struct mem_cgroup *memcg,struct mem_cgroup *get_mem_cgroup_from_page (struct page *page); +struct lruvec *lock_page_lruvec (struct page *page); +struct lruvec *lock_page_lruvec_irq (struct page *page); +struct lruvec *lock_page_lruvec_irqsave (struct page *page, +unsigned long *flags); + +#ifdef CONFIG_DEBUG_VM +void lruvec_memcg_debug (struct lruvec *lruvec,struct page *page); +#else +static inline void lruvec_memcg_debug (struct lruvec *lruvec,struct page *page) +{ +} +#endif +static inline struct mem_cgroup *mem_cgroup_from_css (struct cgroup_subsys_state *css){ return css ? container_of(css, struct mem_cgroup, css) : NULL;
@@ -1233,6 +1246,31 @@ static inline void mem_cgroup_put(struct mem_cgroup *memcg) { } +static inline struct lruvec *lock_page_lruvec (struct page *page) +{ +struct pglist_data *pgdat =page_pgdat (page); + +spin_lock (&pgdat->__lruvec.lru_lock); +return &pgdat->__lruvec; +} + +static inline struct lruvec *lock_page_lruvec_irq (struct page *page) +{ +struct pglist_data *pgdat =page_pgdat (page); + +spin_lock_irq (&pgdat->__lruvec.lru_lock); +return &pgdat->__lruvec; +} + +static inline struct lruvec *lock_page_lruvec_irqsave (struct page *page, +unsigned long *flagsp) +{ +struct pglist_data *pgdat =page_pgdat (page); + +spin_lock_irqsave (&pgdat->__lruvec.lru_lock, *flagsp); +return &pgdat->__lruvec; +} +static inline struct mem_cgroup * mem_cgroup_iter(struct mem_cgroup *root, struct mem_cgroup *prev,
@@ -1476,6 +1514,10 @@ static inline void count_memcg_page_event(struct page *page,void count_memcg_event_mm (struct mm_struct *mm,enum vm_event_item idx) { } + +static inline void lruvec_memcg_debug (struct lruvec *lruvec,struct page *page) +{ +}#endif /* CONFIG_MEMCG */ /* idx can be of type enum memcg_stat_item or node_stat_item */
@@ -1605,6 +1647,22 @@ static inline struct lruvec *parent_lruvec(struct lruvec *lruvec)return mem_cgroup_lruvec (memcg,lruvec_pgdat (lruvec)); } +static inline void unlock_page_lruvec (struct lruvec *lruvec) +{ +spin_unlock (&lruvec->lru_lock); +} + +static inline void unlock_page_lruvec_irq (struct lruvec *lruvec) +{ +spin_unlock_irq (&lruvec->lru_lock); +} + +static inline void unlock_page_lruvec_irqrestore (struct lruvec *lruvec, +unsigned long flags) +{ +spin_unlock_irqrestore (&lruvec->lru_lock, flags); +} +#ifdef CONFIG_CGROUP_WRITEBACK struct wb_domain *mem_cgroup_wb_domain (struct bdi_writeback *wb);
include/linux/mmzone.h +2 -1
@@ -276,6 +276,8 @@ enum lruvec_flags {struct lruvec {struct list_head lists[NR_LRU_LISTS ]; +/* per lruvec lru_lock for memcg */ +spinlock_t lru_lock;/* * These track the cost of reclaiming one LRU - file or anon - * over the other. As the observed cost of reclaiming one LRU
@@ -796,7 +798,6 @@ struct deferred_split {/* Write-intensive fields used by page reclaim */ ZONE_PADDING (_pad1_) - spinlock_t lru_lock; #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT/*
mm/compaction.c +36 -20
105 lines not on this page · raw message
mm/huge_memory.c +4 -7
@@ -2354,7 +2354,7 @@ static void lru_add_page_tail(struct page *head, struct page *tail,VM_BUG_ON_PAGE (!PageHead (head), head);VM_BUG_ON_PAGE (PageCompound (tail), head);VM_BUG_ON_PAGE (PageLRU (tail), head); -lockdep_assert_held (&lruvec_pgdat (lruvec)->lru_lock); +lockdep_assert_held (&lruvec->lru_lock);if (list) {/* page reclaim is reclaiming a huge page */
@@ -2438,7 +2438,6 @@ static void __split_huge_page(struct page *page, struct list_head *list, pgoff_t end) {struct page *head =compound_head (page); -pg_data_t *pgdat =page_pgdat (head);struct lruvec *lruvec;struct address_space *swap_cache =NULL ;unsigned long offset =0 ;
@@ -2456,10 +2455,8 @@ static void __split_huge_page(struct page *page, struct list_head *list,xa_lock (&swap_cache->i_pages); } -/* prevent PageLRU to go away from under us, and freeze lru stats */ -spin_lock (&pgdat->lru_lock); - - lruvec =mem_cgroup_page_lruvec (head, pgdat); +/* lock lru list/PageCompound, ref freezed by page_ref_freeze */ + lruvec =lock_page_lruvec (head);for (i = nr -1 ; i >=1 ; i--) {__split_huge_page_tail (head, i, lruvec, list);
@@ -2480,7 +2477,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, }ClearPageCompound (head); -spin_unlock (&pgdat->lru_lock); +unlock_page_lruvec (lruvec);/* Caller disabled irqs, so they are still disabled here */ split_page_owner (head, nr);
mm/memcontrol.c +71 -3
99 lines not on this page · raw message
mm/mlock.c +16 -6
@@ -262,12 +262,12 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone)int nr =pagevec_count (pvec);int delta_munlocked = -nr;struct pagevec pvec_putback; +struct lruvec *lruvec =NULL ;int pgrescued =0 ;pagevec_init (&pvec_putback);/* Phase 1: page isolation */ -spin_lock_irq (&zone->zone_pgdat->lru_lock);for (i =0 ; i < nr; i++) {struct page *page = pvec->pages[i];
@@ -277,10 +277,16 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone)* so we can spare the get_page() here. */ if (TestClearPageLRU (page)) { -struct lruvec *lruvec; +struct lruvec *new_lruvec; + + new_lruvec =mem_cgroup_page_lruvec (page, +page_pgdat (page)); +if (new_lruvec != lruvec) { +if (lruvec) +unlock_page_lruvec_irq (lruvec); + lruvec =lock_page_lruvec_irq (page); + } - lruvec =mem_cgroup_page_lruvec (page, -page_pgdat (page));del_page_from_lru_list (page, lruvec,page_lru (page));continue ;
@@ -299,8 +305,12 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone)pagevec_add (&pvec_putback, pvec->pages[i]); pvec->pages[i] =NULL ; } -__mod_zone_page_state (zone,NR_MLOCK , delta_munlocked); -spin_unlock_irq (&zone->zone_pgdat->lru_lock); +if (lruvec) { +__mod_zone_page_state (zone,NR_MLOCK , delta_munlocked); +unlock_page_lruvec_irq (lruvec); + }else if (delta_munlocked) { +mod_zone_page_state (zone,NR_MLOCK , delta_munlocked); + }/* Now we can release pins of pages that we are not munlocking */ pagevec_release (&pvec_putback);
mm/mmzone.c +1 -0
@@ -91,6 +91,7 @@ void lruvec_init(struct lruvec *lruvec)enum lru_list lru;memset (lruvec,0 ,sizeof (struct lruvec )); +spin_lock_init (&lruvec->lru_lock);for_each_lru (lru)INIT_LIST_HEAD (&lruvec->lists[lru]);
mm/page_alloc.c +0 -1
@@ -6808,7 +6808,6 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat)init_waitqueue_head (&pgdat->pfmemalloc_wait);pgdat_page_ext_init (pgdat); -spin_lock_init (&pgdat->lru_lock);lruvec_init (&pgdat->__lruvec); }
mm/swap.c +58 -58
225 lines not on this page · raw message
mm/vmscan.c +25 -30
169 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Alex Shi
Acked-by Hugh DickinsRe: [PATCH v21 17/19] mm/lru: replace pgdat lru_lock with lruvec lock
Quoted 9 lines · Alex Shi
Quoted 7 lines · Alex Shi
/**
+ *lock_page_lruvec- lock and return lruvec for a given page.
+ * @page: the page
+ *
+ * This series functions should be used in either conditions:
+ * PageLRU is cleared or unset
+ * or page is locked.
Quoted 3 lines
or page->_refcount is zero.Ops, here is a typo, we need to add back above line. so the patch updated here:
Sorry for aother miss, linux-next removed the page.mem_cgroup, replace it to memcg_data and get by page_memcg(). Hence this and next patch needs update for this:
From 84e69f892119d99612e9668e3fe47a3922bafff1 Mon Sep 17 00:00:00 2001
From: Alex Shi <alex.shi-KPsoFbNs7GizrGE5bRqYAgC/[email protected]>
Date: Tue, 18 Aug 2020 16:44:21 +0800
Subject: [PATCH v21 17/19] mm/lru: replace pgdat lru_lock with lruvec lock
This patch moves per node lru_lock into lruvec, thus bring a lru_lock for each of memcg per node. So on a large machine, each of memcg don't have to suffer from per node pgdat->lru_lock competition. They could go fast with their self lru_lock.
After move memcg charge before lru inserting, page isolation could serialize page's memcg, then per memcg lruvec lock is stable and could replace per node lru lock.
In func isolate_migratepages_block, compact_unlock_should_abort and lock_page_lruvec_irqsave are open coded to work with compact_control. Also add a debug func in locking which may give some clues if there are sth out of hands.
Daniel Jordan's testing show 62% improvement on modified readtwice case on his 2P * 10 core * 2 HT broadwell box.
https://lore.kernel.org/lkml/20200915165807.kpp7uhiw7l3loofu-S51bK0XF4qpuJJETbFA3a0B3C2bhBk7L0E9HWUfgJXw@public.gmane.org/
On a large machine with memcg enabled but not used, the page's lruvec seeking pass a few pointers, that may lead to lru_lock holding time increase and a bit regression.
Hugh Dickins helped on the patch polish, thanks!
--- include/linux/memcontrol.h | 58 +++++++++++++++++++++++ include/linux/mmzone.h | 3 +- mm/compaction.c | 56 ++++++++++++++-------- mm/huge_memory.c | 11 ++--- mm/memcontrol.c | 78 ++++++++++++++++++++++++++++-- mm/mlock.c | 22 ++++++--- mm/mmzone.c | 1 + mm/page_alloc.c | 1 - mm/swap.c | 116 ++++++++++++++++++++++----------------------- mm/vmscan.c | 55 ++++++++++----------- 10 files changed, 275 insertions(+), 126 deletions(-)
include/linux/memcontrol.h +58 -0
@@ -666,6 +666,19 @@ static inline struct lruvec *mem_cgroup_lruvec(struct mem_cgroup *memcg,struct mem_cgroup *get_mem_cgroup_from_page (struct page *page); +struct lruvec *lock_page_lruvec (struct page *page); +struct lruvec *lock_page_lruvec_irq (struct page *page); +struct lruvec *lock_page_lruvec_irqsave (struct page *page, +unsigned long *flags); + +#ifdef CONFIG_DEBUG_VM +void lruvec_memcg_debug (struct lruvec *lruvec,struct page *page); +#else +static inline void lruvec_memcg_debug (struct lruvec *lruvec,struct page *page) +{ +} +#endif +static inline struct mem_cgroup *mem_cgroup_from_css (struct cgroup_subsys_state *css){ return css ? container_of(css, struct mem_cgroup, css) : NULL;
@@ -1233,6 +1246,31 @@ static inline void mem_cgroup_put(struct mem_cgroup *memcg) { } +static inline struct lruvec *lock_page_lruvec (struct page *page) +{ +struct pglist_data *pgdat =page_pgdat (page); + +spin_lock (&pgdat->__lruvec.lru_lock); +return &pgdat->__lruvec; +} + +static inline struct lruvec *lock_page_lruvec_irq (struct page *page) +{ +struct pglist_data *pgdat =page_pgdat (page); + +spin_lock_irq (&pgdat->__lruvec.lru_lock); +return &pgdat->__lruvec; +} + +static inline struct lruvec *lock_page_lruvec_irqsave (struct page *page, +unsigned long *flagsp) +{ +struct pglist_data *pgdat =page_pgdat (page); + +spin_lock_irqsave (&pgdat->__lruvec.lru_lock, *flagsp); +return &pgdat->__lruvec; +} +static inline struct mem_cgroup * mem_cgroup_iter(struct mem_cgroup *root, struct mem_cgroup *prev,
@@ -1476,6 +1514,10 @@ static inline void count_memcg_page_event(struct page *page,void count_memcg_event_mm (struct mm_struct *mm,enum vm_event_item idx) { } + +static inline void lruvec_memcg_debug (struct lruvec *lruvec,struct page *page) +{ +}#endif /* CONFIG_MEMCG */ /* idx can be of type enum memcg_stat_item or node_stat_item */
@@ -1605,6 +1647,22 @@ static inline struct lruvec *parent_lruvec(struct lruvec *lruvec)return mem_cgroup_lruvec (memcg,lruvec_pgdat (lruvec)); } +static inline void unlock_page_lruvec (struct lruvec *lruvec) +{ +spin_unlock (&lruvec->lru_lock); +} + +static inline void unlock_page_lruvec_irq (struct lruvec *lruvec) +{ +spin_unlock_irq (&lruvec->lru_lock); +} + +static inline void unlock_page_lruvec_irqrestore (struct lruvec *lruvec, +unsigned long flags) +{ +spin_unlock_irqrestore (&lruvec->lru_lock, flags); +} +#ifdef CONFIG_CGROUP_WRITEBACK struct wb_domain *mem_cgroup_wb_domain (struct bdi_writeback *wb);
include/linux/mmzone.h +2 -1
@@ -276,6 +276,8 @@ enum lruvec_flags {struct lruvec {struct list_head lists[NR_LRU_LISTS ]; +/* per lruvec lru_lock for memcg */ +spinlock_t lru_lock;/* * These track the cost of reclaiming one LRU - file or anon - * over the other. As the observed cost of reclaiming one LRU
@@ -796,7 +798,6 @@ struct deferred_split {/* Write-intensive fields used by page reclaim */ ZONE_PADDING (_pad1_) - spinlock_t lru_lock; #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT/*
mm/compaction.c +36 -20
105 lines not on this page · raw message
mm/huge_memory.c +4 -7
@@ -2354,7 +2354,7 @@ static void lru_add_page_tail(struct page *head, struct page *tail,VM_BUG_ON_PAGE (!PageHead (head), head);VM_BUG_ON_PAGE (PageCompound (tail), head);VM_BUG_ON_PAGE (PageLRU (tail), head); -lockdep_assert_held (&lruvec_pgdat (lruvec)->lru_lock); +lockdep_assert_held (&lruvec->lru_lock);if (list) {/* page reclaim is reclaiming a huge page */
@@ -2438,7 +2438,6 @@ static void __split_huge_page(struct page *page, struct list_head *list, pgoff_t end) {struct page *head =compound_head (page); -pg_data_t *pgdat =page_pgdat (head);struct lruvec *lruvec;struct address_space *swap_cache =NULL ;unsigned long offset =0 ;
@@ -2456,10 +2455,8 @@ static void __split_huge_page(struct page *page, struct list_head *list,xa_lock (&swap_cache->i_pages); } -/* prevent PageLRU to go away from under us, and freeze lru stats */ -spin_lock (&pgdat->lru_lock); - - lruvec =mem_cgroup_page_lruvec (head, pgdat); +/* lock lru list/PageCompound, ref freezed by page_ref_freeze */ + lruvec =lock_page_lruvec (head);for (i = nr -1 ; i >=1 ; i--) {__split_huge_page_tail (head, i, lruvec, list);
@@ -2480,7 +2477,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, }ClearPageCompound (head); -spin_unlock (&pgdat->lru_lock); +unlock_page_lruvec (lruvec);/* Caller disabled irqs, so they are still disabled here */ split_page_owner (head, nr);
mm/memcontrol.c +75 -3
103 lines not on this page · raw message
mm/mlock.c +16 -6
@@ -262,12 +262,12 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone)int nr =pagevec_count (pvec);int delta_munlocked = -nr;struct pagevec pvec_putback; +struct lruvec *lruvec =NULL ;int pgrescued =0 ;pagevec_init (&pvec_putback);/* Phase 1: page isolation */ -spin_lock_irq (&zone->zone_pgdat->lru_lock);for (i =0 ; i < nr; i++) {struct page *page = pvec->pages[i];
@@ -277,10 +277,16 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone)* so we can spare the get_page() here. */ if (TestClearPageLRU (page)) { -struct lruvec *lruvec; +struct lruvec *new_lruvec; + + new_lruvec =mem_cgroup_page_lruvec (page, +page_pgdat (page)); +if (new_lruvec != lruvec) { +if (lruvec) +unlock_page_lruvec_irq (lruvec); + lruvec =lock_page_lruvec_irq (page); + } - lruvec =mem_cgroup_page_lruvec (page, -page_pgdat (page));del_page_from_lru_list (page, lruvec,page_lru (page));continue ;
@@ -299,8 +305,12 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone)pagevec_add (&pvec_putback, pvec->pages[i]); pvec->pages[i] =NULL ; } -__mod_zone_page_state (zone,NR_MLOCK , delta_munlocked); -spin_unlock_irq (&zone->zone_pgdat->lru_lock); +if (lruvec) { +__mod_zone_page_state (zone,NR_MLOCK , delta_munlocked); +unlock_page_lruvec_irq (lruvec); + }else if (delta_munlocked) { +mod_zone_page_state (zone,NR_MLOCK , delta_munlocked); + }/* Now we can release pins of pages that we are not munlocking */ pagevec_release (&pvec_putback);
mm/mmzone.c +1 -0
@@ -91,6 +91,7 @@ void lruvec_init(struct lruvec *lruvec)enum lru_list lru;memset (lruvec,0 ,sizeof (struct lruvec )); +spin_lock_init (&lruvec->lru_lock);for_each_lru (lru)INIT_LIST_HEAD (&lruvec->lists[lru]);
mm/page_alloc.c +0 -1
@@ -6808,7 +6808,6 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat)init_waitqueue_head (&pgdat->pfmemalloc_wait);pgdat_page_ext_init (pgdat); -spin_lock_init (&pgdat->lru_lock);lruvec_init (&pgdat->__lruvec); }
mm/swap.c +58 -58
225 lines not on this page · raw message
mm/vmscan.c +25 -30
169 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Re: [PATCH v21 17/19] mm/lru: replace pgdat lru_lock with lruvec lock
Quoted 29 lines · Alex Shi
From
84e69f892119d99612e9668e3fe47a3922bafff1Mon Sep 17 00:00:00 2001
From: Alex Shi <alex.shi-KPsoFbNs7GizrGE5bRqYAgC/[email protected]>
Date: Tue, 18 Aug 2020 16:44:21 +0800
Subject: [PATCH v21 17/19] mm/lru: replace pgdatlru_lockwith lruvec lockThis patch moves per node
lru_lockinto lruvec, thus bring alru_lockfor each of memcg per node. So on a large machine, each of memcg don't have to suffer from per nodepgdat->lru_lockcompetition. They could go fast with their selflru_lock.After move memcg charge before lru inserting, page isolation could serialize page's memcg, then per memcg lruvec lock is stable and could replace per node lru lock.
In func
isolate_migratepages_block,compact_unlock_should_abortandlock_page_lruvec_irqsaveare open coded to work withcompact_control. Also add a debug func in locking which may give some clues if there are sth out of hands.Daniel Jordan's testing show 62% improvement on modified readtwice case on his 2P * 10 core * 2 HT broadwell box.
https://lore.kernel.org/lkml/20200915165807.kpp7uhiw7l3loofu-S51bK0XF4qpuJJETbFA3a0B3C2bhBk7L0E9HWUfgJXw@public.gmane.org/On a large machine with memcg enabled but not used, the page's lruvec seeking pass a few pointers, that may lead to
lru_lockholding time increase and a bit regression.Hugh Dickins helped on the patch polish, thanks!
Quoted 15 lines
Acked-by Hugh Dickins <[email protected]>Cc Rong Chen <[email protected]>Cc Hugh Dickins <[email protected]>Cc Andrew Morton <akpm-de/[email protected]>Cc Johannes Weiner <[email protected]>Cc Michal Hocko <[email protected]>Cc Vladimir Davydov <[email protected]>Cc Yang Shi <yang.shi-KPsoFbNs7GizrGE5bRqYAgC/[email protected]>Cc Matthew Wilcox <[email protected]>Cc Konstantin Khlebnikov <[email protected]>Cc Tejun Heo <[email protected]>
Re: [PATCH v21 17/19] mm/lru: replace pgdat lru_lock with lruvec lock
In reply to Alex ShiQuoted 24 lines · Alex Shi
This patch moves per node
lru_lockinto lruvec, thus bring alru_lockfor each of memcg per node. So on a large machine, each of memcg don't have to suffer from per nodepgdat->lru_lockcompetition. They could go fast with their selflru_lock.After move memcg charge before lru inserting, page isolation could serialize page's memcg, then per memcg lruvec lock is stable and could replace per node lru lock.
In func
isolate_migratepages_block,compact_unlock_should_abortandlock_page_lruvec_irqsaveare open coded to work withcompact_control. Also add a debug func in locking which may give some clues if there are sth out of hands.Daniel Jordan's testing show 62% improvement on modified readtwice case on his 2P * 10 core * 2 HT broadwell box.
https://lore.kernel.org/lkml/20200915165807.kpp7uhiw7l3loofu-S51bK0XF4qpuJJETbFA3a0B3C2bhBk7L0E9HWUfgJXw@public.gmane.org/On a large machine with memcg enabled but not used, the page's lruvec seeking pass a few pointers, that may lead to
lru_lockholding time increase and a bit regression.Hugh Dickins helped on the patch polish, thanks!
Quoted 15 lines
Acked-by Hugh Dickins <[email protected]>Cc Rong Chen <[email protected]>Cc Hugh Dickins <[email protected]>Cc Andrew Morton <akpm-de/[email protected]>Cc Johannes Weiner <[email protected]>Cc Michal Hocko <[email protected]>Cc Vladimir Davydov <[email protected]>Cc Yang Shi <yang.shi-KPsoFbNs7GizrGE5bRqYAgC/[email protected]>Cc Matthew Wilcox <[email protected]>Cc Konstantin Khlebnikov <[email protected]>Cc Tejun Heo <[email protected]>
Re: [PATCH v21 17/19] mm/lru: replace pgdat lru_lock with lruvec lock
Quoted 1 line · Vlastimil Babka
Acked-by Vlastimil Babka<[email protected]>
Err, not yet, that was supposed for patch 16/17
Re: [PATCH v21 17/19] mm/lru: replace pgdat lru_lock with lruvec lock
In reply to Alex ShiQuoted 24 lines · Alex Shi
This patch moves per node
lru_lockinto lruvec, thus bring alru_lockfor each of memcg per node. So on a large machine, each of memcg don't have to suffer from per nodepgdat->lru_lockcompetition. They could go fast with their selflru_lock.After move memcg charge before lru inserting, page isolation could serialize page's memcg, then per memcg lruvec lock is stable and could replace per node lru lock.
In func
isolate_migratepages_block,compact_unlock_should_abortandlock_page_lruvec_irqsaveare open coded to work withcompact_control. Also add a debug func in locking which may give some clues if there are sth out of hands.Daniel Jordan's testing show 62% improvement on modified readtwice case on his 2P * 10 core * 2 HT broadwell box.
https://lore.kernel.org/lkml/20200915165807.kpp7uhiw7l3loofu-S51bK0XF4qpuJJETbFA3a0B3C2bhBk7L0E9HWUfgJXw@public.gmane.org/On a large machine with memcg enabled but not used, the page's lruvec seeking pass a few pointers, that may lead to
lru_lockholding time increase and a bit regression.Hugh Dickins helped on the patch polish, thanks!
Quoted 15 lines
Acked-by Hugh Dickins <[email protected]>Cc Rong Chen <[email protected]>Cc Hugh Dickins <[email protected]>Cc Andrew Morton <akpm-de/[email protected]>Cc Johannes Weiner <[email protected]>Cc Michal Hocko <[email protected]>Cc Vladimir Davydov <[email protected]>Cc Yang Shi <yang.shi-KPsoFbNs7GizrGE5bRqYAgC/[email protected]>Cc Matthew Wilcox <[email protected]>Cc Konstantin Khlebnikov <[email protected]>Cc Tejun Heo <[email protected]>
I think I need some explanation about the rcu_read_lock() usage in lock_page_lruvec*() (and places effectively opencoding it). Preferably in form of some code comment, but that can be also added as a additional patch later, I don't want to block the series.
mem_cgroup_page_lruvec() comment says
* This function relies on page->mem_cgroup being stable - see the
* access rules in commit_charge().
commit_charge() comment:
* Any of the following ensures page->mem_cgroup stability:
*
* - the page lock
* - LRU isolation
* - lock_page_memcg()
* - exclusive reference
"LRU isolation" used to be quite clear, but now is it after TestClearPageLRU(page) or after deleting from the lru list as well? Also it doesn't mention rcu_read_lock(), should it?
So what exactly are we protecting by rcu_read_lock() in e.g. lock_page_lruvec()?
rcu_read_lock();
lruvec = mem_cgroup_page_lruvec(page, pgdat);
spin_lock(&lruvec->lru_lock);
rcu_read_unlock();Looks like we are protecting the lruvec from going away and it can't go away anymore after we take the lru_lock?
But then e.g. in __munlock_pagevec() we are doing this without an rcu_read_lock():
new_lruvec = mem_cgroup_page_lruvec(page, page_pgdat(page));
where new_lruvec is potentionally not the one that we have locked
And the last thing mem_cgroup_page_lruvec() is doing is:
if (unlikely(lruvec->pgdat != pgdat))
lruvec->pgdat = pgdat;
return lruvec;So without the rcu_read_lock() is this potentionally accessing the pgdat field of lruvec that might have just gone away?
Thanks,
Vlastimil
Re: [PATCH v21 17/19] mm/lru: replace pgdat lru_lock with lruvec lock
Quoted 41 lines · Vlastimil Babka
Quoted 24 lines · Alex Shi
This patch moves per node
lru_lockinto lruvec, thus bring alru_lockfor each of memcg per node. So on a large machine, each of memcg don't have to suffer from per nodepgdat->lru_lockcompetition. They could go fast with their selflru_lock.After move memcg charge before lru inserting, page isolation could serialize page's memcg, then per memcg lruvec lock is stable and could replace per node lru lock.
In func
isolate_migratepages_block,compact_unlock_should_abortandlock_page_lruvec_irqsaveare open coded to work withcompact_control. Also add a debug func in locking which may give some clues if there are sth out of hands.Daniel Jordan's testing show 62% improvement on modified readtwice case on his 2P * 10 core * 2 HT broadwell box.
https://lore.kernel.org/lkml/20200915165807.kpp7uhiw7l3loofu-S51bK0XF4qpuJJETbFA3a0B3C2bhBk7L0E9HWUfgJXw@public.gmane.org/On a large machine with memcg enabled but not used, the page's lruvec seeking pass a few pointers, that may lead to
lru_lockholding time increase and a bit regression.Hugh Dickins helped on the patch polish, thanks!
Quoted 15 lines
Acked-by Hugh Dickins <[email protected]>Cc Rong Chen <[email protected]>Cc Hugh Dickins <[email protected]>Cc Andrew Morton <akpm-de/[email protected]>Cc Johannes Weiner <[email protected]>Cc Michal Hocko <[email protected]>Cc Vladimir Davydov <[email protected]>Cc Yang Shi <yang.shi-KPsoFbNs7GizrGE5bRqYAgC/[email protected]>Cc Matthew Wilcox <[email protected]>Cc Konstantin Khlebnikov <[email protected]>Cc Tejun Heo <[email protected]>
Quoted 3 lines
I think I need some explanation about the
rcu_read_lock()usage in lock_page_lruvec*() (and places effectively opencoding it). Preferably in form of some code comment, but that can be also added as a additional patch later, I don't want to block the series.
Hi Vlastimil,
Thanks for comments!
Oh, we did talk about the rcu_read_lock which is used to block memcg destroy during locking. and the spin_lock actually includes a rcu_read_lock(). Yes, we could add this comments later.
Quoted 14 lines
mem_cgroup_page_lruvec()comment says* This function relies on
page->mem_cgroupbeing stable - see the
* access rules incommit_charge().
commit_charge()comment:* Any of the following ensures
page->mem_cgroupstability:
*
* - the page lock
* - LRU isolation
* -lock_page_memcg()
* - exclusive reference
Quoted 2 lines
"LRU isolation" used to be quite clear, but now is it after TestClearPageLRU(page) or after deleting from the lru list as well? Also it doesn't mention
rcu_read_lock(), should it?
The lru isolation still is same as old conception, a set actions that take a page from a lru list, and commit_charge do need a isoltion for the page.
but the condition of page_memcg could be change since we don't rely on lru isolation for it. The comments could be changed later.
Quoted 8 lines
So what exactly are we protecting by
rcu_read_lock()in e.g.lock_page_lruvec()?
rcu_read_lock();
lruvec =mem_cgroup_page_lruvec(page, pgdat);spin_lock(&lruvec->lru_lock);rcu_read_unlock();
Quoted 5 lines
Looks like we are protecting the lruvec from going away and it can't go away anymore after we take the
lru_lock?But then e.g. in
__munlock_pagevec()we are doing this without anrcu_read_lock():new_lruvec =
mem_cgroup_page_lruvec(page, page_pgdat(page));
TestClearPageLRU could block the page from memcg migration/destory.
Thanks
Alex
Quoted 13 lines
where
new_lruvecis potentionally not the one that we have lockedAnd the last thing
mem_cgroup_page_lruvec()is doing is:if (
unlikely(lruvec->pgdat != pgdat))
lruvec->pgdat= pgdat;
return lruvec;So without the
rcu_read_lock()is this potentionally accessing the pgdat field of lruvec that might have just gone away?Thanks,
Vlastimil
PATCH v21 18/19 mm/lru: introduce the relock_page_lruvec function
In reply to Alex ShiFrom: Alexander Duyck <alexander.h.duyck-VuQAYsv1563Yd54FQh9/[email protected]>
Use this new function to replace repeated same code, no func change.
When testing for relock we can avoid the need for RCU locking if we simply compare the page pgdat and memcg pointers versus those that the lruvec is holding. By doing this we can avoid the extra pointer walks and accesses of the memory cgroup.
In addition we can avoid the checks entirely if lruvec is currently NULL.
--- include/linux/memcontrol.h | 52 ++++++++++++++++++++++++++++++++++++++++++++++ mm/mlock.c | 11 +--------- mm/swap.c | 33 +++++++---------------------- mm/vmscan.c | 12 ++--------- 4 files changed, 62 insertions(+), 46 deletions(-)
include/linux/memcontrol.h +52 -0
@@ -660,6 +660,22 @@ static inline struct lruvec *mem_cgroup_lruvec(struct mem_cgroup *memcg,struct lruvec *mem_cgroup_page_lruvec (struct page *,struct pglist_data *); +static inline bool lruvec_holds_page_lru_lock (struct page *page, +struct lruvec *lruvec) +{ +pg_data_t *pgdat =page_pgdat (page); +const struct mem_cgroup *memcg; +struct mem_cgroup_per_node *mz; + +if (mem_cgroup_disabled ()) +return lruvec == &pgdat->__lruvec; + + mz = container_of(lruvec, struct mem_cgroup_per_node, lruvec); + memcg = page->mem_cgroup ? : root_mem_cgroup; + +return lruvec->pgdat == pgdat && mz->memcg == memcg; +} +struct mem_cgroup *mem_cgroup_from_task (struct task_struct *p);struct mem_cgroup *get_mem_cgroup_from_mm (struct mm_struct *mm);
@@ -1221,6 +1237,14 @@ static inline struct lruvec *mem_cgroup_page_lruvec(struct page *page,return &pgdat->__lruvec; } +static inline bool lruvec_holds_page_lru_lock (struct page *page, +struct lruvec *lruvec) +{ +pg_data_t *pgdat =page_pgdat (page); + +return lruvec == &pgdat->__lruvec; +} +static inline struct mem_cgroup *parent_mem_cgroup (struct mem_cgroup *memcg) {return NULL ;
@@ -1663,6 +1687,34 @@ static inline void unlock_page_lruvec_irqrestore(struct lruvec *lruvec,spin_unlock_irqrestore (&lruvec->lru_lock, flags); } +/* Don't lock again iff page's lruvec locked */ +static inline struct lruvec *relock_page_lruvec_irq (struct page *page, +struct lruvec *locked_lruvec) +{ +if (locked_lruvec) { +if (lruvec_holds_page_lru_lock (page, locked_lruvec)) +return locked_lruvec; + +unlock_page_lruvec_irq (locked_lruvec); + } + +return lock_page_lruvec_irq (page); +} + +/* Don't lock again iff page's lruvec locked */ +static inline struct lruvec *relock_page_lruvec_irqsave (struct page *page, +struct lruvec *locked_lruvec,unsigned long *flags) +{ +if (locked_lruvec) { +if (lruvec_holds_page_lru_lock (page, locked_lruvec)) +return locked_lruvec; + +unlock_page_lruvec_irqrestore (locked_lruvec, *flags); + } + +return lock_page_lruvec_irqsave (page, flags); +} +#ifdef CONFIG_CGROUP_WRITEBACK struct wb_domain *mem_cgroup_wb_domain (struct bdi_writeback *wb);
mm/mlock.c +1 -10
@@ -277,16 +277,7 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone)* so we can spare the get_page() here. */ if (TestClearPageLRU (page)) { -struct lruvec *new_lruvec; - - new_lruvec =mem_cgroup_page_lruvec (page, -page_pgdat (page)); -if (new_lruvec != lruvec) { -if (lruvec) -unlock_page_lruvec_irq (lruvec); - lruvec =lock_page_lruvec_irq (page); - } - + lruvec =relock_page_lruvec_irq (page, lruvec);del_page_from_lru_list (page, lruvec,page_lru (page));continue ;
mm/swap.c +7 -26
@@ -210,19 +210,12 @@ static void pagevec_lru_move_fn(struct pagevec *pvec,for (i =0 ; i <pagevec_count (pvec); i++) {struct page *page = pvec->pages[i]; -struct lruvec *new_lruvec;/* block memcg migration during page moving between lru */ if (!TestClearPageLRU (page))continue ; - new_lruvec =mem_cgroup_page_lruvec (page,page_pgdat (page)); -if (lruvec != new_lruvec) { -if (lruvec) -unlock_page_lruvec_irqrestore (lruvec, flags); - lruvec =lock_page_lruvec_irqsave (page, &flags); - } - + lruvec =relock_page_lruvec_irqsave (page, lruvec, &flags); (*move_fn)(page, lruvec);SetPageLRU (page);
@@ -918,17 +911,12 @@ void release_pages(struct page **pages, int nr) }if (PageLRU (page)) { -struct lruvec *new_lruvec; - - new_lruvec =mem_cgroup_page_lruvec (page, -page_pgdat (page)); -if (new_lruvec != lruvec) { -if (lruvec) -unlock_page_lruvec_irqrestore (lruvec, - flags); +struct lruvec *prev_lruvec = lruvec; + + lruvec =relock_page_lruvec_irqsave (page, lruvec, + &flags); +if (prev_lruvec != lruvec) lock_batch =0 ; - lruvec =lock_page_lruvec_irqsave (page, &flags); - }VM_BUG_ON_PAGE (!PageLRU (page), page);__ClearPageLRU (page);
@@ -1033,15 +1021,8 @@ void __pagevec_lru_add(struct pagevec *pvec)for (i =0 ; i <pagevec_count (pvec); i++) {struct page *page = pvec->pages[i]; -struct lruvec *new_lruvec; - - new_lruvec =mem_cgroup_page_lruvec (page,page_pgdat (page)); -if (lruvec != new_lruvec) { -if (lruvec) -unlock_page_lruvec_irqrestore (lruvec, flags); - lruvec =lock_page_lruvec_irqsave (page, &flags); - } + lruvec =relock_page_lruvec_irqsave (page, lruvec, &flags);__pagevec_lru_add_fn (page, lruvec); }if (lruvec)
mm/vmscan.c +2 -10
@@ -1884,8 +1884,7 @@ static unsigned noinline_for_stack move_pages_to_lru(struct lruvec *lruvec,* All pages were isolated from the same lruvec (and isolation * inhibits memcg migration). */ -VM_BUG_ON_PAGE (mem_cgroup_page_lruvec (page,page_pgdat (page)) - != lruvec, page); +VM_BUG_ON_PAGE (!lruvec_holds_page_lru_lock (page, lruvec), page); lru =page_lru (page); nr_pages =thp_nr_pages (page);
@@ -4277,7 +4276,6 @@ void check_move_unevictable_pages(struct pagevec *pvec)for (i =0 ; i < pvec->nr; i++) {struct page *page = pvec->pages[i];int nr_pages; -struct lruvec *new_lruvec;if (PageTransTail (page))continue ;
@@ -4289,13 +4287,7 @@ void check_move_unevictable_pages(struct pagevec *pvec)if (!TestClearPageLRU (page))continue ; - new_lruvec =mem_cgroup_page_lruvec (page,page_pgdat (page)); -if (lruvec != new_lruvec) { -if (lruvec) -unlock_page_lruvec_irq (lruvec); - lruvec =lock_page_lruvec_irq (page); - } - + lruvec =relock_page_lruvec_irq (page, lruvec);if (page_evictable (page) &&PageUnevictable (page)) {enum lru_list lru =page_lru_base_type (page);
Signature · 1 line
1.8.3.1
Re: [PATCH v21 18/19] mm/lru: introduce the relock_page_lruvec function
ÔÚ 2020/11/5 ÏÂÎç4:55, Alex Shi дµÀ:
Quoted 1 line
From: Alexander Duyck <[email protected]>
update the patch on page_memcg() change:
From 6c142eb582e7d0dbf473572ad092eca07ab75221 Mon Sep 17 00:00:00 2001
From: Alexander Duyck <[email protected]>
Date: Tue, 26 May 2020 17:31:15 +0800
Subject: [PATCH v21 18/19] mm/lru: introduce the relock_page_lruvec function
Use this new function to replace repeated same code, no func change.
When testing for relock we can avoid the need for RCU locking if we simply compare the page pgdat and memcg pointers versus those that the lruvec is holding. By doing this we can avoid the extra pointer walks and accesses of the memory cgroup.
In addition we can avoid the checks entirely if lruvec is currently NULL.
--- include/linux/memcontrol.h | 52 ++++++++++++++++++++++++++++++++++++++++++++++ mm/mlock.c | 11 +--------- mm/swap.c | 33 +++++++---------------------- mm/vmscan.c | 12 ++--------- 4 files changed, 62 insertions(+), 46 deletions(-)
include/linux/memcontrol.h +52 -0
@@ -660,6 +660,22 @@ static inline struct lruvec *mem_cgroup_lruvec(struct mem_cgroup *memcg,struct lruvec *mem_cgroup_page_lruvec (struct page *,struct pglist_data *); +static inline bool lruvec_holds_page_lru_lock (struct page *page, +struct lruvec *lruvec) +{ +pg_data_t *pgdat =page_pgdat (page); +const struct mem_cgroup *memcg; +struct mem_cgroup_per_node *mz; + +if (mem_cgroup_disabled ()) +return lruvec == &pgdat->__lruvec; + + mz = container_of(lruvec, struct mem_cgroup_per_node, lruvec); + memcg =page_memcg (page) ? : root_mem_cgroup; + +return lruvec->pgdat == pgdat && mz->memcg == memcg; +} +struct mem_cgroup *mem_cgroup_from_task (struct task_struct *p);struct mem_cgroup *get_mem_cgroup_from_mm (struct mm_struct *mm);
@@ -1221,6 +1237,14 @@ static inline struct lruvec *mem_cgroup_page_lruvec(struct page *page,return &pgdat->__lruvec; } +static inline bool lruvec_holds_page_lru_lock (struct page *page, +struct lruvec *lruvec) +{ +pg_data_t *pgdat =page_pgdat (page); + +return lruvec == &pgdat->__lruvec; +} +static inline struct mem_cgroup *parent_mem_cgroup (struct mem_cgroup *memcg) {return NULL ;
@@ -1663,6 +1687,34 @@ static inline void unlock_page_lruvec_irqrestore(struct lruvec *lruvec,spin_unlock_irqrestore (&lruvec->lru_lock, flags); } +/* Don't lock again iff page's lruvec locked */ +static inline struct lruvec *relock_page_lruvec_irq (struct page *page, +struct lruvec *locked_lruvec) +{ +if (locked_lruvec) { +if (lruvec_holds_page_lru_lock (page, locked_lruvec)) +return locked_lruvec; + +unlock_page_lruvec_irq (locked_lruvec); + } + +return lock_page_lruvec_irq (page); +} + +/* Don't lock again iff page's lruvec locked */ +static inline struct lruvec *relock_page_lruvec_irqsave (struct page *page, +struct lruvec *locked_lruvec,unsigned long *flags) +{ +if (locked_lruvec) { +if (lruvec_holds_page_lru_lock (page, locked_lruvec)) +return locked_lruvec; + +unlock_page_lruvec_irqrestore (locked_lruvec, *flags); + } + +return lock_page_lruvec_irqsave (page, flags); +} +#ifdef CONFIG_CGROUP_WRITEBACK struct wb_domain *mem_cgroup_wb_domain (struct bdi_writeback *wb);
mm/mlock.c +1 -10
@@ -277,16 +277,7 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone)* so we can spare the get_page() here. */ if (TestClearPageLRU (page)) { -struct lruvec *new_lruvec; - - new_lruvec =mem_cgroup_page_lruvec (page, -page_pgdat (page)); -if (new_lruvec != lruvec) { -if (lruvec) -unlock_page_lruvec_irq (lruvec); - lruvec =lock_page_lruvec_irq (page); - } - + lruvec =relock_page_lruvec_irq (page, lruvec);del_page_from_lru_list (page, lruvec,page_lru (page));continue ;
mm/swap.c +7 -26
@@ -210,19 +210,12 @@ static void pagevec_lru_move_fn(struct pagevec *pvec,for (i =0 ; i <pagevec_count (pvec); i++) {struct page *page = pvec->pages[i]; -struct lruvec *new_lruvec;/* block memcg migration during page moving between lru */ if (!TestClearPageLRU (page))continue ; - new_lruvec =mem_cgroup_page_lruvec (page,page_pgdat (page)); -if (lruvec != new_lruvec) { -if (lruvec) -unlock_page_lruvec_irqrestore (lruvec, flags); - lruvec =lock_page_lruvec_irqsave (page, &flags); - } - + lruvec =relock_page_lruvec_irqsave (page, lruvec, &flags); (*move_fn)(page, lruvec);SetPageLRU (page);
@@ -918,17 +911,12 @@ void release_pages(struct page **pages, int nr) }if (PageLRU (page)) { -struct lruvec *new_lruvec; - - new_lruvec =mem_cgroup_page_lruvec (page, -page_pgdat (page)); -if (new_lruvec != lruvec) { -if (lruvec) -unlock_page_lruvec_irqrestore (lruvec, - flags); +struct lruvec *prev_lruvec = lruvec; + + lruvec =relock_page_lruvec_irqsave (page, lruvec, + &flags); +if (prev_lruvec != lruvec) lock_batch =0 ; - lruvec =lock_page_lruvec_irqsave (page, &flags); - }VM_BUG_ON_PAGE (!PageLRU (page), page);__ClearPageLRU (page);
@@ -1033,15 +1021,8 @@ void __pagevec_lru_add(struct pagevec *pvec)for (i =0 ; i <pagevec_count (pvec); i++) {struct page *page = pvec->pages[i]; -struct lruvec *new_lruvec; - - new_lruvec =mem_cgroup_page_lruvec (page,page_pgdat (page)); -if (lruvec != new_lruvec) { -if (lruvec) -unlock_page_lruvec_irqrestore (lruvec, flags); - lruvec =lock_page_lruvec_irqsave (page, &flags); - } + lruvec =relock_page_lruvec_irqsave (page, lruvec, &flags);__pagevec_lru_add_fn (page, lruvec); }if (lruvec)
mm/vmscan.c +2 -10
@@ -1884,8 +1884,7 @@ static unsigned noinline_for_stack move_pages_to_lru(struct lruvec *lruvec,* All pages were isolated from the same lruvec (and isolation * inhibits memcg migration). */ -VM_BUG_ON_PAGE (mem_cgroup_page_lruvec (page,page_pgdat (page)) - != lruvec, page); +VM_BUG_ON_PAGE (!lruvec_holds_page_lru_lock (page, lruvec), page); lru =page_lru (page); nr_pages =thp_nr_pages (page);
@@ -4277,7 +4276,6 @@ void check_move_unevictable_pages(struct pagevec *pvec)for (i =0 ; i < pvec->nr; i++) {struct page *page = pvec->pages[i];int nr_pages; -struct lruvec *new_lruvec;if (PageTransTail (page))continue ;
@@ -4289,13 +4287,7 @@ void check_move_unevictable_pages(struct pagevec *pvec)if (!TestClearPageLRU (page))continue ; - new_lruvec =mem_cgroup_page_lruvec (page,page_pgdat (page)); -if (lruvec != new_lruvec) { -if (lruvec) -unlock_page_lruvec_irq (lruvec); - lruvec =lock_page_lruvec_irq (page); - } - + lruvec =relock_page_lruvec_irq (page, lruvec);if (page_evictable (page) &&PageUnevictable (page)) {enum lru_list lru =page_lru_base_type (page);
Signature · 1 line
1.8.3.1
Re: [PATCH v21 18/19] mm/lru: introduce the relock_page_lruvec function
Quoted 14 lines · Alex Shi
From
6c142eb582e7d0dbf473572ad092eca07ab75221Mon Sep 17 00:00:00 2001
From: Alexander Duyck <alexander.h.duyck-VuQAYsv1563Yd54FQh9/[email protected]>
Date: Tue, 26 May 2020 17:31:15 +0800
Subject: [PATCH v21 18/19] mm/lru: introduce therelock_page_lruvecfunctionUse this new function to replace repeated same code, no func change.
When testing for relock we can avoid the need for RCU locking if we simply compare the page pgdat and memcg pointers versus those that the lruvec is holding. By doing this we can avoid the extra pointer walks and accesses of the memory cgroup.
In addition we can avoid the checks entirely if lruvec is currently NULL.
Quoted 16 lines
Acked-by Hugh Dickins <[email protected]>Acked-by Johannes Weiner <[email protected]>Cc Johannes Weiner <[email protected]>Cc Andrew Morton <akpm-de/[email protected]>Cc Thomas Gleixner <[email protected]>Cc Andrey Ryabinin <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Mel Gorman <[email protected]>Cc Konstantin Khlebnikov <[email protected]>Cc Hugh Dickins <[email protected]>Cc Tejun Heo <[email protected]>
Re: [PATCH v21 18/19] mm/lru: introduce the relock_page_lruvec function
In reply to Alex ShiQuoted 8 lines · Alex Shi
From: Alexander Duyck <alexander.h.duyck-VuQAYsv1563Yd54FQh9/[email protected]>
Use this new function to replace repeated same code, no func change.
When testing for relock we can avoid the need for RCU locking if we simply compare the page pgdat and memcg pointers versus those that the lruvec is holding. By doing this we can avoid the extra pointer walks and accesses of the memory cgroup.
Ah, clever. Seems to address my worry from previous patch (except the potential to improve documenting comments).
Quoted 6 lines
In addition we can avoid the checks entirely if lruvec is currently NULL.
Quoted 12 lines
Cc Johannes Weiner <[email protected]>Cc Andrew Morton <akpm-de/[email protected]>Cc Thomas Gleixner <[email protected]>Cc Andrey Ryabinin <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Mel Gorman <[email protected]>Cc Konstantin Khlebnikov <[email protected]>Cc Hugh Dickins <[email protected]>Cc Tejun Heo <[email protected]>
PATCH v21 19/19 mm/lru: revise the comments of lru_lock
In reply to Alex ShiFrom: Hugh Dickins <[email protected]>
Since we changed the pgdat->lru_lock to lruvec->lru_lock, it's time to fix the incorrect comments in code. Also fixed some zone->lru_lock comment error from ancient time. etc.
I struggled to understand the comment above move_pages_to_lru() (surely it never calls page_referenced()), and eventually realized that most of it had got separated from shrink_active_list(): move that comment back.
--- Documentation/admin-guide/cgroup-v1/memcg_test.rst | 15 ++------ Documentation/admin-guide/cgroup-v1/memory.rst | 21 +++++------ Documentation/trace/events-kmem.rst | 2 +- Documentation/vm/unevictable-lru.rst | 22 +++++------- include/linux/mm_types.h | 2 +- include/linux/mmzone.h | 3 +- mm/filemap.c | 4 +-- mm/rmap.c | 4 +-- mm/vmscan.c | 41 ++++++++++++---------- 9 files changed, 50 insertions(+), 64 deletions(-)
Documentation/admin-guide/cgroup-v1/memcg_test.rst +3 -12
@@ -133,18 +133,9 @@ Under below explanation, we assume CONFIG_MEM_RES_CTRL_SWAP=y. 8. LRU ====== - Each memcg has its own private LRU. Now, its handling is under global - VM's control (means that it's handled under global pgdat->lru_lock). - Almost all routines around memcg's LRU is called by global LRU's - list management functions under pgdat->lru_lock. - - A special function is mem_cgroup_isolate_pages(). This scans - memcg's private LRU and call __isolate_lru_page() to extract a page - from LRU. - - (By __isolate_lru_page(), the page is removed from both of global and - private LRU.) - + Each memcg has its own vector of LRUs (inactive anon, active anon, + inactive file, active file, unevictable) of pages from each node, + each LRU handled under a single lru_lock for that memcg and node. 9. Typical Tests. =================
Documentation/admin-guide/cgroup-v1/memory.rst +9 -12
@@ -285,20 +285,17 @@ When oom event notifier is registered, event will be delivered. 2.6 Locking ----------- - lock_page_cgroup()/unlock_page_cgroup() should not be called under - the i_pages lock. +Lock order is as follows: - Other lock order is following: + Page lock (PG_locked bit of page->flags) + mm->page_table_lock or split pte_lock + lock_page_memcg (memcg->move_lock) + mapping->i_pages lock + lruvec->lru_lock. - PG_locked. - mm->page_table_lock - pgdat->lru_lock - lock_page_cgroup. - - In many cases, just lock_page_cgroup() is called. - - per-zone-per-cgroup LRU (cgroup's private LRU) is just guarded by - pgdat->lru_lock, it has no lock of its own. +Per-node-per-memcgroup LRU (cgroup's private LRU) is guarded by +lruvec->lru_lock; PG_lru bit of page->flags is cleared before +isolating a page from its LRU under lruvec->lru_lock. 2.7 Kernel Memory Extension (CONFIG_MEMCG_KMEM) -----------------------------------------------
Documentation/trace/events-kmem.rst +1 -1
@@ -69,7 +69,7 @@ When pages are freed in batch, the also mm_page_free_batched is triggered. Broadly speaking, pages are taken off the LRU lock in bulk and freed in batch with a page list. Significant amounts of activity here could indicate that the system is under memory pressure and can also indicate -contention on the zone->lru_lock. +contention on the lruvec->lru_lock. 4. Per-CPU Allocator Activity =============================
Documentation/vm/unevictable-lru.rst +8 -14
@@ -33,7 +33,7 @@ reclaim in Linux. The problems have been observed at customer sites on large memory x86_64 systems. To illustrate this with an example, a non-NUMA x86_64 platform with 128GB of -main memory will have over 32 million 4k pages in a single zone. When a large +main memory will have over 32 million 4k pages in a single node. When a large fraction of these pages are not evictable for any reason [see below], vmscan will spend a lot of time scanning the LRU lists looking for the small fraction of pages that are evictable. This can result in a situation where all CPUs are
@@ -55,7 +55,7 @@ unevictable, either by definition or by circumstance, in the future. The Unevictable Page List ------------------------- -The Unevictable LRU infrastructure consists of an additional, per-zone, LRU list +The Unevictable LRU infrastructure consists of an additional, per-node, LRU list called the "unevictable" list and an associated page flag, PG_unevictable, to indicate that the page is being managed on the unevictable list.
@@ -84,15 +84,9 @@ The unevictable list does not differentiate between file-backed and anonymous, swap-backed pages. This differentiation is only important while the pages are, in fact, evictable. -The unevictable list benefits from the "arrayification" of the per-zone LRU +The unevictable list benefits from the "arrayification" of the per-node LRU lists and statistics originally proposed and posted by Christoph Lameter. -The unevictable list does not use the LRU pagevec mechanism. Rather, -unevictable pages are placed directly on the page's zone's unevictable list -under the zone lru_lock. This allows us to prevent the stranding of pages on -the unevictable list when one task has the page isolated from the LRU and other -tasks are changing the "evictability" state of the page. - Memory Control Group Interaction --------------------------------
@@ -101,8 +95,8 @@ The unevictable LRU facility interacts with the memory control group [aka memory controller; see Documentation/admin-guide/cgroup-v1/memory.rst] by extending the lru_list enum. -The memory controller data structure automatically gets a per-zone unevictable -list as a result of the "arrayification" of the per-zone LRU lists (one per +The memory controller data structure automatically gets a per-node unevictable +list as a result of the "arrayification" of the per-node LRU lists (one per lru_list enum element). The memory controller tracks the movement of pages to and from the unevictable list.
@@ -196,7 +190,7 @@ for the sake of expediency, to leave a unevictable page on one of the regular active/inactive LRU lists for vmscan to deal with. vmscan checks for such pages in all of the shrink_{active|inactive|page}_list() functions and will "cull" such pages that it encounters: that is, it diverts those pages to the -unevictable list for the zone being scanned. +unevictable list for the node being scanned. There may be situations where a page is mapped into a VM_LOCKED VMA, but the page is not marked as PG_mlocked. Such pages will make it all the way to
@@ -328,7 +322,7 @@ If the page was NOT already mlocked, mlock_vma_page() attempts to isolate the page from the LRU, as it is likely on the appropriate active or inactive list at that time. If the isolate_lru_page() succeeds, mlock_vma_page() will put back the page - by calling putback_lru_page() - which will notice that the page -is now mlocked and divert the page to the zone's unevictable list. If +is now mlocked and divert the page to the node's unevictable list. If mlock_vma_page() is unable to isolate the page from the LRU, vmscan will handle it later if and when it attempts to reclaim the page.
@@ -603,7 +597,7 @@ Some examples of these unevictable pages on the LRU lists are: unevictable list in mlock_vma_page(). shrink_inactive_list() also diverts any unevictable pages that it finds on the -inactive lists to the appropriate zone's unevictable list. +inactive lists to the appropriate node's unevictable list. shrink_inactive_list() should only see SHM_LOCK'd pages that became SHM_LOCK'd after shrink_active_list() had moved them to the inactive list, or pages mapped
include/linux/mm_types.h +1 -1
@@ -78,7 +78,7 @@ struct page {struct {/* Page cache and anonymous pages */ /** * @lru: Pageout list, eg. active_list protected by -* pgdat->lru_lock. Sometimes used as a generic list +* lruvec->lru_lock. Sometimes used as a generic list * by the page owner. */ struct list_head lru;
include/linux/mmzone.h +1 -2
@@ -113,8 +113,7 @@ static inline bool free_area_empty(struct free_area *area, int migratetype)struct pglist_data ;/* -* zone->lock and the zone lru_lock are two of the hottest locks in the kernel. -* So add a wild amount of padding here to ensure that they fall into separate +* Add a wild amount of padding here to ensure datas fall into separate * cachelines. There are very few zone structures in the machine, so space * consumption is not a concern here. */
mm/filemap.c +2 -2
@@ -102,8 +102,8 @@* ->swap_lock (try_to_unmap_one) * ->private_lock (try_to_unmap_one) * ->i_pages lock (try_to_unmap_one) -* ->pgdat->lru_lock (follow_page->mark_page_accessed) -* ->pgdat->lru_lock (check_pte_range->isolate_lru_page) +* ->lruvec->lru_lock (follow_page->mark_page_accessed) +* ->lruvec->lru_lock (check_pte_range->isolate_lru_page) * ->private_lock (page_remove_rmap->set_page_dirty) * ->i_pages lock (page_remove_rmap->set_page_dirty) * bdi.wb->list_lock (page_remove_rmap->set_page_dirty)
mm/rmap.c +2 -2
@@ -28,12 +28,12 @@* hugetlb_fault_mutex (hugetlbfs specific page fault mutex) * anon_vma->rwsem * mm->page_table_lock or pte_lock -* pgdat->lru_lock (in mark_page_accessed, isolate_lru_page) * swap_lock (in swap_duplicate, swap_info_get) * mmlist_lock (in mmput, drain_mmlist and others) * mapping->private_lock (in __set_page_dirty_buffers) -* mem_cgroup_{begin,end}_page_stat (memcg->move_lock) +* lock_page_memcg move_lock (in __set_page_dirty_buffers) * i_pages lock (widely used) +* lruvec->lru_lock (in lock_page_lruvec_irq) * inode->i_lock (in set_page_dirty's __mark_inode_dirty) * bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty) * sb_lock (within inode_lock in fs/fs-writeback.c)
mm/vmscan.c +23 -18
68 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Re: [PATCH v21 19/19] mm/lru: revise the comments of lru_lock
Quoted 10 lines · Alex Shi
From: Hugh Dickins <[email protected]>
Since we changed the
pgdat->lru_locktolruvec->lru_lock, it's time to fix the incorrect comments in code. Also fixed somezone->lru_lockcomment error from ancient time. etc.I struggled to understand the comment above
move_pages_to_lru()(surely it never callspage_referenced()), and eventually realized that most of it had got separated fromshrink_active_list(): move that comment back.
Quoted 14 lines
Signed-off-by Hugh Dickins <[email protected]>Acked-by Johannes Weiner <[email protected]>Cc Andrew Morton <akpm-de/[email protected]>Cc Tejun Heo <[email protected]>Cc Andrey Ryabinin <[email protected]>Cc Jann Horn <[email protected]>Cc Mel Gorman <[email protected]>Cc Johannes Weiner <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Hugh Dickins <[email protected]>
Hi All,
Is any more comments of this version?
Thanks
Alex
Quoted 102 lines · Alex Shi
This version rebase on next/master 20201104, with much of Johannes's Acks and some changes according to Johannes comments. And add a new patch v21-0006-mm-rmap-stop-store-reordering-issue-on-page-mapp.patch to support v21-0007.
This patchset followed 2 memcg
VM_WARN_ON_ONCE_PAGEpatches which were added to -mm tree yesterday.Many thanks for line by line review by Hugh Dickins, Alexander Duyck and Johannes Weiner.
So now this patchset includes 3 parts:
1, some code cleanup and minimum optimization as a preparation. 2, use TestCleanPageLRU as page isolation's precondition. 3, replace per nodelru_lockwith per memcg per nodelru_lock.Current
lru_lockis one for each of node,pgdat->lru_lock, that guard for lru lists, but now we had moved the lru lists into memcg for long time. Still using per nodelru_lockis clearly unscalable, pages on each of memcgs have to compete each others for a wholelru_lock. This patchset try to use per lruvec/memcglru_lockto repleace per node lru lock to guard lru lists, make it scalable for memcgs and get performance gain.Currently
lru_lockstill guards both lru list and page's lru bit, that's ok. but if we want to use specific lruvec lock on the page, we need to pin down the page's lruvec/memcg during locking. Just taking lruvec lock first may be undermined by the page's memcg charge/migration. To fix this problem, we could take out the page's lru bit clear and use it as pin down action to block the memcg changes. That's the reason for new atomic func TestClearPageLRU. So now isolating a page need both actions: TestClearPageLRU and hold thelru_lock.The typical usage of this is
isolate_migratepages_block()incompaction.cwe have to take lru bit before lru lock, that serialized the page isolation in memcg page charge/migration which will change page's lruvec and newlru_lockin it.The above solution suggested by Johannes Weiner, and based on his new memcg charge path, then have this patchset. (Hugh Dickins tested and contributed much code from compaction fix to general code polish, thanks a lot!).
Daniel Jordan's testing show 62% improvement on modified readtwice case on his 2P * 10 core * 2 HT broadwell box on v18, which has no much different with this v20.
https://lore.kernel.org/lkml/20200915165807.kpp7uhiw7l3loofu-S51bK0XF4qpuJJETbFA3a0B3C2bhBk7L0E9HWUfgJXw@public.gmane.org/Thanks Hugh Dickins and Konstantin Khlebnikov, they both brought this idea 8 years ago, and others who give comments as well: Daniel Jordan, Mel Gorman, Shakeel Butt, Matthew Wilcox, Alexander Duyck etc.
Thanks for Testing support from Intel 0day and Rong Chen, Fengguang Wu, and Yun Wang. Hugh Dickins also shared his kbuild-swap case. Thanks!
Alex Shi (16):
mm/thp: movelru_add_page_tailfunc tohuge_memory.cmm/thp: use head for head page inlru_add_page_tailmm/thp: Simplifylru_add_page_tail()
mm/thp: narrow lru locking
mm/vmscan: remove unnecessary lruvec adding mm/rmap: stop store reordering issue onpage->mappingmm/memcg: add debug checking inlock_page_memcgmm/swap.c: fold vm event PGROTATED intopagevec_move_tail_fnmm/lru: move lock intolru_note_cost
mm/vmscan: remove lruvec reget inmove_pages_to_lrumm/mlock: removelru_lockon TestClearPageMlocked mm/mlock: remove__munlock_isolate_lru_pagemm/lru: introduce TestClearPageLRU
mm/compaction: do page isolation first in compactionmm/swap.c: serialize memcg changes inpagevec_lru_move_fnmm/lru: replace pgdatlru_lockwith lruvec lockAlexander Duyck (1):
mm/lru: introduce therelock_page_lruvecfunctionHugh Dickins (2):
mm:page_idle_get_page()does not needlru_lock
mm/lru: revise the comments oflru_lockDocumentation/admin-guide/cgroup-v1/memcg_test.rst | 15 +- Documentation/admin-guide/cgroup-v1/memory.rst | 21 +-- Documentation/trace/events-kmem.rst | 2 +- Documentation/vm/unevictable-lru.rst | 22 +-- include/linux/memcontrol.h | 110 +++++++++++ include/linux/mm_types.h | 2 +- include/linux/mmzone.h | 6 +- include/linux/page-flags.h | 1 + include/linux/swap.h | 4 +- mm/compaction.c | 94 +++++++--- mm/filemap.c | 4 +- mm/huge_memory.c | 45 +++-- mm/memcontrol.c | 79 +++++++- mm/mlock.c | 63 ++----- mm/mmzone.c | 1 + mm/page_alloc.c | 1 - mm/page_idle.c | 4 - mm/rmap.c | 11 +- mm/swap.c | 208 ++++++++------------- mm/vmscan.c | 207 ++++++++++---------- mm/workingset.c | 2 - 21 files changed, 530 insertions(+), 372 deletions(-)
Hi Andrew,
With all patches are acked-by Hugh and Johannes, and full testing from LKP, is this patchset ready for more testing on linux-next? or anything still need be improved?
Thanks
Alex
Quoted 52 lines · Alex Shi
This version rebase on next/master 20201104, with much of Johannes's Acks and some changes according to Johannes comments. And add a new patch v21-0006-mm-rmap-stop-store-reordering-issue-on-page-mapp.patch to support v21-0007.
This patchset followed 2 memcg
VM_WARN_ON_ONCE_PAGEpatches which were added to -mm tree yesterday.Many thanks for line by line review by Hugh Dickins, Alexander Duyck and Johannes Weiner.
So now this patchset includes 3 parts:
1, some code cleanup and minimum optimization as a preparation. 2, use TestCleanPageLRU as page isolation's precondition. 3, replace per nodelru_lockwith per memcg per nodelru_lock.Current
lru_lockis one for each of node,pgdat->lru_lock, that guard for lru lists, but now we had moved the lru lists into memcg for long time. Still using per nodelru_lockis clearly unscalable, pages on each of memcgs have to compete each others for a wholelru_lock. This patchset try to use per lruvec/memcglru_lockto repleace per node lru lock to guard lru lists, make it scalable for memcgs and get performance gain.Currently
lru_lockstill guards both lru list and page's lru bit, that's ok. but if we want to use specific lruvec lock on the page, we need to pin down the page's lruvec/memcg during locking. Just taking lruvec lock first may be undermined by the page's memcg charge/migration. To fix this problem, we could take out the page's lru bit clear and use it as pin down action to block the memcg changes. That's the reason for new atomic func TestClearPageLRU. So now isolating a page need both actions: TestClearPageLRU and hold thelru_lock.The typical usage of this is
isolate_migratepages_block()incompaction.cwe have to take lru bit before lru lock, that serialized the page isolation in memcg page charge/migration which will change page's lruvec and newlru_lockin it.The above solution suggested by Johannes Weiner, and based on his new memcg charge path, then have this patchset. (Hugh Dickins tested and contributed much code from compaction fix to general code polish, thanks a lot!).
Daniel Jordan's testing show 62% improvement on modified readtwice case on his 2P * 10 core * 2 HT broadwell box on v18, which has no much different with this v20.
https://lore.kernel.org/lkml/20200915165807.kpp7uhiw7l3loofu-S51bK0XF4qpuJJETbFA3a0B3C2bhBk7L0E9HWUfgJXw@public.gmane.org/Thanks Hugh Dickins and Konstantin Khlebnikov, they both brought this idea 8 years ago, and others who give comments as well: Daniel Jordan, Mel Gorman, Shakeel Butt, Matthew Wilcox, Alexander Duyck etc.
Thanks for Testing support from Intel 0day and Rong Chen, Fengguang Wu, and Yun Wang. Hugh Dickins also shared his kbuild-swap case. Thanks!
Andrew Morton
maintainerQuoted 4 lines · On Thu, 5 Nov 2020 16:55:30 +0800 Alex Shi…
This version rebase on next/master 20201104, with much of Johannes's Acks and some changes according to Johannes comments. And add a new patch v21-0006-mm-rmap-stop-store-reordering-issue-on-page-mapp.patch to support v21-0007.
I assume the consensus on this series is 'not yet"?
Also, did
https://lkml.kernel.org/r/0000000000000340a105b49441d3-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org get resolved?
Quoted 8 lines · Andrew Morton
Quoted 4 lines · On Thu, 5 Nov 2020 16:55:30 +0800 Alex Shi…
This version rebase on next/master 20201104, with much of Johannes's Acks and some changes according to Johannes comments. And add a new patch v21-0006-mm-rmap-stop-store-reordering-issue-on-page-mapp.patch to support v21-0007.
I assume the consensus on this series is 'not yet"?
Speaking for my part in the consensus: I don't share that assumption, the series by now is well-baked and well reviewed by enough people over more than enough versions, has been completely untroublesome since it entered mmotm/linux-next a month ago, not even any performance bleats from 0day, and has nothing to gain from any further delay.
I think it was my fault that v20 didn't get into 5.10: I'd said "not yet" when you first tried a part of v19 or earlier in mmotm, and by the time I'd completed review it was too late in the cycle; Johannes and Vlastimil have gone over it since then, and I'd be glad to see it go ahead into 5.11 very soon. Silence on v21 meaning that it's good.
Various of us have improvements or cleanups in mind or in private tree, but nothing to hold back what's already there.
Quoted 4 lines
Also, did
https://lkml.kernel.org/r/0000000000000340a105b49441d3-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org get resolved?
Alex found enough precedents for that, before inclusion of his series, so it should not discourage from moving his series forward. I have ignored that syzreport until now, but will take a quick try at the repro now, to see if I'm inspired - probably not, but we'll see.
Hugh
Andrew Morton
maintainerQuoted 11 lines · On Mon, 14 Dec 2020 18:16:34 -0800 (PST) Hugh Dickins…
Quoted 8 lines · Andrew Morton
Quoted 4 lines · On Thu, 5 Nov 2020 16:55:30 +0800 Alex Shi…
This version rebase on next/master 20201104, with much of Johannes's Acks and some changes according to Johannes comments. And add a new patch v21-0006-mm-rmap-stop-store-reordering-issue-on-page-mapp.patch to support v21-0007.
I assume the consensus on this series is 'not yet"?
Speaking for my part in the consensus: I don't share that assumption,
OK, thanks, I'll include it in patch-bomb #2, Tues or Weds.
Quoted 10 lines · Alex Shi
This version rebase on next/master 20201104, with much of Johannes's Acks and some changes according to Johannes comments. And add a new patch v21-0006-mm-rmap-stop-store-reordering-issue-on-page-mapp.patch to support v21-0007.
This patchset followed 2 memcg
VM_WARN_ON_ONCE_PAGEpatches which were added to -mm tree yesterday.Many thanks for line by line review by Hugh Dickins, Alexander Duyck and Johannes Weiner.
Given the troublesome history of this patchset, and had been put into linux-next recently, as well as it touched both THP and mlock. Is it a good idea to suspect this patchset introducing some races and a spontaneous crash with some mlock memory presume?
[10392.154328][T23803] huge_memory: total_mapcount: 5, page_count(): 6 [10392.154835][T23803] page:00000000eb7725ad refcount:6 mapcount:0 mapping:0000000000000000 index:0x7fff72a0 pfn:0x20023760 [10392.154865][T23803] head:00000000eb7725ad order:5 compound_mapcount:0 compound_pincount:0 [10392.154889][T23803] anon flags: 0x87fff800009000d(locked|uptodate|dirty|head|swapbacked) [10392.154908][T23803] raw: 087fff800009000d 5deadbeef0000100 5deadbeef0000122 c0002016ff5e0849 [10392.154933][T23803] raw: 000000007fff72a0 0000000000000000 00000006ffffffff c0002014eb676000 [10392.154965][T23803] page dumped because: total_mapcount(head) > 0 [10392.154987][T23803] pages's memcg:c0002014eb676000 [10392.155023][T23803] ------------[ cut here ]------------ [10392.155042][T23803] kernel BUG at mm/huge_memory.c:2767! [10392.155064][T23803] Oops: Exception in kernel mode, sig: 5 [#1] [10392.155084][T23803] LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=256 NUMA PowerNV [10392.155114][T23803] Modules linked in: loop kvm_hv kvm ip_tables x_tables sd_mod bnx2x ahci tg3 libahci mdio libphy libata firmware_class dm_mirror dm_region_hash dm_log dm_mod [10392.155185][T23803] CPU: 44 PID: 23803 Comm: ranbug Not tainted 5.11.0-rc2-next-20210105 #2 [10392.155217][T23803] NIP: c0000000003b5218 LR: c0000000003b5214 CTR: 0000000000000000 [10392.155247][T23803] REGS: c00000001a8d6ee0 TRAP: 0700 Not tainted (5.11.0-rc2-next-20210105) [10392.155279][T23803] MSR: 9000000002823033 <SF,HV,VEC,VSX,FP,ME,IR,DR,RI,LE> CR: 28422222 XER: 00000000 [10392.155314][T23803] CFAR: c0000000003135ac IRQMASK: 1 [10392.155314][T23803] GPR00: c0000000003b5214 c00000001a8d7180 c000000007f70b00 000000000000001e [10392.155314][T23803] GPR04: c000000000eacd38 0000000000000004 0000000000000027 c000001ffe8a7218 [10392.155314][T23803] GPR08: 0000000000000023 0000000000000000 0000000000000000 c000000007eacfc8 [10392.155314][T23803] GPR12: 0000000000002000 c000001ffffcda00 0000000000000000 0000000000000001 [10392.155314][T23803] GPR16: c00c0008008dd808 0000000000040000 0000000000000000 0000000000000020 [10392.155314][T23803] GPR20: c00c0008008dd800 0000000000000020 0000000000000006 0000000000000001 [10392.155314][T23803] GPR24: 0000000000000005 ffffffffffffffff c0002016ff5e0848 0000000000000000 [10392.155314][T23803] GPR28: c0002014eb676e60 c00c0008008dd800 c00000001a8d73a8 c00c0008008dd800 [10392.155533][T23803] NIP [c0000000003b5218] split_huge_page_to_list+0xa38/0xa40 [10392.155558][T23803] LR [c0000000003b5214] split_huge_page_to_list+0xa34/0xa40 [10392.155579][T23803] Call Trace: [10392.155595][T23803] [c00000001a8d7180] [c0000000003b5214] split_huge_page_to_list+0xa34/0xa40 (unreliable) [10392.155630][T23803] [c00000001a8d7270] [c0000000002dd378] shrink_page_list+0x1568/0x1b00 shrink_page_list at mm/vmscan.c:1251 (discriminator 1) [10392.155655][T23803] [c00000001a8d7380] [c0000000002df798] shrink_inactive_list+0x228/0x5e0 [10392.155678][T23803] [c00000001a8d7450] [c0000000002e0858] shrink_lruvec+0x2b8/0x6f0 shrink_lruvec at mm/vmscan.c:2462 [10392.155710][T23803] [c00000001a8d7590] [c0000000002e0fd8] shrink_node+0x348/0x970 [10392.155742][T23803] [c00000001a8d7660] [c0000000002e1728] do_try_to_free_pages+0x128/0x560 [10392.155765][T23803] [c00000001a8d7710] [c0000000002e3b78] try_to_free_pages+0x198/0x500 [10392.155780][T23803] [c00000001a8d77e0] [c000000000356f5c] __alloc_pages_slowpath.constprop.112+0x64c/0x1380 [10392.155795][T23803] [c00000001a8d79c0] [c000000000358170] __alloc_pages_nodemask+0x4e0/0x590 [10392.155830][T23803] [c00000001a8d7a50] [c000000000381fb8] alloc_pages_vma+0xb8/0x340 [10392.155854][T23803] [c00000001a8d7ac0] [c000000000324fe8] handle_mm_fault+0xf38/0x1bd0 [10392.155887][T23803] [c00000001a8d7ba0] [c000000000316cd4] __get_user_pages+0x434/0x7d0 [10392.155920][T23803] [c00000001a8d7cb0] [c0000000003197d0] __mm_populate+0xe0/0x290 __mm_populate at mm/gup.c:1459 [10392.155952][T23803] [c00000001a8d7d20] [c00000000032d5a0] do_mlock+0x180/0x360 do_mlock at mm/mlock.c:688 [10392.155975][T23803] [c00000001a8d7d90] [c00000000032d954] sys_mlock+0x24/0x40 [10392.155999][T23803] [c00000001a8d7db0] [c00000000002f510] system_call_exception+0x170/0x280 [10392.156032][T23803] [c00000001a8d7e10] [c00000000000d7c8] system_call_common+0xe8/0x218 [10392.156065][T23803] Instruction dump: [10392.156082][T23803] e93d0008 71290001 41820014 7fe3fb78 38800000 4bf5e36d 60000000 3c82f8b7 [10392.156121][T23803] 7fa3eb78 38846b70 4bf5e359 60000000 <0fe00000> 60000000 3c4c07bc 3842b8e0 [10392.156160][T23803] ---[ end trace 2e3423677d4f91f3 ]--- [10392.312793][T23803] [10393.312808][T23803] Kernel panic - not syncing: Fatal exception [10394.723608][T23803] ---[ end Kernel panic - not syncing: Fatal exception ]---
Quoted 94 lines
So now this patchset includes 3 parts:
1, some code cleanup and minimum optimization as a preparation. 2, use TestCleanPageLRU as page isolation's precondition. 3, replace per nodelru_lockwith per memcg per nodelru_lock.Current
lru_lockis one for each of node,pgdat->lru_lock, that guard for lru lists, but now we had moved the lru lists into memcg for long time. Still using per nodelru_lockis clearly unscalable, pages on each of memcgs have to compete each others for a wholelru_lock. This patchset try to use per lruvec/memcglru_lockto repleace per node lru lock to guard lru lists, make it scalable for memcgs and get performance gain.Currently
lru_lockstill guards both lru list and page's lru bit, that's ok. but if we want to use specific lruvec lock on the page, we need to pin down the page's lruvec/memcg during locking. Just taking lruvec lock first may be undermined by the page's memcg charge/migration. To fix this problem, we could take out the page's lru bit clear and use it as pin down action to block the memcg changes. That's the reason for new atomic func TestClearPageLRU. So now isolating a page need both actions: TestClearPageLRU and hold thelru_lock.The typical usage of this is
isolate_migratepages_block()incompaction.cwe have to take lru bit before lru lock, that serialized the page isolation in memcg page charge/migration which will change page's lruvec and newlru_lockin it.The above solution suggested by Johannes Weiner, and based on his new memcg charge path, then have this patchset. (Hugh Dickins tested and contributed much
code from compaction fix to general code polish, thanks a lot!).Daniel Jordan's testing show 62% improvement on modified readtwice case on his 2P * 10 core * 2 HT broadwell box on v18, which has no much different with this v20.
https://lore.kernel.org/lkml/20200915165807.kpp7uhiw7l3loofu-S51bK0XF4qpuJJETbFA3a0B3C2bhBk7L0E9HWUfgJXw@public.gmane.org/Thanks Hugh Dickins and Konstantin Khlebnikov, they both brought this idea 8 years ago, and others who give comments as well: Daniel Jordan, Mel Gorman, Shakeel Butt, Matthew Wilcox, Alexander Duyck etc.
Thanks for Testing support from Intel 0day and Rong Chen, Fengguang Wu, and Yun Wang. Hugh Dickins also shared his kbuild-swap case. Thanks!
Alex Shi (16):
mm/thp: movelru_add_page_tailfunc tohuge_memory.cmm/thp: use head for head page inlru_add_page_tailmm/thp: Simplifylru_add_page_tail()
mm/thp: narrow lru locking
mm/vmscan: remove unnecessary lruvec adding mm/rmap: stop store reordering issue onpage->mappingmm/memcg: add debug checking inlock_page_memcgmm/swap.c: fold vm event PGROTATED intopagevec_move_tail_fnmm/lru: move lock intolru_note_cost
mm/vmscan: remove lruvec reget inmove_pages_to_lrumm/mlock: removelru_lockon TestClearPageMlocked mm/mlock: remove__munlock_isolate_lru_pagemm/lru: introduce TestClearPageLRU
mm/compaction: do page isolation first in compactionmm/swap.c: serialize memcg changes inpagevec_lru_move_fnmm/lru: replace pgdatlru_lockwith lruvec lockAlexander Duyck (1):
mm/lru: introduce therelock_page_lruvecfunctionHugh Dickins (2):
mm:page_idle_get_page()does not needlru_lock
mm/lru: revise the comments oflru_lockDocumentation/admin-guide/cgroup-v1/memcg_test.rst | 15 +- Documentation/admin-guide/cgroup-v1/memory.rst | 21 +-- Documentation/trace/events-kmem.rst | 2 +- Documentation/vm/unevictable-lru.rst | 22 +-- include/linux/memcontrol.h | 110 +++++++++++ include/linux/mm_types.h | 2 +- include/linux/mmzone.h | 6 +- include/linux/page-flags.h | 1 + include/linux/swap.h | 4 +- mm/compaction.c | 94 +++++++--- mm/filemap.c | 4 +- mm/huge_memory.c | 45 +++-- mm/memcontrol.c | 79 +++++++- mm/mlock.c | 63 ++----- mm/mmzone.c | 1 + mm/page_alloc.c | 1 - mm/page_idle.c | 4 - mm/rmap.c | 11 +- mm/swap.c | 208 ++++++++---------- --- mm/vmscan.c | 207 ++++++++++---------- mm/workingset.c | 2 - 21 files changed, 530 insertions(+), 372 deletions(-)
Quoted 10 lines · Qian Cai
Quoted 8 lines · Alex Shi
This version rebase on next/master 20201104, with much of Johannes's Acks and some changes according to Johannes comments. And add a new patch v21-0006-mm-rmap-stop-store-reordering-issue-on-page-mapp.patch to support v21-0007.
This patchset followed 2 memcg
VM_WARN_ON_ONCE_PAGEpatches which were added to -mm tree yesterday.
Quoted 7 lines
Quoted 2 lines
Many thanks for line by line review by Hugh Dickins, Alexander Duyck and Johannes Weiner.
Given the troublesome history of this patchset, and had been put into linux-next recently, as well as it touched both THP and mlock. Is it a good idea to suspect this patchset introducing some races and a spontaneous crash with some mlock memory presume?
This has already been merged into the linus tree. Were you able to get a similar crash on the latest upstream kernel as well?
Quoted 20 lines · Shakeel Butt
Quoted 12 lines · Qian Cai
Quoted 10 lines · Alex Shi
This version rebase on next/master 20201104, with much of Johannes's Acks and some changes according to Johannes comments. And add a new patch v21-0006-mm-rmap-stop-store-reordering-issue-on-page-mapp.patch to support v21-0007.
This patchset followed 2 memcg
VM_WARN_ON_ONCE_PAGEpatches which were added to -mm tree yesterday.Many thanks for line by line review by Hugh Dickins, Alexander Duyck and Johannes Weiner.
Quoted 6 lines
Given the troublesome history of this patchset, and had been put into linux- next
recently, as well as it touched both THP and mlock. Is it a good idea to suspect
this patchset introducing some races and a spontaneous crash with some mlock memory presume?
Quoted 2 lines
This has already been merged into the linus tree. Were you able to get a similar crash on the latest upstream kernel as well?
No, I seldom test the mainline those days. Before the vacations, I have tested linux-next up to something like 12/10 which did not include this patchset IIRC and never saw any crash like this. I am still trying to figure out how to reproduce it fast, so I can try a revert to confirm.
Quoted 21 lines · Qian Cai
Quoted 20 lines · Shakeel Butt
Quoted 18 lines · Qian Cai
Quoted 10 lines · Alex Shi
This version rebase on next/master 20201104, with much of Johannes's Acks and some changes according to Johannes comments. And add a new patch v21-0006-mm-rmap-stop-store-reordering-issue-on-page-mapp.patch to support v21-0007.
This patchset followed 2 memcg
VM_WARN_ON_ONCE_PAGEpatches which were added to -mm tree yesterday.Many thanks for line by line review by Hugh Dickins, Alexander Duyck and Johannes Weiner.
Given the troublesome history of this patchset, and had been put into linux- next
recently, as well as it touched both THP and mlock. Is it a good idea to suspect
this patchset introducing some races and a spontaneous crash with some mlock memory presume?
Quoted 7 lines
Quoted 2 lines
This has already been merged into the linus tree. Were you able to get a similar crash on the latest upstream kernel as well?
No, I seldom test the mainline those days. Before the vacations, I have tested linux-next up to something like 12/10 which did not include this patchset IIRC and never saw any crash like this. I am still trying to figure out how to reproduce it fast, so I can try a revert to confirm.
This patchset went into mmotm 2020-11-16-16-23, so probably linux-next on 2020-11-17: you'll have had three trouble-free weeks testing with it in, so it's not a likely suspect. I haven't looked yet at your report, to think of a more likely suspect: will do.
Hugh
Quoted 4 lines · Hugh Dickins
This patchset went into mmotm 2020-11-16-16-23, so probably linux-next on 2020-11-17: you'll have had three trouble-free weeks testing with it in, so it's not a likely suspect. I haven't looked yet at your report, to think of a more likely suspect: will do.
Probably my memory was bad then. Unfortunately, I had 2 weeks holidays before the Thanksgiving as well. I have tried a few times so far and only been able to reproduce once. Looks nasty...
Quoted 9 lines · Qian Cai
Quoted 4 lines · Hugh Dickins
This patchset went into mmotm 2020-11-16-16-23, so probably linux-next on 2020-11-17: you'll have had three trouble-free weeks testing with it in, so it's not a likely suspect. I haven't looked yet at your report, to think of a more likely suspect: will do.
Probably my memory was bad then. Unfortunately, I had 2 weeks holidays before the Thanksgiving as well. I have tried a few times so far and only been able to reproduce once. Looks nasty...
I have not found a likely suspect.
What it smells like is a defect in cloning anon_vma during fork, such that mappings of the THP can get added even after all that could be found were unmapped (tree lookup ordering should prevent that). But I've not seen any recent change there.
It would be very easily fixed by deleting the whole BUG() block, which is only there as a sanity check for developers: but we would not want to delete it without understanding why it has gone wrong (and would also have to reconsider two related VM_BUG_ON_PAGEs).
It is possible that b6769834aac1 ("mm/thp: narrow lru locking") of this patchset has changed the timing and made a pre-existing bug more likely in some situations: it used to hold an lru_lock before that BUG() on total_mapcount(), and now does not; but that's not a lock which should be relevant to the check.
When you get more info (or not), please repost the bugstack in a new email thread: this thread is not really useful for pursuing it.
Hugh