Skip to contentSkip to conversation

incoming

PATCH New Andrew Morton 2 participants 42 messages Dec 15, 2020

Subsystems (15)
lore
Share & tools
lore

Andrew Morton

author maintainer
#0 lore raw

- more MM work: a memcg scalability improvememt

19 patches, based on 148842c98a24e508aecb929718818fbf4c2a6ff3.

Subsystems affected by this patch series:

Alex Shi <[email protected]>:
Patch series "per memcg lru lock", v21:
mm/thp: move lru_add_page_tail() to huge_memory.c mm/thp: use head for head page in lru_add_page_tail() mm/thp: simplify lru_add_page_tail()
mm/thp: narrow lru locking
mm/vmscan: remove unnecessary lruvec adding mm/rmap: stop store reordering issue on page->mapping

Hugh Dickins <[email protected]>:
mm: page_idle_get_page() does not need lru_lock

Alex Shi <[email protected]>:
mm/memcg: add debug checking in lock_page_memcg mm/swap.c: fold vm event PGROTATED into pagevec_move_tail_fn mm/lru: move lock into lru_note_cost
mm/vmscan: remove lruvec reget in move_pages_to_lru mm/mlock: remove lru_lock on TestClearPageMlocked mm/mlock: remove __munlock_isolate_lru_page() mm/lru: introduce TestClearPageLRU() mm/compaction: do page isolation first in compaction mm/swap.c: serialize memcg changes in pagevec_lru_move_fn mm/lru: replace pgdat lru_lock with lruvec lock

Alexander Duyck <[email protected]>:
mm/lru: introduce relock_page_lruvec()

Hugh Dickins <[email protected]>:
mm/lru: revise the comments of lru_lock

 Documentation/admin-guide/cgroup-v1/memcg_test.rst |   15 -
 Documentation/admin-guide/cgroup-v1/memory.rst     |   23 -
 Documentation/trace/events-kmem.rst                |    2 
 Documentation/vm/unevictable-lru.rst               |   22 -
 include/linux/memcontrol.h                         |  110 +++++++
 include/linux/mm_types.h                           |    2 
 include/linux/mmzone.h                             |    6 
 include/linux/page-flags.h                         |    1 
 include/linux/swap.h                               |    4 
 mm/compaction.c                                    |   98 ++++---
 mm/filemap.c                                       |    4 
 mm/huge_memory.c                                   |  109 ++++---
 mm/memcontrol.c                                    |   84 +++++-
 mm/mlock.c                                         |   93 ++----
 mm/mmzone.c                                        |    1 
 mm/page_alloc.c                                    |    1 
 mm/page_idle.c                                     |    4 
 mm/rmap.c                                          |   12 
 mm/swap.c                                          |  292 ++++++++-------------
 mm/vmscan.c                                        |  239 ++++++++---------
 mm/workingset.c                                    |    2 
 21 files changed, 644 insertions(+), 480 deletions(-)
author maintainer
#1 lore raw

PATCH 1/19 mm/thp: move lru_add_page_tail() to huge_memory.c

From: Alex Shi <[email protected]>
Subject: mm/thp: move lru_add_page_tail() to huge_memory.c

Patch series "per memcg lru lock", v21.

This patchset includes 3 parts:

1, some code cleanup and minimum optimization as a preparation. 2, use TestCleanPageLRU as page isolation's precondition. 3, replace per node lru_lock with per memcg per node lru_lock.

Current lru_lock is one for each of node, pgdat->lru_lock, that guard for lru lists, but now we had moved the lru lists into memcg for long time. Still using per node lru_lock is clearly unscalable, pages on each of memcgs have to compete each others for a whole lru_lock. This patchset try to use per lruvec/memcg lru_lock to repleace per node lru lock to guard lru lists, make it scalable for memcgs and get performance gain.

Currently lru_lock still guards both lru list and page's lru bit, that's ok. but if we want to use specific lruvec lock on the page, we need to pin down the page's lruvec/memcg during locking. Just taking lruvec lock first may be undermined by the page's memcg charge/migration. To fix this problem, we could take out the page's lru bit clear and use it as pin down action to block the memcg changes. That's the reason for new atomic func TestClearPageLRU. So now isolating a page need both actions:
TestClearPageLRU and hold the lru_lock.

The typical usage of this is isolate_migratepages_block() in compaction.c we have to take lru bit before lru lock, that serialized the page isolation in memcg page charge/migration which will change page's lruvec and new lru_lock in it.

The above solution suggested by Johannes Weiner, and based on his new memcg charge path, then have this patchset. (Hugh Dickins tested and contributed much code from compaction fix to general code polish, thanks a lot!).

Daniel Jordan's testing show 62% improvement on modified readtwice case on his 2P * 10 core * 2 HT broadwell box on v18, which has no much different with this v20.
https://lore.kernel.org/lkml/[email protected]/

Thanks to Hugh Dickins and Konstantin Khlebnikov, they both brought this idea 8 years ago, and others who gave comments as well: Daniel Jordan, Mel Gorman, Shakeel Butt, Matthew Wilcox, Alexander Duyck etc.

Thanks for Testing support from Intel 0day and Rong Chen, Fengguang Wu, and Yun Wang. Hugh Dickins also shared his kbuild-swap case.

This patch (of 19):

lru_add_page_tail() is only used in huge_memory.c, defining it in other file with a CONFIG_TRANSPARENT_HUGEPAGE macro restrict just looks weird.

Let's move it THP. And make it static as Hugh Dickins suggested.

Cc Matthew Wilcox <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Wei Yang <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Yang Shi <[email protected]>

---

 include/linux/swap.h |    2 --
 mm/huge_memory.c     |   30 ++++++++++++++++++++++++++++++
 mm/swap.c            |   33 ---------------------------------
 3 files changed, 30 insertions(+), 35 deletions(-)
include/linux/swap.h renamed from include/linux/swap.h~mm-thp-move-lru_add_page_tail-func-to-huge_memoryc +0 -2
@@ -338,8 +338,6 @@ extern void lru_note_cost(struct lruvec
 			  unsigned int nr_pages);
 extern void lru_note_cost_page(struct page *);
 extern void lru_cache_add(struct page *);
-extern void lru_add_page_tail(struct page *page, struct page *page_tail,
-			 struct lruvec *lruvec, struct list_head *head);
 extern void mark_page_accessed(struct page *);
 extern void lru_add_drain(void);
 extern void lru_add_drain_cpu(int cpu);
mm/huge_memory.c renamed from mm/huge_memory.c~mm-thp-move-lru_add_page_tail-func-to-huge_memoryc +30 -0
@@ -2359,6 +2359,36 @@ static void remap_page(struct page *page
 	}
 }
 
+static void lru_add_page_tail(struct page *page, struct page *page_tail,
+		struct lruvec *lruvec, struct list_head *list)
+{
+	VM_BUG_ON_PAGE(!PageHead(page), page);
+	VM_BUG_ON_PAGE(PageCompound(page_tail), page);
+	VM_BUG_ON_PAGE(PageLRU(page_tail), page);
+	lockdep_assert_held(&lruvec_pgdat(lruvec)->lru_lock);
+
+	if (!list)
+		SetPageLRU(page_tail);
+
+	if (likely(PageLRU(page)))
+		list_add_tail(&page_tail->lru, &page->lru);
+	else if (list) {
+		/* page reclaim is reclaiming a huge page */
+		get_page(page_tail);
+		list_add_tail(&page_tail->lru, list);
+	} else {
+		/*
+		 * Head page has not yet been counted, as an hpage,
+		 * so we must account for each subpage individually.
+		 *
+		 * Put page_tail on the list at the correct position
+		 * so they all end up in order.
+		 */
+		add_page_to_lru_list_tail(page_tail, lruvec,
+					  page_lru(page_tail));
+	}
+}
+
 static void __split_huge_page_tail(struct page *head, int tail,
 		struct lruvec *lruvec, struct list_head *list)
 {
mm/swap.c renamed from mm/swap.c~mm-thp-move-lru_add_page_tail-func-to-huge_memoryc +0 -33
@@ -977,39 +977,6 @@ void __pagevec_release(struct pagevec *p
 }
 EXPORT_SYMBOL(__pagevec_release);
 
-#ifdef CONFIG_TRANSPARENT_HUGEPAGE
-/* used by __split_huge_page_refcount() */
-void lru_add_page_tail(struct page *page, struct page *page_tail,
-		       struct lruvec *lruvec, struct list_head *list)
-{
-	VM_BUG_ON_PAGE(!PageHead(page), page);
-	VM_BUG_ON_PAGE(PageCompound(page_tail), page);
-	VM_BUG_ON_PAGE(PageLRU(page_tail), page);
-	lockdep_assert_held(&lruvec_pgdat(lruvec)->lru_lock);
-
-	if (!list)
-		SetPageLRU(page_tail);
-
-	if (likely(PageLRU(page)))
-		list_add_tail(&page_tail->lru, &page->lru);
-	else if (list) {
-		/* page reclaim is reclaiming a huge page */
-		get_page(page_tail);
-		list_add_tail(&page_tail->lru, list);
-	} else {
-		/*
-		 * Head page has not yet been counted, as an hpage,
-		 * so we must account for each subpage individually.
-		 *
-		 * Put page_tail on the list at the correct position
-		 * so they all end up in order.
-		 */
-		add_page_to_lru_list_tail(page_tail, lruvec,
-					  page_lru(page_tail));
-	}
-}
-#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
-
 static void __pagevec_lru_add_fn(struct page *page, struct lruvec *lruvec,
 				 void *arg)
 {

_

author maintainer
#2 lore raw

PATCH 1/19 mm/thp: move lru_add_page_tail() to huge_memory.c

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/thp: move lru_add_page_tail() to huge_memory.c

Patch series "per memcg lru lock", v21.

This patchset includes 3 parts:

1, some code cleanup and minimum optimization as a preparation. 2, use TestCleanPageLRU as page isolation's precondition. 3, replace per node lru_lock with per memcg per node lru_lock.

Current lru_lock is one for each of node, pgdat->lru_lock, that guard for lru lists, but now we had moved the lru lists into memcg for long time. Still using per node lru_lock is clearly unscalable, pages on each of memcgs have to compete each others for a whole lru_lock. This patchset try to use per lruvec/memcg lru_lock to repleace per node lru lock to guard lru lists, make it scalable for memcgs and get performance gain.

Currently lru_lock still guards both lru list and page's lru bit, that's ok. but if we want to use specific lruvec lock on the page, we need to pin down the page's lruvec/memcg during locking. Just taking lruvec lock first may be undermined by the page's memcg charge/migration. To fix this problem, we could take out the page's lru bit clear and use it as pin down action to block the memcg changes. That's the reason for new atomic func TestClearPageLRU. So now isolating a page need both actions:
TestClearPageLRU and hold the lru_lock.

The typical usage of this is isolate_migratepages_block() in compaction.c we have to take lru bit before lru lock, that serialized the page isolation in memcg page charge/migration which will change page's lruvec and new lru_lock in it.

The above solution suggested by Johannes Weiner, and based on his new memcg charge path, then have this patchset. (Hugh Dickins tested and contributed much code from compaction fix to general code polish, thanks a lot!).

Daniel Jordan's testing show 62% improvement on modified readtwice case on his 2P * 10 core * 2 HT broadwell box on v18, which has no much different with this v20.
https://lore.kernel.org/lkml/[email protected]/

Thanks to Hugh Dickins and Konstantin Khlebnikov, they both brought this idea 8 years ago, and others who gave comments as well: Daniel Jordan, Mel Gorman, Shakeel Butt, Matthew Wilcox, Alexander Duyck etc.

Thanks for Testing support from Intel 0day and Rong Chen, Fengguang Wu, and Yun Wang. Hugh Dickins also shared his kbuild-swap case.

This patch (of 19):

lru_add_page_tail() is only used in huge_memory.c, defining it in other file with a CONFIG_TRANSPARENT_HUGEPAGE macro restrict just looks weird.

Let's move it THP. And make it static as Hugh Dickins suggested.

Cc Matthew Wilcox <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Wei Yang <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Yang Shi <[email protected]>

---

 include/linux/swap.h |    2 --
 mm/huge_memory.c     |   30 ++++++++++++++++++++++++++++++
 mm/swap.c            |   33 ---------------------------------
 3 files changed, 30 insertions(+), 35 deletions(-)
include/linux/swap.h renamed from include/linux/swap.h~mm-thp-move-lru_add_page_tail-func-to-huge_memoryc +0 -2
@@ -338,8 +338,6 @@ extern void lru_note_cost(struct lruvec
 			  unsigned int nr_pages);
 extern void lru_note_cost_page(struct page *);
 extern void lru_cache_add(struct page *);
-extern void lru_add_page_tail(struct page *page, struct page *page_tail,
-			 struct lruvec *lruvec, struct list_head *head);
 extern void mark_page_accessed(struct page *);
 extern void lru_add_drain(void);
 extern void lru_add_drain_cpu(int cpu);
mm/huge_memory.c renamed from mm/huge_memory.c~mm-thp-move-lru_add_page_tail-func-to-huge_memoryc +30 -0
@@ -2359,6 +2359,36 @@ static void remap_page(struct page *page
 	}
 }
 
+static void lru_add_page_tail(struct page *page, struct page *page_tail,
+		struct lruvec *lruvec, struct list_head *list)
+{
+	VM_BUG_ON_PAGE(!PageHead(page), page);
+	VM_BUG_ON_PAGE(PageCompound(page_tail), page);
+	VM_BUG_ON_PAGE(PageLRU(page_tail), page);
+	lockdep_assert_held(&lruvec_pgdat(lruvec)->lru_lock);
+
+	if (!list)
+		SetPageLRU(page_tail);
+
+	if (likely(PageLRU(page)))
+		list_add_tail(&page_tail->lru, &page->lru);
+	else if (list) {
+		/* page reclaim is reclaiming a huge page */
+		get_page(page_tail);
+		list_add_tail(&page_tail->lru, list);
+	} else {
+		/*
+		 * Head page has not yet been counted, as an hpage,
+		 * so we must account for each subpage individually.
+		 *
+		 * Put page_tail on the list at the correct position
+		 * so they all end up in order.
+		 */
+		add_page_to_lru_list_tail(page_tail, lruvec,
+					  page_lru(page_tail));
+	}
+}
+
 static void __split_huge_page_tail(struct page *head, int tail,
 		struct lruvec *lruvec, struct list_head *list)
 {
mm/swap.c renamed from mm/swap.c~mm-thp-move-lru_add_page_tail-func-to-huge_memoryc +0 -33
@@ -977,39 +977,6 @@ void __pagevec_release(struct pagevec *p
 }
 EXPORT_SYMBOL(__pagevec_release);
 
-#ifdef CONFIG_TRANSPARENT_HUGEPAGE
-/* used by __split_huge_page_refcount() */
-void lru_add_page_tail(struct page *page, struct page *page_tail,
-		       struct lruvec *lruvec, struct list_head *list)
-{
-	VM_BUG_ON_PAGE(!PageHead(page), page);
-	VM_BUG_ON_PAGE(PageCompound(page_tail), page);
-	VM_BUG_ON_PAGE(PageLRU(page_tail), page);
-	lockdep_assert_held(&lruvec_pgdat(lruvec)->lru_lock);
-
-	if (!list)
-		SetPageLRU(page_tail);
-
-	if (likely(PageLRU(page)))
-		list_add_tail(&page_tail->lru, &page->lru);
-	else if (list) {
-		/* page reclaim is reclaiming a huge page */
-		get_page(page_tail);
-		list_add_tail(&page_tail->lru, list);
-	} else {
-		/*
-		 * Head page has not yet been counted, as an hpage,
-		 * so we must account for each subpage individually.
-		 *
-		 * Put page_tail on the list at the correct position
-		 * so they all end up in order.
-		 */
-		add_page_to_lru_list_tail(page_tail, lruvec,
-					  page_lru(page_tail));
-	}
-}
-#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
-
 static void __pagevec_lru_add_fn(struct page *page, struct lruvec *lruvec,
 				 void *arg)
 {

_

author maintainer
#3 lore raw

PATCH 2/19 mm/thp: use head for head page in lru_add_page_tail()

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/thp: use head for head page in lru_add_page_tail()

Since the first parameter is only used by head page, it's better to make it explicit.

Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 mm/huge_memory.c |   23 +++++++++++------------
 1 file changed, 11 insertions(+), 12 deletions(-)
mm/huge_memory.c renamed from mm/huge_memory.c~mm-thp-use-head-for-head-page-in-lru_add_page_tail +11 -12
@@ -2359,33 +2359,32 @@ static void remap_page(struct page *page
 	}
 }
 
-static void lru_add_page_tail(struct page *page, struct page *page_tail,
+static void lru_add_page_tail(struct page *head, struct page *tail,
 		struct lruvec *lruvec, struct list_head *list)
 {
-	VM_BUG_ON_PAGE(!PageHead(page), page);
-	VM_BUG_ON_PAGE(PageCompound(page_tail), page);
-	VM_BUG_ON_PAGE(PageLRU(page_tail), page);
+	VM_BUG_ON_PAGE(!PageHead(head), head);
+	VM_BUG_ON_PAGE(PageCompound(tail), head);
+	VM_BUG_ON_PAGE(PageLRU(tail), head);
 	lockdep_assert_held(&lruvec_pgdat(lruvec)->lru_lock);
 
 	if (!list)
-		SetPageLRU(page_tail);
+		SetPageLRU(tail);
 
-	if (likely(PageLRU(page)))
-		list_add_tail(&page_tail->lru, &page->lru);
+	if (likely(PageLRU(head)))
+		list_add_tail(&tail->lru, &head->lru);
 	else if (list) {
 		/* page reclaim is reclaiming a huge page */
-		get_page(page_tail);
-		list_add_tail(&page_tail->lru, list);
+		get_page(tail);
+		list_add_tail(&tail->lru, list);
 	} else {
 		/*
 		 * Head page has not yet been counted, as an hpage,
 		 * so we must account for each subpage individually.
 		 *
-		 * Put page_tail on the list at the correct position
+		 * Put tail on the list at the correct position
 		 * so they all end up in order.
 		 */
-		add_page_to_lru_list_tail(page_tail, lruvec,
-					  page_lru(page_tail));
+		add_page_to_lru_list_tail(tail, lruvec, page_lru(tail));
 	}
 }
 

_

author maintainer
#4 lore raw

PATCH 2/19 mm/thp: use head for head page in lru_add_page_tail()

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/thp: use head for head page in lru_add_page_tail()

Since the first parameter is only used by head page, it's better to make it explicit.

Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 mm/huge_memory.c |   23 +++++++++++------------
 1 file changed, 11 insertions(+), 12 deletions(-)
mm/huge_memory.c renamed from mm/huge_memory.c~mm-thp-use-head-for-head-page-in-lru_add_page_tail +11 -12
@@ -2359,33 +2359,32 @@ static void remap_page(struct page *page
 	}
 }
 
-static void lru_add_page_tail(struct page *page, struct page *page_tail,
+static void lru_add_page_tail(struct page *head, struct page *tail,
 		struct lruvec *lruvec, struct list_head *list)
 {
-	VM_BUG_ON_PAGE(!PageHead(page), page);
-	VM_BUG_ON_PAGE(PageCompound(page_tail), page);
-	VM_BUG_ON_PAGE(PageLRU(page_tail), page);
+	VM_BUG_ON_PAGE(!PageHead(head), head);
+	VM_BUG_ON_PAGE(PageCompound(tail), head);
+	VM_BUG_ON_PAGE(PageLRU(tail), head);
 	lockdep_assert_held(&lruvec_pgdat(lruvec)->lru_lock);
 
 	if (!list)
-		SetPageLRU(page_tail);
+		SetPageLRU(tail);
 
-	if (likely(PageLRU(page)))
-		list_add_tail(&page_tail->lru, &page->lru);
+	if (likely(PageLRU(head)))
+		list_add_tail(&tail->lru, &head->lru);
 	else if (list) {
 		/* page reclaim is reclaiming a huge page */
-		get_page(page_tail);
-		list_add_tail(&page_tail->lru, list);
+		get_page(tail);
+		list_add_tail(&tail->lru, list);
 	} else {
 		/*
 		 * Head page has not yet been counted, as an hpage,
 		 * so we must account for each subpage individually.
 		 *
-		 * Put page_tail on the list at the correct position
+		 * Put tail on the list at the correct position
 		 * so they all end up in order.
 		 */
-		add_page_to_lru_list_tail(page_tail, lruvec,
-					  page_lru(page_tail));
+		add_page_to_lru_list_tail(tail, lruvec, page_lru(tail));
 	}
 }
 

_

author maintainer
#5 lore raw

PATCH 3/19 mm/thp: simplify lru_add_page_tail()

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/thp: simplify lru_add_page_tail()

Simplify lru_add_page_tail(), there are actually only two cases possible:
split_huge_page_to_list(), with list supplied and head isolated from lru by its caller; or split_huge_page(), with NULL list and head on lru - because when head is racily isolated from lru, the isolator's reference will stop the split from getting any further than its page_ref_freeze().

So decide between the two cases by "list", but add VM_WARN_ON()s to verify that they match our lru expectations.

[Hugh Dickins: rewrite commit log]
Cc Johannes Weiner <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 mm/huge_memory.c |   20 ++++++--------------
 1 file changed, 6 insertions(+), 14 deletions(-)
mm/huge_memory.c renamed from mm/huge_memory.c~mm-thp-simplify-lru_add_page_tail +6 -14
@@ -2367,24 +2367,16 @@ static void lru_add_page_tail(struct pag
 	VM_BUG_ON_PAGE(PageLRU(tail), head);
 	lockdep_assert_held(&lruvec_pgdat(lruvec)->lru_lock);
 
-	if (!list)
-		SetPageLRU(tail);
-
-	if (likely(PageLRU(head)))
-		list_add_tail(&tail->lru, &head->lru);
-	else if (list) {
+	if (list) {
 		/* page reclaim is reclaiming a huge page */
+		VM_WARN_ON(PageLRU(head));
 		get_page(tail);
 		list_add_tail(&tail->lru, list);
 	} else {
-		/*
-		 * Head page has not yet been counted, as an hpage,
-		 * so we must account for each subpage individually.
-		 *
-		 * Put tail on the list at the correct position
-		 * so they all end up in order.
-		 */
-		add_page_to_lru_list_tail(tail, lruvec, page_lru(tail));
+		/* head is still on lru (and we have it frozen) */
+		VM_WARN_ON(!PageLRU(head));
+		SetPageLRU(tail);
+		list_add_tail(&tail->lru, &head->lru);
 	}
 }
 

_

author maintainer
#6 lore raw

PATCH 3/19 mm/thp: simplify lru_add_page_tail()

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/thp: simplify lru_add_page_tail()

Simplify lru_add_page_tail(), there are actually only two cases possible:
split_huge_page_to_list(), with list supplied and head isolated from lru by its caller; or split_huge_page(), with NULL list and head on lru - because when head is racily isolated from lru, the isolator's reference will stop the split from getting any further than its page_ref_freeze().

So decide between the two cases by "list", but add VM_WARN_ON()s to verify that they match our lru expectations.

[Hugh Dickins: rewrite commit log]
Cc Johannes Weiner <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 mm/huge_memory.c |   20 ++++++--------------
 1 file changed, 6 insertions(+), 14 deletions(-)
mm/huge_memory.c renamed from mm/huge_memory.c~mm-thp-simplify-lru_add_page_tail +6 -14
@@ -2367,24 +2367,16 @@ static void lru_add_page_tail(struct pag
 	VM_BUG_ON_PAGE(PageLRU(tail), head);
 	lockdep_assert_held(&lruvec_pgdat(lruvec)->lru_lock);
 
-	if (!list)
-		SetPageLRU(tail);
-
-	if (likely(PageLRU(head)))
-		list_add_tail(&tail->lru, &head->lru);
-	else if (list) {
+	if (list) {
 		/* page reclaim is reclaiming a huge page */
+		VM_WARN_ON(PageLRU(head));
 		get_page(tail);
 		list_add_tail(&tail->lru, list);
 	} else {
-		/*
-		 * Head page has not yet been counted, as an hpage,
-		 * so we must account for each subpage individually.
-		 *
-		 * Put tail on the list at the correct position
-		 * so they all end up in order.
-		 */
-		add_page_to_lru_list_tail(tail, lruvec, page_lru(tail));
+		/* head is still on lru (and we have it frozen) */
+		VM_WARN_ON(!PageLRU(head));
+		SetPageLRU(tail);
+		list_add_tail(&tail->lru, &head->lru);
 	}
 }
 

_

author maintainer
#7 lore raw

PATCH 4/19 mm/thp: narrow lru locking

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/thp: narrow lru locking

lru_lock and page cache xa_lock have no obvious reason to be taken one way round or the other: until now, lru_lock has been taken before page cache xa_lock, when splitting a THP; but nothing else takes them together. Reverse that ordering: let's narrow the lru locking - but leave local_irq_disable to block interrupts throughout, like before.

Hugh Dickins point: split_huge_page_to_list() was already silly, to be using the _irqsave variant: it's just been taking sleeping locks, so would already be broken if entered with interrupts enabled. So we can save passing flags argument down to __split_huge_page().

Why change the lock ordering here? That was hard to decide. One reason:
when this series reaches per-memcg lru locking, it relies on the THP's memcg to be stable when taking the lru_lock: that is now done after the THP's refcount has been frozen, which ensures page memcg cannot change.

Another reason: previously, lock_page_memcg()'s move_lock was presumed to nest inside lru_lock; but now lru_lock must nest inside (page cache lock inside) move_lock, so it becomes possible to use lock_page_memcg() to stabilize page memcg before taking its lru_lock. That is not the mechanism used in this series, but it is an option we want to keep open.

[[email protected]: rewrite commit log]
Cc Kirill A. Shutemov <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Johannes Weiner <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 mm/huge_memory.c |   25 +++++++++++++------------
 1 file changed, 13 insertions(+), 12 deletions(-)
mm/huge_memory.c renamed from mm/huge_memory.c~mm-thp-narrow-lru-locking +13 -12
@@ -2446,7 +2446,7 @@ static void __split_huge_page_tail(struc
 }
 
 static void __split_huge_page(struct page *page, struct list_head *list,
-		pgoff_t end, unsigned long flags)
+		pgoff_t end)
 {
 	struct page *head = compound_head(page);
 	pg_data_t *pgdat = page_pgdat(head);
@@ -2456,8 +2456,6 @@ static void __split_huge_page(struct pag
 	unsigned int nr = thp_nr_pages(head);
 	int i;
 
-	lruvec = mem_cgroup_page_lruvec(head, pgdat);
-
 	/* complete memcg works before add pages to LRU */
 	mem_cgroup_split_huge_fixup(head);
 
@@ -2469,6 +2467,11 @@ static void __split_huge_page(struct pag
 		xa_lock(&swap_cache->i_pages);
 	}
 
+	/* prevent PageLRU to go away from under us, and freeze lru stats */
+	spin_lock(&pgdat->lru_lock);
+
+	lruvec = mem_cgroup_page_lruvec(head, pgdat);
+
 	for (i = nr - 1; i >= 1; i--) {
 		__split_huge_page_tail(head, i, lruvec, list);
 		/* Some pages can be beyond i_size: drop them from page cache */
@@ -2488,6 +2491,8 @@ static void __split_huge_page(struct pag
 	}
 
 	ClearPageCompound(head);
+	spin_unlock(&pgdat->lru_lock);
+	/* Caller disabled irqs, so they are still disabled here */
 
 	split_page_owner(head, nr);
 
@@ -2505,8 +2510,7 @@ static void __split_huge_page(struct pag
 		page_ref_add(head, 2);
 		xa_unlock(&head->mapping->i_pages);
 	}
-
-	spin_unlock_irqrestore(&pgdat->lru_lock, flags);
+	local_irq_enable();
 
 	remap_page(head, nr);
 
@@ -2652,12 +2656,10 @@ bool can_split_huge_page(struct page *pa
 int split_huge_page_to_list(struct page *page, struct list_head *list)
 {
 	struct page *head = compound_head(page);
-	struct pglist_data *pgdata = NODE_DATA(page_to_nid(head));
 	struct deferred_split *ds_queue = get_deferred_split_queue(head);
 	struct anon_vma *anon_vma = NULL;
 	struct address_space *mapping = NULL;
 	int count, mapcount, extra_pins, ret;
-	unsigned long flags;
 	pgoff_t end;
 
 	VM_BUG_ON_PAGE(is_huge_zero_page(head), head);
@@ -2718,9 +2720,8 @@ int split_huge_page_to_list(struct page
 	unmap_page(head);
 	VM_BUG_ON_PAGE(compound_mapcount(head), head);
 
-	/* prevent PageLRU to go away from under us, and freeze lru stats */
-	spin_lock_irqsave(&pgdata->lru_lock, flags);
-
+	/* block interrupt reentry in xa_lock and spinlock */
+	local_irq_disable();
 	if (mapping) {
 		XA_STATE(xas, &mapping->i_pages, page_index(head));
 
@@ -2750,7 +2751,7 @@ int split_huge_page_to_list(struct page
 				__dec_lruvec_page_state(head, NR_FILE_THPS);
 		}
 
-		__split_huge_page(page, list, end, flags);
+		__split_huge_page(page, list, end);
 		ret = 0;
 	} else {
 		if (IS_ENABLED(CONFIG_DEBUG_VM) && mapcount) {
@@ -2764,7 +2765,7 @@ int split_huge_page_to_list(struct page
 		spin_unlock(&ds_queue->split_queue_lock);
 fail:		if (mapping)
 			xa_unlock(&mapping->i_pages);
-		spin_unlock_irqrestore(&pgdata->lru_lock, flags);
+		local_irq_enable();
 		remap_page(head, thp_nr_pages(head));
 		ret = -EBUSY;
 	}

_

author maintainer
#8 lore raw

PATCH 4/19 mm/thp: narrow lru locking

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/thp: narrow lru locking

lru_lock and page cache xa_lock have no obvious reason to be taken one way round or the other: until now, lru_lock has been taken before page cache xa_lock, when splitting a THP; but nothing else takes them together. Reverse that ordering: let's narrow the lru locking - but leave local_irq_disable to block interrupts throughout, like before.

Hugh Dickins point: split_huge_page_to_list() was already silly, to be using the _irqsave variant: it's just been taking sleeping locks, so would already be broken if entered with interrupts enabled. So we can save passing flags argument down to __split_huge_page().

Why change the lock ordering here? That was hard to decide. One reason:
when this series reaches per-memcg lru locking, it relies on the THP's memcg to be stable when taking the lru_lock: that is now done after the THP's refcount has been frozen, which ensures page memcg cannot change.

Another reason: previously, lock_page_memcg()'s move_lock was presumed to nest inside lru_lock; but now lru_lock must nest inside (page cache lock inside) move_lock, so it becomes possible to use lock_page_memcg() to stabilize page memcg before taking its lru_lock. That is not the mechanism used in this series, but it is an option we want to keep open.

[[email protected]: rewrite commit log]
Cc Kirill A. Shutemov <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Johannes Weiner <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 mm/huge_memory.c |   25 +++++++++++++------------
 1 file changed, 13 insertions(+), 12 deletions(-)
mm/huge_memory.c renamed from mm/huge_memory.c~mm-thp-narrow-lru-locking +13 -12
@@ -2446,7 +2446,7 @@ static void __split_huge_page_tail(struc
 }
 
 static void __split_huge_page(struct page *page, struct list_head *list,
-		pgoff_t end, unsigned long flags)
+		pgoff_t end)
 {
 	struct page *head = compound_head(page);
 	pg_data_t *pgdat = page_pgdat(head);
@@ -2456,8 +2456,6 @@ static void __split_huge_page(struct pag
 	unsigned int nr = thp_nr_pages(head);
 	int i;
 
-	lruvec = mem_cgroup_page_lruvec(head, pgdat);
-
 	/* complete memcg works before add pages to LRU */
 	mem_cgroup_split_huge_fixup(head);
 
@@ -2469,6 +2467,11 @@ static void __split_huge_page(struct pag
 		xa_lock(&swap_cache->i_pages);
 	}
 
+	/* prevent PageLRU to go away from under us, and freeze lru stats */
+	spin_lock(&pgdat->lru_lock);
+
+	lruvec = mem_cgroup_page_lruvec(head, pgdat);
+
 	for (i = nr - 1; i >= 1; i--) {
 		__split_huge_page_tail(head, i, lruvec, list);
 		/* Some pages can be beyond i_size: drop them from page cache */
@@ -2488,6 +2491,8 @@ static void __split_huge_page(struct pag
 	}
 
 	ClearPageCompound(head);
+	spin_unlock(&pgdat->lru_lock);
+	/* Caller disabled irqs, so they are still disabled here */
 
 	split_page_owner(head, nr);
 
@@ -2505,8 +2510,7 @@ static void __split_huge_page(struct pag
 		page_ref_add(head, 2);
 		xa_unlock(&head->mapping->i_pages);
 	}
-
-	spin_unlock_irqrestore(&pgdat->lru_lock, flags);
+	local_irq_enable();
 
 	remap_page(head, nr);
 
@@ -2652,12 +2656,10 @@ bool can_split_huge_page(struct page *pa
 int split_huge_page_to_list(struct page *page, struct list_head *list)
 {
 	struct page *head = compound_head(page);
-	struct pglist_data *pgdata = NODE_DATA(page_to_nid(head));
 	struct deferred_split *ds_queue = get_deferred_split_queue(head);
 	struct anon_vma *anon_vma = NULL;
 	struct address_space *mapping = NULL;
 	int count, mapcount, extra_pins, ret;
-	unsigned long flags;
 	pgoff_t end;
 
 	VM_BUG_ON_PAGE(is_huge_zero_page(head), head);
@@ -2718,9 +2720,8 @@ int split_huge_page_to_list(struct page
 	unmap_page(head);
 	VM_BUG_ON_PAGE(compound_mapcount(head), head);
 
-	/* prevent PageLRU to go away from under us, and freeze lru stats */
-	spin_lock_irqsave(&pgdata->lru_lock, flags);
-
+	/* block interrupt reentry in xa_lock and spinlock */
+	local_irq_disable();
 	if (mapping) {
 		XA_STATE(xas, &mapping->i_pages, page_index(head));
 
@@ -2750,7 +2751,7 @@ int split_huge_page_to_list(struct page
 				__dec_lruvec_page_state(head, NR_FILE_THPS);
 		}
 
-		__split_huge_page(page, list, end, flags);
+		__split_huge_page(page, list, end);
 		ret = 0;
 	} else {
 		if (IS_ENABLED(CONFIG_DEBUG_VM) && mapcount) {
@@ -2764,7 +2765,7 @@ int split_huge_page_to_list(struct page
 		spin_unlock(&ds_queue->split_queue_lock);
 fail:		if (mapping)
 			xa_unlock(&mapping->i_pages);
-		spin_unlock_irqrestore(&pgdata->lru_lock, flags);
+		local_irq_enable();
 		remap_page(head, thp_nr_pages(head));
 		ret = -EBUSY;
 	}

_

author maintainer
#9 lore raw

PATCH 5/19 mm/vmscan: remove unnecessary lruvec adding

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/vmscan: remove unnecessary lruvec adding

We don't have to add a freeable page into lru and then remove from it. This change saves a couple of actions and makes the moving more clear.

The SetPageLRU needs to be kept before put_page_testzero for list integrity, otherwise:

#0 move_pages_to_lru #1 release_pages
if !put_page_testzero

     			           if (put_page_testzero())
     			              !PageLRU //skip lru_lock
     SetPageLRU()
     list_add(&page->lru,)
                                         list_add(&page->lru,)
[[email protected]: coding style fixes]
Cc Tejun Heo <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 mm/vmscan.c |   38 +++++++++++++++++++++++++-------------
 1 file changed, 25 insertions(+), 13 deletions(-)
mm/vmscan.c renamed from mm/vmscan.c~mm-vmscan-remove-unnecessary-lruvec-adding +25 -13
@@ -1851,26 +1851,30 @@ static unsigned noinline_for_stack move_
 	while (!list_empty(list)) {
 		page = lru_to_page(list);
 		VM_BUG_ON_PAGE(PageLRU(page), page);
+		list_del(&page->lru);
 		if (unlikely(!page_evictable(page))) {
-			list_del(&page->lru);
 			spin_unlock_irq(&pgdat->lru_lock);
 			putback_lru_page(page);
 			spin_lock_irq(&pgdat->lru_lock);
 			continue;
 		}
-		lruvec = mem_cgroup_page_lruvec(page, pgdat);
 
+		/*
+		 * The SetPageLRU needs to be kept here for list integrity.
+		 * Otherwise:
+		 *   #0 move_pages_to_lru             #1 release_pages
+		 *   if !put_page_testzero
+		 *				      if (put_page_testzero())
+		 *				        !PageLRU //skip lru_lock
+		 *     SetPageLRU()
+		 *     list_add(&page->lru,)
+		 *                                        list_add(&page->lru,)
+		 */
 		SetPageLRU(page);
-		lru = page_lru(page);
 
-		nr_pages = thp_nr_pages(page);
-		update_lru_size(lruvec, lru, page_zonenum(page), nr_pages);
-		list_move(&page->lru, &lruvec->lists[lru]);
-
-		if (put_page_testzero(page)) {
+		if (unlikely(put_page_testzero(page))) {
 			__ClearPageLRU(page);
 			__ClearPageActive(page);
-			del_page_from_lru_list(page, lruvec, lru);
 
 			if (unlikely(PageCompound(page))) {
 				spin_unlock_irq(&pgdat->lru_lock);
@@ -1878,11 +1882,19 @@ static unsigned noinline_for_stack move_
 				spin_lock_irq(&pgdat->lru_lock);
 			} else
 				list_add(&page->lru, &pages_to_free);
-		} else {
-			nr_moved += nr_pages;
-			if (PageActive(page))
-				workingset_age_nonresident(lruvec, nr_pages);
+
+			continue;
 		}
+
+		lruvec = mem_cgroup_page_lruvec(page, pgdat);
+		lru = page_lru(page);
+		nr_pages = thp_nr_pages(page);
+
+		update_lru_size(lruvec, lru, page_zonenum(page), nr_pages);
+		list_add(&page->lru, &lruvec->lists[lru]);
+		nr_moved += nr_pages;
+		if (PageActive(page))
+			workingset_age_nonresident(lruvec, nr_pages);
 	}
 
 	/*

_

author maintainer
#10 lore raw

PATCH 5/19 mm/vmscan: remove unnecessary lruvec adding

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/vmscan: remove unnecessary lruvec adding

We don't have to add a freeable page into lru and then remove from it. This change saves a couple of actions and makes the moving more clear.

The SetPageLRU needs to be kept before put_page_testzero for list integrity, otherwise:

#0 move_pages_to_lru #1 release_pages
if !put_page_testzero

     			           if (put_page_testzero())
     			              !PageLRU //skip lru_lock
     SetPageLRU()
     list_add(&page->lru,)
                                         list_add(&page->lru,)
[[email protected]: coding style fixes]
Cc Tejun Heo <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 mm/vmscan.c |   38 +++++++++++++++++++++++++-------------
 1 file changed, 25 insertions(+), 13 deletions(-)
mm/vmscan.c renamed from mm/vmscan.c~mm-vmscan-remove-unnecessary-lruvec-adding +25 -13
@@ -1851,26 +1851,30 @@ static unsigned noinline_for_stack move_
 	while (!list_empty(list)) {
 		page = lru_to_page(list);
 		VM_BUG_ON_PAGE(PageLRU(page), page);
+		list_del(&page->lru);
 		if (unlikely(!page_evictable(page))) {
-			list_del(&page->lru);
 			spin_unlock_irq(&pgdat->lru_lock);
 			putback_lru_page(page);
 			spin_lock_irq(&pgdat->lru_lock);
 			continue;
 		}
-		lruvec = mem_cgroup_page_lruvec(page, pgdat);
 
+		/*
+		 * The SetPageLRU needs to be kept here for list integrity.
+		 * Otherwise:
+		 *   #0 move_pages_to_lru             #1 release_pages
+		 *   if !put_page_testzero
+		 *				      if (put_page_testzero())
+		 *				        !PageLRU //skip lru_lock
+		 *     SetPageLRU()
+		 *     list_add(&page->lru,)
+		 *                                        list_add(&page->lru,)
+		 */
 		SetPageLRU(page);
-		lru = page_lru(page);
 
-		nr_pages = thp_nr_pages(page);
-		update_lru_size(lruvec, lru, page_zonenum(page), nr_pages);
-		list_move(&page->lru, &lruvec->lists[lru]);
-
-		if (put_page_testzero(page)) {
+		if (unlikely(put_page_testzero(page))) {
 			__ClearPageLRU(page);
 			__ClearPageActive(page);
-			del_page_from_lru_list(page, lruvec, lru);
 
 			if (unlikely(PageCompound(page))) {
 				spin_unlock_irq(&pgdat->lru_lock);
@@ -1878,11 +1882,19 @@ static unsigned noinline_for_stack move_
 				spin_lock_irq(&pgdat->lru_lock);
 			} else
 				list_add(&page->lru, &pages_to_free);
-		} else {
-			nr_moved += nr_pages;
-			if (PageActive(page))
-				workingset_age_nonresident(lruvec, nr_pages);
+
+			continue;
 		}
+
+		lruvec = mem_cgroup_page_lruvec(page, pgdat);
+		lru = page_lru(page);
+		nr_pages = thp_nr_pages(page);
+
+		update_lru_size(lruvec, lru, page_zonenum(page), nr_pages);
+		list_add(&page->lru, &lruvec->lists[lru]);
+		nr_moved += nr_pages;
+		if (PageActive(page))
+			workingset_age_nonresident(lruvec, nr_pages);
 	}
 
 	/*

_

author maintainer
#11 lore raw

PATCH 6/19 mm/rmap: stop store reordering issue on page->mapping

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/rmap: stop store reordering issue on page->mapping

Hugh Dickins and Minchan Kim observed a long time issue which discussed here, but actully the mentioned fix missed.
https://lore.kernel.org/lkml/20150504031722.GA2768@blaptop/ The store reordering may cause problem in the scenario:

	CPU 0						CPU1
   do_anonymous_page
	page_add_new_anon_rmap()
	  page->mapping = anon_vma + PAGE_MAPPING_ANON
	lru_cache_add_inactive_or_unevictable()
	  spin_lock(lruvec->lock)
	  SetPageLRU()
	  spin_unlock(lruvec->lock)
						/* idletacking judged it as LRU
						 * page so pass the page in
						 * page_idle_clear_pte_refs
						 */
						page_idle_clear_pte_refs
						  rmap_walk
						    if PageAnon(page)

Johannes give detailed examples how the store reordering could cause a
trouble: "The concern is the SetPageLRU may get reorder before 'page->mapping' setting, That would make CPU 1 will observe at page->mapping after observing PageLRU set on the page.

1. anon_vma + PAGE_MAPPING_ANON

That's the in-order scenario and is fine.

2. NULL

That's possible if the page->mapping store gets reordered to occur after SetPageLRU. That's fine too because we check for it.

3. anon_vma without the PAGE_MAPPING_ANON bit

That would be a problem and could lead to all kinds of undesirable behavior including crashes and data corruption.

Is it possible? AFAICT the compiler is allowed to tear the store to page->mapping and I don't see anything that would prevent it.

That said, I also don't see how the reader testing PageLRU under the lru_lock would prevent that in the first place. AFAICT we need that WRITE_ONCE() around the page->mapping assignment."

[[email protected]: updated for comments change from Johannes]
Cc Matthew Wilcox <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 mm/rmap.c |    8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
mm/rmap.c renamed from mm/rmap.c~mm-rmap-stop-store-reordering-issue-on-page-mapping +7 -1
@@ -1054,8 +1054,14 @@ static void __page_set_anon_rmap(struct
 	if (!exclusive)
 		anon_vma = anon_vma->root;
 
+	/*
+	 * page_idle does a lockless/optimistic rmap scan on page->mapping.
+	 * Make sure the compiler doesn't split the stores of anon_vma and
+	 * the PAGE_MAPPING_ANON type identifier, otherwise the rmap code
+	 * could mistake the mapping for a struct address_space and crash.
+	 */
 	anon_vma = (void *) anon_vma + PAGE_MAPPING_ANON;
-	page->mapping = (struct address_space *) anon_vma;
+	WRITE_ONCE(page->mapping, (struct address_space *) anon_vma);
 	page->index = linear_page_index(vma, address);
 }
 

_

author maintainer
#12 lore raw

PATCH 6/19 mm/rmap: stop store reordering issue on page->mapping

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/rmap: stop store reordering issue on page->mapping

Hugh Dickins and Minchan Kim observed a long time issue which discussed here, but actully the mentioned fix missed.
https://lore.kernel.org/lkml/20150504031722.GA2768@blaptop/ The store reordering may cause problem in the scenario:

	CPU 0						CPU1
   do_anonymous_page
	page_add_new_anon_rmap()
	  page->mapping = anon_vma + PAGE_MAPPING_ANON
	lru_cache_add_inactive_or_unevictable()
	  spin_lock(lruvec->lock)
	  SetPageLRU()
	  spin_unlock(lruvec->lock)
						/* idletacking judged it as LRU
						 * page so pass the page in
						 * page_idle_clear_pte_refs
						 */
						page_idle_clear_pte_refs
						  rmap_walk
						    if PageAnon(page)

Johannes give detailed examples how the store reordering could cause a
trouble: "The concern is the SetPageLRU may get reorder before 'page->mapping' setting, That would make CPU 1 will observe at page->mapping after observing PageLRU set on the page.

1. anon_vma + PAGE_MAPPING_ANON

That's the in-order scenario and is fine.

2. NULL

That's possible if the page->mapping store gets reordered to occur after SetPageLRU. That's fine too because we check for it.

3. anon_vma without the PAGE_MAPPING_ANON bit

That would be a problem and could lead to all kinds of undesirable behavior including crashes and data corruption.

Is it possible? AFAICT the compiler is allowed to tear the store to page->mapping and I don't see anything that would prevent it.

That said, I also don't see how the reader testing PageLRU under the lru_lock would prevent that in the first place. AFAICT we need that WRITE_ONCE() around the page->mapping assignment."

[[email protected]: updated for comments change from Johannes]
Cc Matthew Wilcox <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 mm/rmap.c |    8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
mm/rmap.c renamed from mm/rmap.c~mm-rmap-stop-store-reordering-issue-on-page-mapping +7 -1
@@ -1054,8 +1054,14 @@ static void __page_set_anon_rmap(struct
 	if (!exclusive)
 		anon_vma = anon_vma->root;
 
+	/*
+	 * page_idle does a lockless/optimistic rmap scan on page->mapping.
+	 * Make sure the compiler doesn't split the stores of anon_vma and
+	 * the PAGE_MAPPING_ANON type identifier, otherwise the rmap code
+	 * could mistake the mapping for a struct address_space and crash.
+	 */
 	anon_vma = (void *) anon_vma + PAGE_MAPPING_ANON;
-	page->mapping = (struct address_space *) anon_vma;
+	WRITE_ONCE(page->mapping, (struct address_space *) anon_vma);
 	page->index = linear_page_index(vma, address);
 }
 

_

author maintainer
#13 lore raw

PATCH 7/19 mm: page_idle_get_page() does not need lru_lock

In reply to Andrew Morton

From: Hugh Dickins <[email protected]>
Subject: mm: page_idle_get_page() does not need lru_lock

It is necessary for page_idle_get_page() to recheck PageLRU() after get_page_unless_zero(), but holding lru_lock around that serves no useful purpose, and adds to lru_lock contention: delete it.

See https://lore.kernel.org/lkml/20150504031722.GA2768@blaptop for the discussion that led to lru_lock there; but __page_set_anon_rmap() now uses WRITE_ONCE(), and I see no other risk in page_idle_clear_pte_refs() using rmap_walk() (beyond the risk of racing PageAnon->PageKsm, mostly but not entirely prevented by page_count() check in ksm.c's write_protect_page():
that risk being shared with page_referenced() and not helped by lru_lock).

Cc Vladimir Davydov <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Alex Shi <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 mm/page_idle.c |    4 ----
 1 file changed, 4 deletions(-)
mm/page_idle.c renamed from mm/page_idle.c~mm-page_idle_get_page-does-not-need-lru_lock +0 -4
@@ -32,19 +32,15 @@
 static struct page *page_idle_get_page(unsigned long pfn)
 {
 	struct page *page = pfn_to_online_page(pfn);
-	pg_data_t *pgdat;
 
 	if (!page || !PageLRU(page) ||
 	    !get_page_unless_zero(page))
 		return NULL;
 
-	pgdat = page_pgdat(page);
-	spin_lock_irq(&pgdat->lru_lock);
 	if (unlikely(!PageLRU(page))) {
 		put_page(page);
 		page = NULL;
 	}
-	spin_unlock_irq(&pgdat->lru_lock);
 	return page;
 }
 

_

author maintainer
#14 lore raw

PATCH 7/19 mm: page_idle_get_page() does not need lru_lock

In reply to Andrew Morton

From: Hugh Dickins <[email protected]>
Subject: mm: page_idle_get_page() does not need lru_lock

It is necessary for page_idle_get_page() to recheck PageLRU() after get_page_unless_zero(), but holding lru_lock around that serves no useful purpose, and adds to lru_lock contention: delete it.

See https://lore.kernel.org/lkml/20150504031722.GA2768@blaptop for the discussion that led to lru_lock there; but __page_set_anon_rmap() now uses WRITE_ONCE(), and I see no other risk in page_idle_clear_pte_refs() using rmap_walk() (beyond the risk of racing PageAnon->PageKsm, mostly but not entirely prevented by page_count() check in ksm.c's write_protect_page():
that risk being shared with page_referenced() and not helped by lru_lock).

Cc Vladimir Davydov <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Alex Shi <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 mm/page_idle.c |    4 ----
 1 file changed, 4 deletions(-)
mm/page_idle.c renamed from mm/page_idle.c~mm-page_idle_get_page-does-not-need-lru_lock +0 -4
@@ -32,19 +32,15 @@
 static struct page *page_idle_get_page(unsigned long pfn)
 {
 	struct page *page = pfn_to_online_page(pfn);
-	pg_data_t *pgdat;
 
 	if (!page || !PageLRU(page) ||
 	    !get_page_unless_zero(page))
 		return NULL;
 
-	pgdat = page_pgdat(page);
-	spin_lock_irq(&pgdat->lru_lock);
 	if (unlikely(!PageLRU(page))) {
 		put_page(page);
 		page = NULL;
 	}
-	spin_unlock_irq(&pgdat->lru_lock);
 	return page;
 }
 

_

author maintainer
#15 lore raw

PATCH 8/19 mm/memcg: add debug checking in lock_page_memcg

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/memcg: add debug checking in lock_page_memcg

Add a debug checking in lock_page_memcg, then we could get alarm if anything wrong here.

Cc Michal Hocko <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 mm/memcontrol.c |    6 ++++++
 1 file changed, 6 insertions(+)
mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcg-add-debug-checking-in-lock_page_memcg +6 -0
@@ -2150,6 +2150,12 @@ again:
 	if (unlikely(!memcg))
 		return NULL;
 
+#ifdef CONFIG_PROVE_LOCKING
+	local_irq_save(flags);
+	might_lock(&memcg->move_lock);
+	local_irq_restore(flags);
+#endif
+
 	if (atomic_read(&memcg->moving_account) <= 0)
 		return memcg;
 

_

author maintainer
#16 lore raw

PATCH 8/19 mm/memcg: add debug checking in lock_page_memcg

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/memcg: add debug checking in lock_page_memcg

Add a debug checking in lock_page_memcg, then we could get alarm if anything wrong here.

Cc Michal Hocko <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 mm/memcontrol.c |    6 ++++++
 1 file changed, 6 insertions(+)
mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcg-add-debug-checking-in-lock_page_memcg +6 -0
@@ -2150,6 +2150,12 @@ again:
 	if (unlikely(!memcg))
 		return NULL;
 
+#ifdef CONFIG_PROVE_LOCKING
+	local_irq_save(flags);
+	might_lock(&memcg->move_lock);
+	local_irq_restore(flags);
+#endif
+
 	if (atomic_read(&memcg->moving_account) <= 0)
 		return memcg;
 

_

author maintainer
#17 lore raw

PATCH 9/19 mm/swap.c: fold vm event PGROTATED into pagevec_move_tail_fn

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/swap.c: fold vm event PGROTATED into pagevec_move_tail_fn

Fold the PGROTATED event collection into pagevec_move_tail_fn call back func like other funcs does in pagevec_lru_move_fn. Thus we could save func call pagevec_move_tail(). Now all usage of pagevec_lru_move_fn are same and no needs of its 3rd parameter.

It's just simply the calling. No functional change.

[[email protected]: found a build issue in the original patch, thanks]
Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 mm/swap.c |   65 ++++++++++++++++++----------------------------------
 1 file changed, 23 insertions(+), 42 deletions(-)
mm/swap.c renamed from mm/swap.c~mm-swapc-fold-vm-event-pgrotated-into-pagevec_move_tail_fn +23 -42
@@ -204,8 +204,7 @@ int get_kernel_page(unsigned long start,
 EXPORT_SYMBOL_GPL(get_kernel_page);
 
 static void pagevec_lru_move_fn(struct pagevec *pvec,
-	void (*move_fn)(struct page *page, struct lruvec *lruvec, void *arg),
-	void *arg)
+	void (*move_fn)(struct page *page, struct lruvec *lruvec))
 {
 	int i;
 	struct pglist_data *pgdat = NULL;
@@ -224,7 +223,7 @@ static void pagevec_lru_move_fn(struct p
 		}
 
 		lruvec = mem_cgroup_page_lruvec(page, pgdat);
-		(*move_fn)(page, lruvec, arg);
+		(*move_fn)(page, lruvec);
 	}
 	if (pgdat)
 		spin_unlock_irqrestore(&pgdat->lru_lock, flags);
@@ -232,35 +231,22 @@ static void pagevec_lru_move_fn(struct p
 	pagevec_reinit(pvec);
 }
 
-static void pagevec_move_tail_fn(struct page *page, struct lruvec *lruvec,
-				 void *arg)
+static void pagevec_move_tail_fn(struct page *page, struct lruvec *lruvec)
 {
-	int *pgmoved = arg;
-
 	if (PageLRU(page) && !PageUnevictable(page)) {
 		del_page_from_lru_list(page, lruvec, page_lru(page));
 		ClearPageActive(page);
 		add_page_to_lru_list_tail(page, lruvec, page_lru(page));
-		(*pgmoved) += thp_nr_pages(page);
+		__count_vm_events(PGROTATED, thp_nr_pages(page));
 	}
 }
 
 /*
- * pagevec_move_tail() must be called with IRQ disabled.
- * Otherwise this may cause nasty races.
- */
-static void pagevec_move_tail(struct pagevec *pvec)
-{
-	int pgmoved = 0;
-
-	pagevec_lru_move_fn(pvec, pagevec_move_tail_fn, &pgmoved);
-	__count_vm_events(PGROTATED, pgmoved);
-}
-
-/*
  * Writeback is about to end against a page which has been marked for immediate
  * reclaim.  If it still appears to be reclaimable, move it to the tail of the
  * inactive list.
+ *
+ * rotate_reclaimable_page() must disable IRQs, to prevent nasty races.
  */
 void rotate_reclaimable_page(struct page *page)
 {
@@ -273,7 +259,7 @@ void rotate_reclaimable_page(struct page
 		local_lock_irqsave(&lru_rotate.lock, flags);
 		pvec = this_cpu_ptr(&lru_rotate.pvec);
 		if (!pagevec_add(pvec, page) || PageCompound(page))
-			pagevec_move_tail(pvec);
+			pagevec_lru_move_fn(pvec, pagevec_move_tail_fn);
 		local_unlock_irqrestore(&lru_rotate.lock, flags);
 	}
 }
@@ -315,8 +301,7 @@ void lru_note_cost_page(struct page *pag
 		      page_is_file_lru(page), thp_nr_pages(page));
 }
 
-static void __activate_page(struct page *page, struct lruvec *lruvec,
-			    void *arg)
+static void __activate_page(struct page *page, struct lruvec *lruvec)
 {
 	if (PageLRU(page) && !PageActive(page) && !PageUnevictable(page)) {
 		int lru = page_lru_base_type(page);
@@ -340,7 +325,7 @@ static void activate_page_drain(int cpu)
 	struct pagevec *pvec = &per_cpu(lru_pvecs.activate_page, cpu);
 
 	if (pagevec_count(pvec))
-		pagevec_lru_move_fn(pvec, __activate_page, NULL);
+		pagevec_lru_move_fn(pvec, __activate_page);
 }
 
 static bool need_activate_page_drain(int cpu)
@@ -358,7 +343,7 @@ static void activate_page(struct page *p
 		pvec = this_cpu_ptr(&lru_pvecs.activate_page);
 		get_page(page);
 		if (!pagevec_add(pvec, page) || PageCompound(page))
-			pagevec_lru_move_fn(pvec, __activate_page, NULL);
+			pagevec_lru_move_fn(pvec, __activate_page);
 		local_unlock(&lru_pvecs.lock);
 	}
 }
@@ -374,7 +359,7 @@ static void activate_page(struct page *p
 
 	page = compound_head(page);
 	spin_lock_irq(&pgdat->lru_lock);
-	__activate_page(page, mem_cgroup_page_lruvec(page, pgdat), NULL);
+	__activate_page(page, mem_cgroup_page_lruvec(page, pgdat));
 	spin_unlock_irq(&pgdat->lru_lock);
 }
 #endif
@@ -525,8 +510,7 @@ void lru_cache_add_inactive_or_unevictab
  * be write it out by flusher threads as this is much more effective
  * than the single-page writeout from reclaim.
  */
-static void lru_deactivate_file_fn(struct page *page, struct lruvec *lruvec,
-			      void *arg)
+static void lru_deactivate_file_fn(struct page *page, struct lruvec *lruvec)
 {
 	int lru;
 	bool active;
@@ -573,8 +557,7 @@ static void lru_deactivate_file_fn(struc
 	}
 }
 
-static void lru_deactivate_fn(struct page *page, struct lruvec *lruvec,
-			    void *arg)
+static void lru_deactivate_fn(struct page *page, struct lruvec *lruvec)
 {
 	if (PageLRU(page) && PageActive(page) && !PageUnevictable(page)) {
 		int lru = page_lru_base_type(page);
@@ -591,8 +574,7 @@ static void lru_deactivate_fn(struct pag
 	}
 }
 
-static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec,
-			    void *arg)
+static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec)
 {
 	if (PageLRU(page) && PageAnon(page) && PageSwapBacked(page) &&
 	    !PageSwapCache(page) && !PageUnevictable(page)) {
@@ -636,21 +618,21 @@ void lru_add_drain_cpu(int cpu)
 
 		/* No harm done if a racing interrupt already did this */
 		local_lock_irqsave(&lru_rotate.lock, flags);
-		pagevec_move_tail(pvec);
+		pagevec_lru_move_fn(pvec, pagevec_move_tail_fn);
 		local_unlock_irqrestore(&lru_rotate.lock, flags);
 	}
 
 	pvec = &per_cpu(lru_pvecs.lru_deactivate_file, cpu);
 	if (pagevec_count(pvec))
-		pagevec_lru_move_fn(pvec, lru_deactivate_file_fn, NULL);
+		pagevec_lru_move_fn(pvec, lru_deactivate_file_fn);
 
 	pvec = &per_cpu(lru_pvecs.lru_deactivate, cpu);
 	if (pagevec_count(pvec))
-		pagevec_lru_move_fn(pvec, lru_deactivate_fn, NULL);
+		pagevec_lru_move_fn(pvec, lru_deactivate_fn);
 
 	pvec = &per_cpu(lru_pvecs.lru_lazyfree, cpu);
 	if (pagevec_count(pvec))
-		pagevec_lru_move_fn(pvec, lru_lazyfree_fn, NULL);
+		pagevec_lru_move_fn(pvec, lru_lazyfree_fn);
 
 	activate_page_drain(cpu);
 }
@@ -679,7 +661,7 @@ void deactivate_file_page(struct page *p
 		pvec = this_cpu_ptr(&lru_pvecs.lru_deactivate_file);
 
 		if (!pagevec_add(pvec, page) || PageCompound(page))
-			pagevec_lru_move_fn(pvec, lru_deactivate_file_fn, NULL);
+			pagevec_lru_move_fn(pvec, lru_deactivate_file_fn);
 		local_unlock(&lru_pvecs.lock);
 	}
 }
@@ -701,7 +683,7 @@ void deactivate_page(struct page *page)
 		pvec = this_cpu_ptr(&lru_pvecs.lru_deactivate);
 		get_page(page);
 		if (!pagevec_add(pvec, page) || PageCompound(page))
-			pagevec_lru_move_fn(pvec, lru_deactivate_fn, NULL);
+			pagevec_lru_move_fn(pvec, lru_deactivate_fn);
 		local_unlock(&lru_pvecs.lock);
 	}
 }
@@ -723,7 +705,7 @@ void mark_page_lazyfree(struct page *pag
 		pvec = this_cpu_ptr(&lru_pvecs.lru_lazyfree);
 		get_page(page);
 		if (!pagevec_add(pvec, page) || PageCompound(page))
-			pagevec_lru_move_fn(pvec, lru_lazyfree_fn, NULL);
+			pagevec_lru_move_fn(pvec, lru_lazyfree_fn);
 		local_unlock(&lru_pvecs.lock);
 	}
 }
@@ -977,8 +959,7 @@ void __pagevec_release(struct pagevec *p
 }
 EXPORT_SYMBOL(__pagevec_release);
 
-static void __pagevec_lru_add_fn(struct page *page, struct lruvec *lruvec,
-				 void *arg)
+static void __pagevec_lru_add_fn(struct page *page, struct lruvec *lruvec)
 {
 	enum lru_list lru;
 	int was_unevictable = TestClearPageUnevictable(page);
@@ -1037,7 +1018,7 @@ static void __pagevec_lru_add_fn(struct
  */
 void __pagevec_lru_add(struct pagevec *pvec)
 {
-	pagevec_lru_move_fn(pvec, __pagevec_lru_add_fn, NULL);
+	pagevec_lru_move_fn(pvec, __pagevec_lru_add_fn);
 }
 
 /**

_

author maintainer
#18 lore raw

PATCH 9/19 mm/swap.c: fold vm event PGROTATED into pagevec_move_tail_fn

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/swap.c: fold vm event PGROTATED into pagevec_move_tail_fn

Fold the PGROTATED event collection into pagevec_move_tail_fn call back func like other funcs does in pagevec_lru_move_fn. Thus we could save func call pagevec_move_tail(). Now all usage of pagevec_lru_move_fn are same and no needs of its 3rd parameter.

It's just simply the calling. No functional change.

[[email protected]: found a build issue in the original patch, thanks]
Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 mm/swap.c |   65 ++++++++++++++++++----------------------------------
 1 file changed, 23 insertions(+), 42 deletions(-)
mm/swap.c renamed from mm/swap.c~mm-swapc-fold-vm-event-pgrotated-into-pagevec_move_tail_fn +23 -42
@@ -204,8 +204,7 @@ int get_kernel_page(unsigned long start,
 EXPORT_SYMBOL_GPL(get_kernel_page);
 
 static void pagevec_lru_move_fn(struct pagevec *pvec,
-	void (*move_fn)(struct page *page, struct lruvec *lruvec, void *arg),
-	void *arg)
+	void (*move_fn)(struct page *page, struct lruvec *lruvec))
 {
 	int i;
 	struct pglist_data *pgdat = NULL;
@@ -224,7 +223,7 @@ static void pagevec_lru_move_fn(struct p
 		}
 
 		lruvec = mem_cgroup_page_lruvec(page, pgdat);
-		(*move_fn)(page, lruvec, arg);
+		(*move_fn)(page, lruvec);
 	}
 	if (pgdat)
 		spin_unlock_irqrestore(&pgdat->lru_lock, flags);
@@ -232,35 +231,22 @@ static void pagevec_lru_move_fn(struct p
 	pagevec_reinit(pvec);
 }
 
-static void pagevec_move_tail_fn(struct page *page, struct lruvec *lruvec,
-				 void *arg)
+static void pagevec_move_tail_fn(struct page *page, struct lruvec *lruvec)
 {
-	int *pgmoved = arg;
-
 	if (PageLRU(page) && !PageUnevictable(page)) {
 		del_page_from_lru_list(page, lruvec, page_lru(page));
 		ClearPageActive(page);
 		add_page_to_lru_list_tail(page, lruvec, page_lru(page));
-		(*pgmoved) += thp_nr_pages(page);
+		__count_vm_events(PGROTATED, thp_nr_pages(page));
 	}
 }
 
 /*
- * pagevec_move_tail() must be called with IRQ disabled.
- * Otherwise this may cause nasty races.
- */
-static void pagevec_move_tail(struct pagevec *pvec)
-{
-	int pgmoved = 0;
-
-	pagevec_lru_move_fn(pvec, pagevec_move_tail_fn, &pgmoved);
-	__count_vm_events(PGROTATED, pgmoved);
-}
-
-/*
  * Writeback is about to end against a page which has been marked for immediate
  * reclaim.  If it still appears to be reclaimable, move it to the tail of the
  * inactive list.
+ *
+ * rotate_reclaimable_page() must disable IRQs, to prevent nasty races.
  */
 void rotate_reclaimable_page(struct page *page)
 {
@@ -273,7 +259,7 @@ void rotate_reclaimable_page(struct page
 		local_lock_irqsave(&lru_rotate.lock, flags);
 		pvec = this_cpu_ptr(&lru_rotate.pvec);
 		if (!pagevec_add(pvec, page) || PageCompound(page))
-			pagevec_move_tail(pvec);
+			pagevec_lru_move_fn(pvec, pagevec_move_tail_fn);
 		local_unlock_irqrestore(&lru_rotate.lock, flags);
 	}
 }
@@ -315,8 +301,7 @@ void lru_note_cost_page(struct page *pag
 		      page_is_file_lru(page), thp_nr_pages(page));
 }
 
-static void __activate_page(struct page *page, struct lruvec *lruvec,
-			    void *arg)
+static void __activate_page(struct page *page, struct lruvec *lruvec)
 {
 	if (PageLRU(page) && !PageActive(page) && !PageUnevictable(page)) {
 		int lru = page_lru_base_type(page);
@@ -340,7 +325,7 @@ static void activate_page_drain(int cpu)
 	struct pagevec *pvec = &per_cpu(lru_pvecs.activate_page, cpu);
 
 	if (pagevec_count(pvec))
-		pagevec_lru_move_fn(pvec, __activate_page, NULL);
+		pagevec_lru_move_fn(pvec, __activate_page);
 }
 
 static bool need_activate_page_drain(int cpu)
@@ -358,7 +343,7 @@ static void activate_page(struct page *p
 		pvec = this_cpu_ptr(&lru_pvecs.activate_page);
 		get_page(page);
 		if (!pagevec_add(pvec, page) || PageCompound(page))
-			pagevec_lru_move_fn(pvec, __activate_page, NULL);
+			pagevec_lru_move_fn(pvec, __activate_page);
 		local_unlock(&lru_pvecs.lock);
 	}
 }
@@ -374,7 +359,7 @@ static void activate_page(struct page *p
 
 	page = compound_head(page);
 	spin_lock_irq(&pgdat->lru_lock);
-	__activate_page(page, mem_cgroup_page_lruvec(page, pgdat), NULL);
+	__activate_page(page, mem_cgroup_page_lruvec(page, pgdat));
 	spin_unlock_irq(&pgdat->lru_lock);
 }
 #endif
@@ -525,8 +510,7 @@ void lru_cache_add_inactive_or_unevictab
  * be write it out by flusher threads as this is much more effective
  * than the single-page writeout from reclaim.
  */
-static void lru_deactivate_file_fn(struct page *page, struct lruvec *lruvec,
-			      void *arg)
+static void lru_deactivate_file_fn(struct page *page, struct lruvec *lruvec)
 {
 	int lru;
 	bool active;
@@ -573,8 +557,7 @@ static void lru_deactivate_file_fn(struc
 	}
 }
 
-static void lru_deactivate_fn(struct page *page, struct lruvec *lruvec,
-			    void *arg)
+static void lru_deactivate_fn(struct page *page, struct lruvec *lruvec)
 {
 	if (PageLRU(page) && PageActive(page) && !PageUnevictable(page)) {
 		int lru = page_lru_base_type(page);
@@ -591,8 +574,7 @@ static void lru_deactivate_fn(struct pag
 	}
 }
 
-static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec,
-			    void *arg)
+static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec)
 {
 	if (PageLRU(page) && PageAnon(page) && PageSwapBacked(page) &&
 	    !PageSwapCache(page) && !PageUnevictable(page)) {
@@ -636,21 +618,21 @@ void lru_add_drain_cpu(int cpu)
 
 		/* No harm done if a racing interrupt already did this */
 		local_lock_irqsave(&lru_rotate.lock, flags);
-		pagevec_move_tail(pvec);
+		pagevec_lru_move_fn(pvec, pagevec_move_tail_fn);
 		local_unlock_irqrestore(&lru_rotate.lock, flags);
 	}
 
 	pvec = &per_cpu(lru_pvecs.lru_deactivate_file, cpu);
 	if (pagevec_count(pvec))
-		pagevec_lru_move_fn(pvec, lru_deactivate_file_fn, NULL);
+		pagevec_lru_move_fn(pvec, lru_deactivate_file_fn);
 
 	pvec = &per_cpu(lru_pvecs.lru_deactivate, cpu);
 	if (pagevec_count(pvec))
-		pagevec_lru_move_fn(pvec, lru_deactivate_fn, NULL);
+		pagevec_lru_move_fn(pvec, lru_deactivate_fn);
 
 	pvec = &per_cpu(lru_pvecs.lru_lazyfree, cpu);
 	if (pagevec_count(pvec))
-		pagevec_lru_move_fn(pvec, lru_lazyfree_fn, NULL);
+		pagevec_lru_move_fn(pvec, lru_lazyfree_fn);
 
 	activate_page_drain(cpu);
 }
@@ -679,7 +661,7 @@ void deactivate_file_page(struct page *p
 		pvec = this_cpu_ptr(&lru_pvecs.lru_deactivate_file);
 
 		if (!pagevec_add(pvec, page) || PageCompound(page))
-			pagevec_lru_move_fn(pvec, lru_deactivate_file_fn, NULL);
+			pagevec_lru_move_fn(pvec, lru_deactivate_file_fn);
 		local_unlock(&lru_pvecs.lock);
 	}
 }
@@ -701,7 +683,7 @@ void deactivate_page(struct page *page)
 		pvec = this_cpu_ptr(&lru_pvecs.lru_deactivate);
 		get_page(page);
 		if (!pagevec_add(pvec, page) || PageCompound(page))
-			pagevec_lru_move_fn(pvec, lru_deactivate_fn, NULL);
+			pagevec_lru_move_fn(pvec, lru_deactivate_fn);
 		local_unlock(&lru_pvecs.lock);
 	}
 }
@@ -723,7 +705,7 @@ void mark_page_lazyfree(struct page *pag
 		pvec = this_cpu_ptr(&lru_pvecs.lru_lazyfree);
 		get_page(page);
 		if (!pagevec_add(pvec, page) || PageCompound(page))
-			pagevec_lru_move_fn(pvec, lru_lazyfree_fn, NULL);
+			pagevec_lru_move_fn(pvec, lru_lazyfree_fn);
 		local_unlock(&lru_pvecs.lock);
 	}
 }
@@ -977,8 +959,7 @@ void __pagevec_release(struct pagevec *p
 }
 EXPORT_SYMBOL(__pagevec_release);
 
-static void __pagevec_lru_add_fn(struct page *page, struct lruvec *lruvec,
-				 void *arg)
+static void __pagevec_lru_add_fn(struct page *page, struct lruvec *lruvec)
 {
 	enum lru_list lru;
 	int was_unevictable = TestClearPageUnevictable(page);
@@ -1037,7 +1018,7 @@ static void __pagevec_lru_add_fn(struct
  */
 void __pagevec_lru_add(struct pagevec *pvec)
 {
-	pagevec_lru_move_fn(pvec, __pagevec_lru_add_fn, NULL);
+	pagevec_lru_move_fn(pvec, __pagevec_lru_add_fn);
 }
 
 /**

_

#19 lore raw
In reply to Andrew Morton
Quoted 4 lines · Andrew Morton

- more MM work: a memcg scalability improvememt

19 patches, based on 148842c98a24e508aecb929718818fbf4c2a6ff3.

I'm not seeing patch 10/19 at all.

And patch 19/19 is corrupted and has an attachment with a '^P' character in it. I could fix it up, but with the missing patch in the middle I'm not going to even try. 'b4' is also very unhappy about that patch 19/19.

I don't know what went wrong, but I'll ignore this send - please re-send the series at your leisure, ok?

Linus

author maintainer
#20 lore raw

PATCH 10/19 mm/lru: move lock into lru_note_cost

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/lru: move lock into lru_note_cost

We have to move lru_lock into lru_note_cost, since it cycle up on memcg tree, for future per lruvec lru_lock replace. It's a bit ugly and may cost a bit more locking, but benefit from multiple memcg locking could cover the lost.

Cc Johannes Weiner <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 mm/swap.c       |    3 +++
 mm/vmscan.c     |    4 +---
 mm/workingset.c |    2 --
 3 files changed, 4 insertions(+), 5 deletions(-)
mm/swap.c renamed from mm/swap.c~mm-lru-move-lock-into-lru_note_cost +3 -0
@@ -268,7 +268,9 @@ void lru_note_cost(struct lruvec *lruvec
 {
 	do {
 		unsigned long lrusize;
+		struct pglist_data *pgdat = lruvec_pgdat(lruvec);
 
+		spin_lock_irq(&pgdat->lru_lock);
 		/* Record cost event */
 		if (file)
 			lruvec->file_cost += nr_pages;
@@ -292,6 +294,7 @@ void lru_note_cost(struct lruvec *lruvec
 			lruvec->file_cost /= 2;
 			lruvec->anon_cost /= 2;
 		}
+		spin_unlock_irq(&pgdat->lru_lock);
 	} while ((lruvec = parent_lruvec(lruvec)));
 }
 
mm/vmscan.c renamed from mm/vmscan.c~mm-lru-move-lock-into-lru_note_cost +1 -3
@@ -1971,19 +1971,17 @@ shrink_inactive_list(unsigned long nr_to
 	nr_reclaimed = shrink_page_list(&page_list, pgdat, sc, &stat, false);
 
 	spin_lock_irq(&pgdat->lru_lock);
-
 	move_pages_to_lru(lruvec, &page_list);
 
 	__mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken);
-	lru_note_cost(lruvec, file, stat.nr_pageout);
 	item = current_is_kswapd() ? PGSTEAL_KSWAPD : PGSTEAL_DIRECT;
 	if (!cgroup_reclaim(sc))
 		__count_vm_events(item, nr_reclaimed);
 	__count_memcg_events(lruvec_memcg(lruvec), item, nr_reclaimed);
 	__count_vm_events(PGSTEAL_ANON + file, nr_reclaimed);
-
 	spin_unlock_irq(&pgdat->lru_lock);
 
+	lru_note_cost(lruvec, file, stat.nr_pageout);
 	mem_cgroup_uncharge_list(&page_list);
 	free_unref_page_list(&page_list);
 
mm/workingset.c renamed from mm/workingset.c~mm-lru-move-lock-into-lru_note_cost +0 -2
@@ -381,9 +381,7 @@ void workingset_refault(struct page *pag
 	if (workingset) {
 		SetPageWorkingset(page);
 		/* XXX: Move to lru_cache_add() when it supports new vs putback */
-		spin_lock_irq(&page_pgdat(page)->lru_lock);
 		lru_note_cost_page(page);
-		spin_unlock_irq(&page_pgdat(page)->lru_lock);
 		inc_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + file);
 	}
 out:

_

author maintainer
#21 lore raw

PATCH 11/19 mm/vmscan: remove lruvec reget in move_pages_to_lru

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/vmscan: remove lruvec reget in move_pages_to_lru

Isolated page shouldn't be recharged by memcg since the memcg migration isn't possible at the time. All pages were isolated from the same lruvec (and isolation inhibits memcg migration). So remove unnecessary regetting.

Thanks to Alexander Duyck for pointing this out.

Cc Konstantin Khlebnikov <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 mm/vmscan.c |    7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
mm/vmscan.c renamed from mm/vmscan.c~mm-vmscan-remove-lruvec-reget-in-move_pages_to_lru +6 -1
@@ -1886,7 +1886,12 @@ static unsigned noinline_for_stack move_
 			continue;
 		}
 
-		lruvec = mem_cgroup_page_lruvec(page, pgdat);
+		/*
+		 * All pages were isolated from the same lruvec (and isolation
+		 * inhibits memcg migration).
+		 */
+		VM_BUG_ON_PAGE(mem_cgroup_page_lruvec(page, page_pgdat(page))
+							!= lruvec, page);
 		lru = page_lru(page);
 		nr_pages = thp_nr_pages(page);
 

_

author maintainer
#22 lore raw

PATCH 11/19 mm/vmscan: remove lruvec reget in move_pages_to_lru

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/vmscan: remove lruvec reget in move_pages_to_lru

Isolated page shouldn't be recharged by memcg since the memcg migration isn't possible at the time. All pages were isolated from the same lruvec (and isolation inhibits memcg migration). So remove unnecessary regetting.

Thanks to Alexander Duyck for pointing this out.

Cc Konstantin Khlebnikov <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 mm/vmscan.c |    7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
mm/vmscan.c renamed from mm/vmscan.c~mm-vmscan-remove-lruvec-reget-in-move_pages_to_lru +6 -1
@@ -1886,7 +1886,12 @@ static unsigned noinline_for_stack move_
 			continue;
 		}
 
-		lruvec = mem_cgroup_page_lruvec(page, pgdat);
+		/*
+		 * All pages were isolated from the same lruvec (and isolation
+		 * inhibits memcg migration).
+		 */
+		VM_BUG_ON_PAGE(mem_cgroup_page_lruvec(page, page_pgdat(page))
+							!= lruvec, page);
 		lru = page_lru(page);
 		nr_pages = thp_nr_pages(page);
 

_

author maintainer
#23 lore raw

PATCH 12/19 mm/mlock: remove lru_lock on TestClearPageMlocked

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/mlock: remove lru_lock on TestClearPageMlocked

In the func munlock_vma_page, comments mentained lru_lock needed for serialization with split_huge_pages. But the page must be PageLocked as well as pages in split_huge_page series funcs. Thus the PageLocked is enough to serialize both funcs.

Further more, Hugh Dickins pointed: before splitting in split_huge_page_to_list, the page was unmap_page() to remove pmd/ptes which protect the page from munlock. Thus, no needs to guard __split_huge_page_tail for mlock clean, just keep the lru_lock there for isolation purpose.

LKP found a preempt issue on __mod_zone_page_state which need change to mod_zone_page_state. Thanks!

Cc Kirill A. Shutemov <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 mm/mlock.c |   26 +++++---------------------
 1 file changed, 5 insertions(+), 21 deletions(-)
mm/mlock.c renamed from mm/mlock.c~mm-mlock-remove-lru_lock-on-testclearpagemlocked +5 -21
@@ -187,40 +187,24 @@ static void __munlock_isolation_failed(s
 unsigned int munlock_vma_page(struct page *page)
 {
 	int nr_pages;
-	pg_data_t *pgdat = page_pgdat(page);
 
 	/* For try_to_munlock() and to serialize with page migration */
 	BUG_ON(!PageLocked(page));
-
 	VM_BUG_ON_PAGE(PageTail(page), page);
 
-	/*
-	 * Serialize with any parallel __split_huge_page_refcount() which
-	 * might otherwise copy PageMlocked to part of the tail pages before
-	 * we clear it in the head page. It also stabilizes thp_nr_pages().
-	 */
-	spin_lock_irq(&pgdat->lru_lock);
-
 	if (!TestClearPageMlocked(page)) {
 		/* Potentially, PTE-mapped THP: do not skip the rest PTEs */
-		nr_pages = 1;
-		goto unlock_out;
+		return 0;
 	}
 
 	nr_pages = thp_nr_pages(page);
-	__mod_zone_page_state(page_zone(page), NR_MLOCK, -nr_pages);
+	mod_zone_page_state(page_zone(page), NR_MLOCK, -nr_pages);
 
-	if (__munlock_isolate_lru_page(page, true)) {
-		spin_unlock_irq(&pgdat->lru_lock);
+	if (!isolate_lru_page(page))
 		__munlock_isolated_page(page);
-		goto out;
-	}
-	__munlock_isolation_failed(page);
-
-unlock_out:
-	spin_unlock_irq(&pgdat->lru_lock);
+	else
+		__munlock_isolation_failed(page);
 
-out:
 	return nr_pages - 1;
 }
 

_

author maintainer
#24 lore raw

PATCH 12/19 mm/mlock: remove lru_lock on TestClearPageMlocked

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/mlock: remove lru_lock on TestClearPageMlocked

In the func munlock_vma_page, comments mentained lru_lock needed for serialization with split_huge_pages. But the page must be PageLocked as well as pages in split_huge_page series funcs. Thus the PageLocked is enough to serialize both funcs.

Further more, Hugh Dickins pointed: before splitting in split_huge_page_to_list, the page was unmap_page() to remove pmd/ptes which protect the page from munlock. Thus, no needs to guard __split_huge_page_tail for mlock clean, just keep the lru_lock there for isolation purpose.

LKP found a preempt issue on __mod_zone_page_state which need change to mod_zone_page_state. Thanks!

Cc Kirill A. Shutemov <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 mm/mlock.c |   26 +++++---------------------
 1 file changed, 5 insertions(+), 21 deletions(-)
mm/mlock.c renamed from mm/mlock.c~mm-mlock-remove-lru_lock-on-testclearpagemlocked +5 -21
@@ -187,40 +187,24 @@ static void __munlock_isolation_failed(s
 unsigned int munlock_vma_page(struct page *page)
 {
 	int nr_pages;
-	pg_data_t *pgdat = page_pgdat(page);
 
 	/* For try_to_munlock() and to serialize with page migration */
 	BUG_ON(!PageLocked(page));
-
 	VM_BUG_ON_PAGE(PageTail(page), page);
 
-	/*
-	 * Serialize with any parallel __split_huge_page_refcount() which
-	 * might otherwise copy PageMlocked to part of the tail pages before
-	 * we clear it in the head page. It also stabilizes thp_nr_pages().
-	 */
-	spin_lock_irq(&pgdat->lru_lock);
-
 	if (!TestClearPageMlocked(page)) {
 		/* Potentially, PTE-mapped THP: do not skip the rest PTEs */
-		nr_pages = 1;
-		goto unlock_out;
+		return 0;
 	}
 
 	nr_pages = thp_nr_pages(page);
-	__mod_zone_page_state(page_zone(page), NR_MLOCK, -nr_pages);
+	mod_zone_page_state(page_zone(page), NR_MLOCK, -nr_pages);
 
-	if (__munlock_isolate_lru_page(page, true)) {
-		spin_unlock_irq(&pgdat->lru_lock);
+	if (!isolate_lru_page(page))
 		__munlock_isolated_page(page);
-		goto out;
-	}
-	__munlock_isolation_failed(page);
-
-unlock_out:
-	spin_unlock_irq(&pgdat->lru_lock);
+	else
+		__munlock_isolation_failed(page);
 
-out:
 	return nr_pages - 1;
 }
 

_

author maintainer
#25 lore raw

PATCH 13/19 mm/mlock: remove __munlock_isolate_lru_page()

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/mlock: remove __munlock_isolate_lru_page()

__munlock_isolate_lru_page() only has one caller, remove it to clean up and simplify code.

Cc Kirill A. Shutemov <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 mm/mlock.c |   31 +++++++++----------------------
 1 file changed, 9 insertions(+), 22 deletions(-)
mm/mlock.c renamed from mm/mlock.c~mm-mlock-remove-__munlock_isolate_lru_page +9 -22
@@ -106,26 +106,6 @@ void mlock_vma_page(struct page *page)
 }
 
 /*
- * Isolate a page from LRU with optional get_page() pin.
- * Assumes lru_lock already held and page already pinned.
- */
-static bool __munlock_isolate_lru_page(struct page *page, bool getpage)
-{
-	if (PageLRU(page)) {
-		struct lruvec *lruvec;
-
-		lruvec = mem_cgroup_page_lruvec(page, page_pgdat(page));
-		if (getpage)
-			get_page(page);
-		ClearPageLRU(page);
-		del_page_from_lru_list(page, lruvec, page_lru(page));
-		return true;
-	}
-
-	return false;
-}
-
-/*
  * Finish munlock after successful page isolation
  *
  * Page must be locked. This is a wrapper for try_to_munlock()
@@ -296,9 +276,16 @@ static void __munlock_pagevec(struct pag
 			 * We already have pin from follow_page_mask()
 			 * so we can spare the get_page() here.
 			 */
-			if (__munlock_isolate_lru_page(page, false))
+			if (PageLRU(page)) {
+				struct lruvec *lruvec;
+
+				ClearPageLRU(page);
+				lruvec = mem_cgroup_page_lruvec(page,
+							page_pgdat(page));
+				del_page_from_lru_list(page, lruvec,
+							page_lru(page));
 				continue;
-			else
+			} else
 				__munlock_isolation_failed(page);
 		} else {
 			delta_munlocked++;

_

author maintainer
#26 lore raw

PATCH 13/19 mm/mlock: remove __munlock_isolate_lru_page()

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/mlock: remove __munlock_isolate_lru_page()

__munlock_isolate_lru_page() only has one caller, remove it to clean up and simplify code.

Cc Kirill A. Shutemov <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 mm/mlock.c |   31 +++++++++----------------------
 1 file changed, 9 insertions(+), 22 deletions(-)
mm/mlock.c renamed from mm/mlock.c~mm-mlock-remove-__munlock_isolate_lru_page +9 -22
@@ -106,26 +106,6 @@ void mlock_vma_page(struct page *page)
 }
 
 /*
- * Isolate a page from LRU with optional get_page() pin.
- * Assumes lru_lock already held and page already pinned.
- */
-static bool __munlock_isolate_lru_page(struct page *page, bool getpage)
-{
-	if (PageLRU(page)) {
-		struct lruvec *lruvec;
-
-		lruvec = mem_cgroup_page_lruvec(page, page_pgdat(page));
-		if (getpage)
-			get_page(page);
-		ClearPageLRU(page);
-		del_page_from_lru_list(page, lruvec, page_lru(page));
-		return true;
-	}
-
-	return false;
-}
-
-/*
  * Finish munlock after successful page isolation
  *
  * Page must be locked. This is a wrapper for try_to_munlock()
@@ -296,9 +276,16 @@ static void __munlock_pagevec(struct pag
 			 * We already have pin from follow_page_mask()
 			 * so we can spare the get_page() here.
 			 */
-			if (__munlock_isolate_lru_page(page, false))
+			if (PageLRU(page)) {
+				struct lruvec *lruvec;
+
+				ClearPageLRU(page);
+				lruvec = mem_cgroup_page_lruvec(page,
+							page_pgdat(page));
+				del_page_from_lru_list(page, lruvec,
+							page_lru(page));
 				continue;
-			else
+			} else
 				__munlock_isolation_failed(page);
 		} else {
 			delta_munlocked++;

_

author maintainer
#27 lore raw

PATCH 14/19 mm/lru: introduce TestClearPageLRU()

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/lru: introduce TestClearPageLRU()

Currently lru_lock still guards both lru list and page's lru bit, that's ok. but if we want to use specific lruvec lock on the page, we need to pin down the page's lruvec/memcg during locking. Just taking lruvec lock first may be undermined by the page's memcg charge/migration. To fix this problem, we will clear the lru bit out of locking and use it as pin down action to block the page isolation in memcg changing.

So now a standard steps of page isolation is following:

	1, get_page(); 	       #pin the page avoid to be free
	2, TestClearPageLRU(); #block other isolation like memcg change
	3, spin_lock on lru_lock; #serialize lru list access
	4, delete page from lru list;

This patch start with the first part: TestClearPageLRU, which combines PageLRU check and ClearPageLRU into a macro func TestClearPageLRU. This function will be used as page isolation precondition to prevent other isolations some where else. Then there are may !PageLRU page on lru list, need to remove BUG() checking accordingly.

There 2 rules for lru bit now:
1, the lru bit still indicate if a page on lru list, just in some
temporary moment(isolating), the page may have no lru bit when it's on lru list. but the page still must be on lru list when the lru bit set.
2, have to remove lru bit before delete it from lru list.

As Andrew Morton mentioned this change would dirty cacheline for a page which isn't on the LRU. But the loss would be acceptable in Rong Chen <[email protected]> report:
https://lore.kernel.org/lkml/20200304090301.GB5972@shao2-debian/

Cc Michal Hocko <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 include/linux/page-flags.h |    1 
 mm/mlock.c                 |    3 --
 mm/vmscan.c                |   39 +++++++++++++++++------------------
 3 files changed, 21 insertions(+), 22 deletions(-)
include/linux/page-flags.h renamed from include/linux/page-flags.h~mm-lru-introduce-testclearpagelru +1 -0
@@ -334,6 +334,7 @@ PAGEFLAG(Referenced, referenced, PF_HEAD
 PAGEFLAG(Dirty, dirty, PF_HEAD) TESTSCFLAG(Dirty, dirty, PF_HEAD)
 	__CLEARPAGEFLAG(Dirty, dirty, PF_HEAD)
 PAGEFLAG(LRU, lru, PF_HEAD) __CLEARPAGEFLAG(LRU, lru, PF_HEAD)
+	TESTCLEARFLAG(LRU, lru, PF_HEAD)
 PAGEFLAG(Active, active, PF_HEAD) __CLEARPAGEFLAG(Active, active, PF_HEAD)
 	TESTCLEARFLAG(Active, active, PF_HEAD)
 PAGEFLAG(Workingset, workingset, PF_HEAD)
mm/mlock.c renamed from mm/mlock.c~mm-lru-introduce-testclearpagelru +1 -2
@@ -276,10 +276,9 @@ static void __munlock_pagevec(struct pag
 			 * We already have pin from follow_page_mask()
 			 * so we can spare the get_page() here.
 			 */
-			if (PageLRU(page)) {
+			if (TestClearPageLRU(page)) {
 				struct lruvec *lruvec;
 
-				ClearPageLRU(page);
 				lruvec = mem_cgroup_page_lruvec(page,
 							page_pgdat(page));
 				del_page_from_lru_list(page, lruvec,
mm/vmscan.c renamed from mm/vmscan.c~mm-lru-introduce-testclearpagelru +19 -20
@@ -1541,7 +1541,7 @@ unsigned int reclaim_clean_pages_from_li
  */
 int __isolate_lru_page(struct page *page, isolate_mode_t mode)
 {
-	int ret = -EINVAL;
+	int ret = -EBUSY;
 
 	/* Only take pages on the LRU. */
 	if (!PageLRU(page))
@@ -1551,8 +1551,6 @@ int __isolate_lru_page(struct page *page
 	if (PageUnevictable(page) && !(mode & ISOLATE_UNEVICTABLE))
 		return ret;
 
-	ret = -EBUSY;
-
 	/*
 	 * To minimise LRU disruption, the caller can indicate that it only
 	 * wants to isolate pages it will be able to operate on without
@@ -1599,8 +1597,10 @@ int __isolate_lru_page(struct page *page
 		 * sure the page is not being freed elsewhere -- the
 		 * page release code relies on it.
 		 */
-		ClearPageLRU(page);
-		ret = 0;
+		if (TestClearPageLRU(page))
+			ret = 0;
+		else
+			put_page(page);
 	}
 
 	return ret;
@@ -1666,8 +1666,6 @@ static unsigned long isolate_lru_pages(u
 		page = lru_to_page(src);
 		prefetchw_prev_lru_page(page, src, flags);
 
-		VM_BUG_ON_PAGE(!PageLRU(page), page);
-
 		nr_pages = compound_nr(page);
 		total_scan += nr_pages;
 
@@ -1764,21 +1762,18 @@ int isolate_lru_page(struct page *page)
 	VM_BUG_ON_PAGE(!page_count(page), page);
 	WARN_RATELIMIT(PageTail(page), "trying to isolate tail page");
 
-	if (PageLRU(page)) {
+	if (TestClearPageLRU(page)) {
 		pg_data_t *pgdat = page_pgdat(page);
 		struct lruvec *lruvec;
 
-		spin_lock_irq(&pgdat->lru_lock);
+		get_page(page);
 		lruvec = mem_cgroup_page_lruvec(page, pgdat);
-		if (PageLRU(page)) {
-			int lru = page_lru(page);
-			get_page(page);
-			ClearPageLRU(page);
-			del_page_from_lru_list(page, lruvec, lru);
-			ret = 0;
-		}
+		spin_lock_irq(&pgdat->lru_lock);
+		del_page_from_lru_list(page, lruvec, page_lru(page));
 		spin_unlock_irq(&pgdat->lru_lock);
+		ret = 0;
 	}
+
 	return ret;
 }
 
@@ -4289,6 +4284,10 @@ void check_move_unevictable_pages(struct
 		nr_pages = thp_nr_pages(page);
 		pgscanned += nr_pages;
 
+		/* block memcg migration during page moving between lru */
+		if (!TestClearPageLRU(page))
+			continue;
+
 		if (pagepgdat != pgdat) {
 			if (pgdat)
 				spin_unlock_irq(&pgdat->lru_lock);
@@ -4297,10 +4296,7 @@ void check_move_unevictable_pages(struct
 		}
 		lruvec = mem_cgroup_page_lruvec(page, pgdat);
 
-		if (!PageLRU(page) || !PageUnevictable(page))
-			continue;
-
-		if (page_evictable(page)) {
+		if (page_evictable(page) && PageUnevictable(page)) {
 			enum lru_list lru = page_lru_base_type(page);
 
 			VM_BUG_ON_PAGE(PageActive(page), page);
@@ -4309,12 +4305,15 @@ void check_move_unevictable_pages(struct
 			add_page_to_lru_list(page, lruvec, lru);
 			pgrescued += nr_pages;
 		}
+		SetPageLRU(page);
 	}
 
 	if (pgdat) {
 		__count_vm_events(UNEVICTABLE_PGRESCUED, pgrescued);
 		__count_vm_events(UNEVICTABLE_PGSCANNED, pgscanned);
 		spin_unlock_irq(&pgdat->lru_lock);
+	} else if (pgscanned) {
+		count_vm_events(UNEVICTABLE_PGSCANNED, pgscanned);
 	}
 }
 EXPORT_SYMBOL_GPL(check_move_unevictable_pages);

_

author maintainer
#28 lore raw

PATCH 14/19 mm/lru: introduce TestClearPageLRU()

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/lru: introduce TestClearPageLRU()

Currently lru_lock still guards both lru list and page's lru bit, that's ok. but if we want to use specific lruvec lock on the page, we need to pin down the page's lruvec/memcg during locking. Just taking lruvec lock first may be undermined by the page's memcg charge/migration. To fix this problem, we will clear the lru bit out of locking and use it as pin down action to block the page isolation in memcg changing.

So now a standard steps of page isolation is following:

	1, get_page(); 	       #pin the page avoid to be free
	2, TestClearPageLRU(); #block other isolation like memcg change
	3, spin_lock on lru_lock; #serialize lru list access
	4, delete page from lru list;

This patch start with the first part: TestClearPageLRU, which combines PageLRU check and ClearPageLRU into a macro func TestClearPageLRU. This function will be used as page isolation precondition to prevent other isolations some where else. Then there are may !PageLRU page on lru list, need to remove BUG() checking accordingly.

There 2 rules for lru bit now:
1, the lru bit still indicate if a page on lru list, just in some
temporary moment(isolating), the page may have no lru bit when it's on lru list. but the page still must be on lru list when the lru bit set.
2, have to remove lru bit before delete it from lru list.

As Andrew Morton mentioned this change would dirty cacheline for a page which isn't on the LRU. But the loss would be acceptable in Rong Chen <[email protected]> report:
https://lore.kernel.org/lkml/20200304090301.GB5972@shao2-debian/

Cc Michal Hocko <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 include/linux/page-flags.h |    1 
 mm/mlock.c                 |    3 --
 mm/vmscan.c                |   39 +++++++++++++++++------------------
 3 files changed, 21 insertions(+), 22 deletions(-)
include/linux/page-flags.h renamed from include/linux/page-flags.h~mm-lru-introduce-testclearpagelru +1 -0
@@ -334,6 +334,7 @@ PAGEFLAG(Referenced, referenced, PF_HEAD
 PAGEFLAG(Dirty, dirty, PF_HEAD) TESTSCFLAG(Dirty, dirty, PF_HEAD)
 	__CLEARPAGEFLAG(Dirty, dirty, PF_HEAD)
 PAGEFLAG(LRU, lru, PF_HEAD) __CLEARPAGEFLAG(LRU, lru, PF_HEAD)
+	TESTCLEARFLAG(LRU, lru, PF_HEAD)
 PAGEFLAG(Active, active, PF_HEAD) __CLEARPAGEFLAG(Active, active, PF_HEAD)
 	TESTCLEARFLAG(Active, active, PF_HEAD)
 PAGEFLAG(Workingset, workingset, PF_HEAD)
mm/mlock.c renamed from mm/mlock.c~mm-lru-introduce-testclearpagelru +1 -2
@@ -276,10 +276,9 @@ static void __munlock_pagevec(struct pag
 			 * We already have pin from follow_page_mask()
 			 * so we can spare the get_page() here.
 			 */
-			if (PageLRU(page)) {
+			if (TestClearPageLRU(page)) {
 				struct lruvec *lruvec;
 
-				ClearPageLRU(page);
 				lruvec = mem_cgroup_page_lruvec(page,
 							page_pgdat(page));
 				del_page_from_lru_list(page, lruvec,
mm/vmscan.c renamed from mm/vmscan.c~mm-lru-introduce-testclearpagelru +19 -20
@@ -1541,7 +1541,7 @@ unsigned int reclaim_clean_pages_from_li
  */
 int __isolate_lru_page(struct page *page, isolate_mode_t mode)
 {
-	int ret = -EINVAL;
+	int ret = -EBUSY;
 
 	/* Only take pages on the LRU. */
 	if (!PageLRU(page))
@@ -1551,8 +1551,6 @@ int __isolate_lru_page(struct page *page
 	if (PageUnevictable(page) && !(mode & ISOLATE_UNEVICTABLE))
 		return ret;
 
-	ret = -EBUSY;
-
 	/*
 	 * To minimise LRU disruption, the caller can indicate that it only
 	 * wants to isolate pages it will be able to operate on without
@@ -1599,8 +1597,10 @@ int __isolate_lru_page(struct page *page
 		 * sure the page is not being freed elsewhere -- the
 		 * page release code relies on it.
 		 */
-		ClearPageLRU(page);
-		ret = 0;
+		if (TestClearPageLRU(page))
+			ret = 0;
+		else
+			put_page(page);
 	}
 
 	return ret;
@@ -1666,8 +1666,6 @@ static unsigned long isolate_lru_pages(u
 		page = lru_to_page(src);
 		prefetchw_prev_lru_page(page, src, flags);
 
-		VM_BUG_ON_PAGE(!PageLRU(page), page);
-
 		nr_pages = compound_nr(page);
 		total_scan += nr_pages;
 
@@ -1764,21 +1762,18 @@ int isolate_lru_page(struct page *page)
 	VM_BUG_ON_PAGE(!page_count(page), page);
 	WARN_RATELIMIT(PageTail(page), "trying to isolate tail page");
 
-	if (PageLRU(page)) {
+	if (TestClearPageLRU(page)) {
 		pg_data_t *pgdat = page_pgdat(page);
 		struct lruvec *lruvec;
 
-		spin_lock_irq(&pgdat->lru_lock);
+		get_page(page);
 		lruvec = mem_cgroup_page_lruvec(page, pgdat);
-		if (PageLRU(page)) {
-			int lru = page_lru(page);
-			get_page(page);
-			ClearPageLRU(page);
-			del_page_from_lru_list(page, lruvec, lru);
-			ret = 0;
-		}
+		spin_lock_irq(&pgdat->lru_lock);
+		del_page_from_lru_list(page, lruvec, page_lru(page));
 		spin_unlock_irq(&pgdat->lru_lock);
+		ret = 0;
 	}
+
 	return ret;
 }
 
@@ -4289,6 +4284,10 @@ void check_move_unevictable_pages(struct
 		nr_pages = thp_nr_pages(page);
 		pgscanned += nr_pages;
 
+		/* block memcg migration during page moving between lru */
+		if (!TestClearPageLRU(page))
+			continue;
+
 		if (pagepgdat != pgdat) {
 			if (pgdat)
 				spin_unlock_irq(&pgdat->lru_lock);
@@ -4297,10 +4296,7 @@ void check_move_unevictable_pages(struct
 		}
 		lruvec = mem_cgroup_page_lruvec(page, pgdat);
 
-		if (!PageLRU(page) || !PageUnevictable(page))
-			continue;
-
-		if (page_evictable(page)) {
+		if (page_evictable(page) && PageUnevictable(page)) {
 			enum lru_list lru = page_lru_base_type(page);
 
 			VM_BUG_ON_PAGE(PageActive(page), page);
@@ -4309,12 +4305,15 @@ void check_move_unevictable_pages(struct
 			add_page_to_lru_list(page, lruvec, lru);
 			pgrescued += nr_pages;
 		}
+		SetPageLRU(page);
 	}
 
 	if (pgdat) {
 		__count_vm_events(UNEVICTABLE_PGRESCUED, pgrescued);
 		__count_vm_events(UNEVICTABLE_PGSCANNED, pgscanned);
 		spin_unlock_irq(&pgdat->lru_lock);
+	} else if (pgscanned) {
+		count_vm_events(UNEVICTABLE_PGSCANNED, pgscanned);
 	}
 }
 EXPORT_SYMBOL_GPL(check_move_unevictable_pages);

_

author maintainer
#29 lore raw

PATCH 15/19 mm/compaction: do page isolation first in compaction

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/compaction: do page isolation first in compaction

Currently, compaction would get the lru_lock and then do page isolation which works fine with pgdat->lru_lock, since any page isoltion would compete for the lru_lock. If we want to change to memcg lru_lock, we have to isolate the page before getting lru_lock, thus isoltion would block page's memcg change which relay on page isoltion too. Then we could safely use per memcg lru_lock later.

The new page isolation use previous introduced TestClearPageLRU() + pgdat lru locking which will be changed to memcg lru lock later.

Hugh Dickins <[email protected]> fixed following bugs in this patch's early version:

Fix lots of crashes under compaction load: isolate_migratepages_block() must clean up appropriately when rejecting a page, setting PageLRU again if it had been cleared; and a put_page() after get_page_unless_zero() cannot safely be done while holding locked_lruvec - it may turn out to be the final put_page(), which will take an lruvec lock when PageLRU.

And move __isolate_lru_page_prepare back after get_page_unless_zero to make trylock_page() safe: trylock_page() is not safe to use at this time:
its setting PG_locked can race with the page being freed or allocated ("Bad page"), and can also erase flags being set by one of those "sole owners" of a freshly allocated page who use non-atomic __SetPageFlag().

Cc Matthew Wilcox <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 include/linux/swap.h |    2 -
 mm/compaction.c      |   42 +++++++++++++++++++++++++++++++---------
 mm/vmscan.c          |   43 ++++++++++++++++++++---------------------
 3 files changed, 56 insertions(+), 31 deletions(-)
include/linux/swap.h renamed from include/linux/swap.h~mm-compaction-do-page-isolation-first-in-compaction +1 -1
@@ -356,7 +356,7 @@ extern void lru_cache_add_inactive_or_un
 extern unsigned long zone_reclaimable_pages(struct zone *zone);
 extern unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
 					gfp_t gfp_mask, nodemask_t *mask);
-extern int __isolate_lru_page(struct page *page, isolate_mode_t mode);
+extern int __isolate_lru_page_prepare(struct page *page, isolate_mode_t mode);
 extern unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,
 						  unsigned long nr_pages,
 						  gfp_t gfp_mask,
mm/compaction.c renamed from mm/compaction.c~mm-compaction-do-page-isolation-first-in-compaction +33 -9

86 lines not on this page · raw message

mm/vmscan.c renamed from mm/vmscan.c~mm-compaction-do-page-isolation-first-in-compaction +22 -21

71 lines not on this page · raw message

_

author maintainer
#30 lore raw

PATCH 15/19 mm/compaction: do page isolation first in compaction

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/compaction: do page isolation first in compaction

Currently, compaction would get the lru_lock and then do page isolation which works fine with pgdat->lru_lock, since any page isoltion would compete for the lru_lock. If we want to change to memcg lru_lock, we have to isolate the page before getting lru_lock, thus isoltion would block page's memcg change which relay on page isoltion too. Then we could safely use per memcg lru_lock later.

The new page isolation use previous introduced TestClearPageLRU() + pgdat lru locking which will be changed to memcg lru lock later.

Hugh Dickins <[email protected]> fixed following bugs in this patch's early version:

Fix lots of crashes under compaction load: isolate_migratepages_block() must clean up appropriately when rejecting a page, setting PageLRU again if it had been cleared; and a put_page() after get_page_unless_zero() cannot safely be done while holding locked_lruvec - it may turn out to be the final put_page(), which will take an lruvec lock when PageLRU.

And move __isolate_lru_page_prepare back after get_page_unless_zero to make trylock_page() safe: trylock_page() is not safe to use at this time:
its setting PG_locked can race with the page being freed or allocated ("Bad page"), and can also erase flags being set by one of those "sole owners" of a freshly allocated page who use non-atomic __SetPageFlag().

Cc Matthew Wilcox <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 include/linux/swap.h |    2 -
 mm/compaction.c      |   42 +++++++++++++++++++++++++++++++---------
 mm/vmscan.c          |   43 ++++++++++++++++++++---------------------
 3 files changed, 56 insertions(+), 31 deletions(-)
include/linux/swap.h renamed from include/linux/swap.h~mm-compaction-do-page-isolation-first-in-compaction +1 -1
@@ -356,7 +356,7 @@ extern void lru_cache_add_inactive_or_un
 extern unsigned long zone_reclaimable_pages(struct zone *zone);
 extern unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
 					gfp_t gfp_mask, nodemask_t *mask);
-extern int __isolate_lru_page(struct page *page, isolate_mode_t mode);
+extern int __isolate_lru_page_prepare(struct page *page, isolate_mode_t mode);
 extern unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,
 						  unsigned long nr_pages,
 						  gfp_t gfp_mask,
mm/compaction.c renamed from mm/compaction.c~mm-compaction-do-page-isolation-first-in-compaction +33 -9

86 lines not on this page · raw message

mm/vmscan.c renamed from mm/vmscan.c~mm-compaction-do-page-isolation-first-in-compaction +22 -21

71 lines not on this page · raw message

_

author maintainer
#31 lore raw

PATCH 16/19 mm/swap.c: serialize memcg changes in pagevec_lru_move_fn

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/swap.c: serialize memcg changes in pagevec_lru_move_fn

Hugh Dickins' found a memcg change bug on original version: If we want to change the pgdat->lru_lock to memcg's lruvec lock, we have to serialize mem_cgroup_move_account during pagevec_lru_move_fn. The possible bad scenario would like:

	cpu 0					cpu 1
lruvec = mem_cgroup_page_lruvec()
					if (!isolate_lru_page())
						mem_cgroup_move_account

spin_lock_irqsave(&lruvec->lru_lock <== wrong lock.

So we need TestClearPageLRU to block isolate_lru_page(), that serializes the memcg change. and then removing the PageLRU check in move_fn callee as the consequence.

__pagevec_lru_add_fn() is different from the others, because the pages it deals with are, by definition, not yet on the lru. TestClearPageLRU is not needed and would not work, so __pagevec_lru_add() goes its own way.

Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 mm/swap.c |   44 +++++++++++++++++++++++++++++++++++---------
 1 file changed, 35 insertions(+), 9 deletions(-)
mm/swap.c renamed from mm/swap.c~mm-swapc-serialize-memcg-changes-in-pagevec_lru_move_fn +35 -9
@@ -222,8 +222,14 @@ static void pagevec_lru_move_fn(struct p
 			spin_lock_irqsave(&pgdat->lru_lock, flags);
 		}
 
+		/* block memcg migration during page moving between lru */
+		if (!TestClearPageLRU(page))
+			continue;
+
 		lruvec = mem_cgroup_page_lruvec(page, pgdat);
 		(*move_fn)(page, lruvec);
+
+		SetPageLRU(page);
 	}
 	if (pgdat)
 		spin_unlock_irqrestore(&pgdat->lru_lock, flags);
@@ -233,7 +239,7 @@ static void pagevec_lru_move_fn(struct p
 
 static void pagevec_move_tail_fn(struct page *page, struct lruvec *lruvec)
 {
-	if (PageLRU(page) && !PageUnevictable(page)) {
+	if (!PageUnevictable(page)) {
 		del_page_from_lru_list(page, lruvec, page_lru(page));
 		ClearPageActive(page);
 		add_page_to_lru_list_tail(page, lruvec, page_lru(page));
@@ -306,7 +312,7 @@ void lru_note_cost_page(struct page *pag
 
 static void __activate_page(struct page *page, struct lruvec *lruvec)
 {
-	if (PageLRU(page) && !PageActive(page) && !PageUnevictable(page)) {
+	if (!PageActive(page) && !PageUnevictable(page)) {
 		int lru = page_lru_base_type(page);
 		int nr_pages = thp_nr_pages(page);
 
@@ -362,7 +368,8 @@ static void activate_page(struct page *p
 
 	page = compound_head(page);
 	spin_lock_irq(&pgdat->lru_lock);
-	__activate_page(page, mem_cgroup_page_lruvec(page, pgdat));
+	if (PageLRU(page))
+		__activate_page(page, mem_cgroup_page_lruvec(page, pgdat));
 	spin_unlock_irq(&pgdat->lru_lock);
 }
 #endif
@@ -519,9 +526,6 @@ static void lru_deactivate_file_fn(struc
 	bool active;
 	int nr_pages = thp_nr_pages(page);
 
-	if (!PageLRU(page))
-		return;
-
 	if (PageUnevictable(page))
 		return;
 
@@ -562,7 +566,7 @@ static void lru_deactivate_file_fn(struc
 
 static void lru_deactivate_fn(struct page *page, struct lruvec *lruvec)
 {
-	if (PageLRU(page) && PageActive(page) && !PageUnevictable(page)) {
+	if (PageActive(page) && !PageUnevictable(page)) {
 		int lru = page_lru_base_type(page);
 		int nr_pages = thp_nr_pages(page);
 
@@ -579,7 +583,7 @@ static void lru_deactivate_fn(struct pag
 
 static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec)
 {
-	if (PageLRU(page) && PageAnon(page) && PageSwapBacked(page) &&
+	if (PageAnon(page) && PageSwapBacked(page) &&
 	    !PageSwapCache(page) && !PageUnevictable(page)) {
 		bool active = PageActive(page);
 		int nr_pages = thp_nr_pages(page);
@@ -1021,7 +1025,29 @@ static void __pagevec_lru_add_fn(struct
  */
 void __pagevec_lru_add(struct pagevec *pvec)
 {
-	pagevec_lru_move_fn(pvec, __pagevec_lru_add_fn);
+	int i;
+	struct pglist_data *pgdat = NULL;
+	struct lruvec *lruvec;
+	unsigned long flags = 0;
+
+	for (i = 0; i < pagevec_count(pvec); i++) {
+		struct page *page = pvec->pages[i];
+		struct pglist_data *pagepgdat = page_pgdat(page);
+
+		if (pagepgdat != pgdat) {
+			if (pgdat)
+				spin_unlock_irqrestore(&pgdat->lru_lock, flags);
+			pgdat = pagepgdat;
+			spin_lock_irqsave(&pgdat->lru_lock, flags);
+		}
+
+		lruvec = mem_cgroup_page_lruvec(page, pgdat);
+		__pagevec_lru_add_fn(page, lruvec);
+	}
+	if (pgdat)
+		spin_unlock_irqrestore(&pgdat->lru_lock, flags);
+	release_pages(pvec->pages, pvec->nr);
+	pagevec_reinit(pvec);
 }
 
 /**

_

author maintainer
#32 lore raw

PATCH 16/19 mm/swap.c: serialize memcg changes in pagevec_lru_move_fn

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/swap.c: serialize memcg changes in pagevec_lru_move_fn

Hugh Dickins' found a memcg change bug on original version: If we want to change the pgdat->lru_lock to memcg's lruvec lock, we have to serialize mem_cgroup_move_account during pagevec_lru_move_fn. The possible bad scenario would like:

	cpu 0					cpu 1
lruvec = mem_cgroup_page_lruvec()
					if (!isolate_lru_page())
						mem_cgroup_move_account

spin_lock_irqsave(&lruvec->lru_lock <== wrong lock.

So we need TestClearPageLRU to block isolate_lru_page(), that serializes the memcg change. and then removing the PageLRU check in move_fn callee as the consequence.

__pagevec_lru_add_fn() is different from the others, because the pages it deals with are, by definition, not yet on the lru. TestClearPageLRU is not needed and would not work, so __pagevec_lru_add() goes its own way.

Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 mm/swap.c |   44 +++++++++++++++++++++++++++++++++++---------
 1 file changed, 35 insertions(+), 9 deletions(-)
mm/swap.c renamed from mm/swap.c~mm-swapc-serialize-memcg-changes-in-pagevec_lru_move_fn +35 -9
@@ -222,8 +222,14 @@ static void pagevec_lru_move_fn(struct p
 			spin_lock_irqsave(&pgdat->lru_lock, flags);
 		}
 
+		/* block memcg migration during page moving between lru */
+		if (!TestClearPageLRU(page))
+			continue;
+
 		lruvec = mem_cgroup_page_lruvec(page, pgdat);
 		(*move_fn)(page, lruvec);
+
+		SetPageLRU(page);
 	}
 	if (pgdat)
 		spin_unlock_irqrestore(&pgdat->lru_lock, flags);
@@ -233,7 +239,7 @@ static void pagevec_lru_move_fn(struct p
 
 static void pagevec_move_tail_fn(struct page *page, struct lruvec *lruvec)
 {
-	if (PageLRU(page) && !PageUnevictable(page)) {
+	if (!PageUnevictable(page)) {
 		del_page_from_lru_list(page, lruvec, page_lru(page));
 		ClearPageActive(page);
 		add_page_to_lru_list_tail(page, lruvec, page_lru(page));
@@ -306,7 +312,7 @@ void lru_note_cost_page(struct page *pag
 
 static void __activate_page(struct page *page, struct lruvec *lruvec)
 {
-	if (PageLRU(page) && !PageActive(page) && !PageUnevictable(page)) {
+	if (!PageActive(page) && !PageUnevictable(page)) {
 		int lru = page_lru_base_type(page);
 		int nr_pages = thp_nr_pages(page);
 
@@ -362,7 +368,8 @@ static void activate_page(struct page *p
 
 	page = compound_head(page);
 	spin_lock_irq(&pgdat->lru_lock);
-	__activate_page(page, mem_cgroup_page_lruvec(page, pgdat));
+	if (PageLRU(page))
+		__activate_page(page, mem_cgroup_page_lruvec(page, pgdat));
 	spin_unlock_irq(&pgdat->lru_lock);
 }
 #endif
@@ -519,9 +526,6 @@ static void lru_deactivate_file_fn(struc
 	bool active;
 	int nr_pages = thp_nr_pages(page);
 
-	if (!PageLRU(page))
-		return;
-
 	if (PageUnevictable(page))
 		return;
 
@@ -562,7 +566,7 @@ static void lru_deactivate_file_fn(struc
 
 static void lru_deactivate_fn(struct page *page, struct lruvec *lruvec)
 {
-	if (PageLRU(page) && PageActive(page) && !PageUnevictable(page)) {
+	if (PageActive(page) && !PageUnevictable(page)) {
 		int lru = page_lru_base_type(page);
 		int nr_pages = thp_nr_pages(page);
 
@@ -579,7 +583,7 @@ static void lru_deactivate_fn(struct pag
 
 static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec)
 {
-	if (PageLRU(page) && PageAnon(page) && PageSwapBacked(page) &&
+	if (PageAnon(page) && PageSwapBacked(page) &&
 	    !PageSwapCache(page) && !PageUnevictable(page)) {
 		bool active = PageActive(page);
 		int nr_pages = thp_nr_pages(page);
@@ -1021,7 +1025,29 @@ static void __pagevec_lru_add_fn(struct
  */
 void __pagevec_lru_add(struct pagevec *pvec)
 {
-	pagevec_lru_move_fn(pvec, __pagevec_lru_add_fn);
+	int i;
+	struct pglist_data *pgdat = NULL;
+	struct lruvec *lruvec;
+	unsigned long flags = 0;
+
+	for (i = 0; i < pagevec_count(pvec); i++) {
+		struct page *page = pvec->pages[i];
+		struct pglist_data *pagepgdat = page_pgdat(page);
+
+		if (pagepgdat != pgdat) {
+			if (pgdat)
+				spin_unlock_irqrestore(&pgdat->lru_lock, flags);
+			pgdat = pagepgdat;
+			spin_lock_irqsave(&pgdat->lru_lock, flags);
+		}
+
+		lruvec = mem_cgroup_page_lruvec(page, pgdat);
+		__pagevec_lru_add_fn(page, lruvec);
+	}
+	if (pgdat)
+		spin_unlock_irqrestore(&pgdat->lru_lock, flags);
+	release_pages(pvec->pages, pvec->nr);
+	pagevec_reinit(pvec);
 }
 
 /**

_

author maintainer
#33 lore raw

PATCH 17/19 mm/lru: replace pgdat lru_lock with lruvec lock

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/lru: replace pgdat lru_lock with lruvec lock

This patch moves per node lru_lock into lruvec, thus bring a lru_lock for each of memcg per node. So on a large machine, each of memcg don't have to suffer from per node pgdat->lru_lock competition. They could go fast with their self lru_lock.

After move memcg charge before lru inserting, page isolation could serialize page's memcg, then per memcg lruvec lock is stable and could replace per node lru lock.

In isolate_migratepages_block(), compact_unlock_should_abort and lock_page_lruvec_irqsave are open coded to work with compact_control. Also add a debug func in locking which may give some clues if there are sth out of hands.

Daniel Jordan's testing show 62% improvement on modified readtwice case on his 2P * 10 core * 2 HT broadwell box.
https://lore.kernel.org/lkml/[email protected]/

Hugh Dickins helped on the patch polish, thanks!

[[email protected]: fix comment typo]
[[email protected]: use page_memcg()]
Cc Rong Chen <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Yang Shi <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Wei Yang <[email protected]>

---

 include/linux/memcontrol.h |   58 +++++++++++++++++
 include/linux/mmzone.h     |    3 
 mm/compaction.c            |   56 ++++++++++------
 mm/huge_memory.c           |   11 +--
 mm/memcontrol.c            |   78 ++++++++++++++++++++++-
 mm/mlock.c                 |   22 ++++--
 mm/mmzone.c                |    1 
 mm/page_alloc.c            |    1 
 mm/swap.c                  |  116 +++++++++++++++++------------------
 mm/vmscan.c                |   55 +++++++---------
 10 files changed, 275 insertions(+), 126 deletions(-)
include/linux/memcontrol.h renamed from include/linux/memcontrol.h~mm-lru-replace-pgdat-lru_lock-with-lruvec-lock +58 -0
@@ -491,6 +491,19 @@ struct mem_cgroup *get_mem_cgroup_from_m
 
 struct mem_cgroup *get_mem_cgroup_from_page(struct page *page);
 
+struct lruvec *lock_page_lruvec(struct page *page);
+struct lruvec *lock_page_lruvec_irq(struct page *page);
+struct lruvec *lock_page_lruvec_irqsave(struct page *page,
+						unsigned long *flags);
+
+#ifdef CONFIG_DEBUG_VM
+void lruvec_memcg_debug(struct lruvec *lruvec, struct page *page);
+#else
+static inline void lruvec_memcg_debug(struct lruvec *lruvec, struct page *page)
+{
+}
+#endif
+
 static inline
 struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css){
 	return css ? container_of(css, struct mem_cgroup, css) : NULL;
@@ -996,6 +1009,31 @@ static inline void mem_cgroup_put(struct
 {
 }
 
+static inline struct lruvec *lock_page_lruvec(struct page *page)
+{
+	struct pglist_data *pgdat = page_pgdat(page);
+
+	spin_lock(&pgdat->__lruvec.lru_lock);
+	return &pgdat->__lruvec;
+}
+
+static inline struct lruvec *lock_page_lruvec_irq(struct page *page)
+{
+	struct pglist_data *pgdat = page_pgdat(page);
+
+	spin_lock_irq(&pgdat->__lruvec.lru_lock);
+	return &pgdat->__lruvec;
+}
+
+static inline struct lruvec *lock_page_lruvec_irqsave(struct page *page,
+		unsigned long *flagsp)
+{
+	struct pglist_data *pgdat = page_pgdat(page);
+
+	spin_lock_irqsave(&pgdat->__lruvec.lru_lock, *flagsp);
+	return &pgdat->__lruvec;
+}
+
 static inline struct mem_cgroup *
 mem_cgroup_iter(struct mem_cgroup *root,
 		struct mem_cgroup *prev,
@@ -1215,6 +1253,10 @@ static inline
 void count_memcg_event_mm(struct mm_struct *mm, enum vm_event_item idx)
 {
 }
+
+static inline void lruvec_memcg_debug(struct lruvec *lruvec, struct page *page)
+{
+}
 #endif /* CONFIG_MEMCG */
 
 /* idx can be of type enum memcg_stat_item or node_stat_item */
@@ -1296,6 +1338,22 @@ static inline struct lruvec *parent_lruv
 	return mem_cgroup_lruvec(memcg, lruvec_pgdat(lruvec));
 }
 
+static inline void unlock_page_lruvec(struct lruvec *lruvec)
+{
+	spin_unlock(&lruvec->lru_lock);
+}
+
+static inline void unlock_page_lruvec_irq(struct lruvec *lruvec)
+{
+	spin_unlock_irq(&lruvec->lru_lock);
+}
+
+static inline void unlock_page_lruvec_irqrestore(struct lruvec *lruvec,
+		unsigned long flags)
+{
+	spin_unlock_irqrestore(&lruvec->lru_lock, flags);
+}
+
 #ifdef CONFIG_CGROUP_WRITEBACK
 
 struct wb_domain *mem_cgroup_wb_domain(struct bdi_writeback *wb);
include/linux/mmzone.h renamed from include/linux/mmzone.h~mm-lru-replace-pgdat-lru_lock-with-lruvec-lock +2 -1
@@ -276,6 +276,8 @@ enum lruvec_flags {
 
 struct lruvec {
 	struct list_head		lists[NR_LRU_LISTS];
+	/* per lruvec lru_lock for memcg */
+	spinlock_t			lru_lock;
 	/*
 	 * These track the cost of reclaiming one LRU - file or anon -
 	 * over the other. As the observed cost of reclaiming one LRU
@@ -782,7 +784,6 @@ typedef struct pglist_data {
 
 	/* Write-intensive fields used by page reclaim */
 	ZONE_PADDING(_pad1_)
-	spinlock_t		lru_lock;
 
 #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT
 	/*
mm/compaction.c renamed from mm/compaction.c~mm-lru-replace-pgdat-lru_lock-with-lruvec-lock +36 -20

105 lines not on this page · raw message

mm/huge_memory.c renamed from mm/huge_memory.c~mm-lru-replace-pgdat-lru_lock-with-lruvec-lock +4 -7
@@ -2365,7 +2365,7 @@ static void lru_add_page_tail(struct pag
 	VM_BUG_ON_PAGE(!PageHead(head), head);
 	VM_BUG_ON_PAGE(PageCompound(tail), head);
 	VM_BUG_ON_PAGE(PageLRU(tail), head);
-	lockdep_assert_held(&lruvec_pgdat(lruvec)->lru_lock);
+	lockdep_assert_held(&lruvec->lru_lock);
 
 	if (list) {
 		/* page reclaim is reclaiming a huge page */
@@ -2449,7 +2449,6 @@ static void __split_huge_page(struct pag
 		pgoff_t end)
 {
 	struct page *head = compound_head(page);
-	pg_data_t *pgdat = page_pgdat(head);
 	struct lruvec *lruvec;
 	struct address_space *swap_cache = NULL;
 	unsigned long offset = 0;
@@ -2467,10 +2466,8 @@ static void __split_huge_page(struct pag
 		xa_lock(&swap_cache->i_pages);
 	}
 
-	/* prevent PageLRU to go away from under us, and freeze lru stats */
-	spin_lock(&pgdat->lru_lock);
-
-	lruvec = mem_cgroup_page_lruvec(head, pgdat);
+	/* lock lru list/PageCompound, ref freezed by page_ref_freeze */
+	lruvec = lock_page_lruvec(head);
 
 	for (i = nr - 1; i >= 1; i--) {
 		__split_huge_page_tail(head, i, lruvec, list);
@@ -2491,7 +2488,7 @@ static void __split_huge_page(struct pag
 	}
 
 	ClearPageCompound(head);
-	spin_unlock(&pgdat->lru_lock);
+	unlock_page_lruvec(lruvec);
 	/* Caller disabled irqs, so they are still disabled here */
 
 	split_page_owner(head, nr);
mm/memcontrol.c renamed from mm/memcontrol.c~mm-lru-replace-pgdat-lru_lock-with-lruvec-lock +75 -3

103 lines not on this page · raw message

mm/mlock.c renamed from mm/mlock.c~mm-lru-replace-pgdat-lru_lock-with-lruvec-lock +16 -6
@@ -262,12 +262,12 @@ static void __munlock_pagevec(struct pag
 	int nr = pagevec_count(pvec);
 	int delta_munlocked = -nr;
 	struct pagevec pvec_putback;
+	struct lruvec *lruvec = NULL;
 	int pgrescued = 0;
 
 	pagevec_init(&pvec_putback);
 
 	/* Phase 1: page isolation */
-	spin_lock_irq(&zone->zone_pgdat->lru_lock);
 	for (i = 0; i < nr; i++) {
 		struct page *page = pvec->pages[i];
 
@@ -277,10 +277,16 @@ static void __munlock_pagevec(struct pag
 			 * so we can spare the get_page() here.
 			 */
 			if (TestClearPageLRU(page)) {
-				struct lruvec *lruvec;
+				struct lruvec *new_lruvec;
+
+				new_lruvec = mem_cgroup_page_lruvec(page,
+						page_pgdat(page));
+				if (new_lruvec != lruvec) {
+					if (lruvec)
+						unlock_page_lruvec_irq(lruvec);
+					lruvec = lock_page_lruvec_irq(page);
+				}
 
-				lruvec = mem_cgroup_page_lruvec(page,
-							page_pgdat(page));
 				del_page_from_lru_list(page, lruvec,
 							page_lru(page));
 				continue;
@@ -299,8 +305,12 @@ static void __munlock_pagevec(struct pag
 		pagevec_add(&pvec_putback, pvec->pages[i]);
 		pvec->pages[i] = NULL;
 	}
-	__mod_zone_page_state(zone, NR_MLOCK, delta_munlocked);
-	spin_unlock_irq(&zone->zone_pgdat->lru_lock);
+	if (lruvec) {
+		__mod_zone_page_state(zone, NR_MLOCK, delta_munlocked);
+		unlock_page_lruvec_irq(lruvec);
+	} else if (delta_munlocked) {
+		mod_zone_page_state(zone, NR_MLOCK, delta_munlocked);
+	}
 
 	/* Now we can release pins of pages that we are not munlocking */
 	pagevec_release(&pvec_putback);
mm/mmzone.c renamed from mm/mmzone.c~mm-lru-replace-pgdat-lru_lock-with-lruvec-lock +1 -0
@@ -77,6 +77,7 @@ void lruvec_init(struct lruvec *lruvec)
 	enum lru_list lru;
 
 	memset(lruvec, 0, sizeof(struct lruvec));
+	spin_lock_init(&lruvec->lru_lock);
 
 	for_each_lru(lru)
 		INIT_LIST_HEAD(&lruvec->lists[lru]);
mm/page_alloc.c renamed from mm/page_alloc.c~mm-lru-replace-pgdat-lru_lock-with-lruvec-lock +0 -1
@@ -6870,7 +6870,6 @@ static void __meminit pgdat_init_interna
 	init_waitqueue_head(&pgdat->pfmemalloc_wait);
 
 	pgdat_page_ext_init(pgdat);
-	spin_lock_init(&pgdat->lru_lock);
 	lruvec_init(&pgdat->__lruvec);
 }
 
mm/swap.c renamed from mm/swap.c~mm-lru-replace-pgdat-lru_lock-with-lruvec-lock +58 -58

225 lines not on this page · raw message

mm/vmscan.c renamed from mm/vmscan.c~mm-lru-replace-pgdat-lru_lock-with-lruvec-lock +25 -30

169 lines not on this page · raw message

_

author maintainer
#34 lore raw

PATCH 17/19 mm/lru: replace pgdat lru_lock with lruvec lock

In reply to Andrew Morton

From: Alex Shi <[email protected]>
Subject: mm/lru: replace pgdat lru_lock with lruvec lock

This patch moves per node lru_lock into lruvec, thus bring a lru_lock for each of memcg per node. So on a large machine, each of memcg don't have to suffer from per node pgdat->lru_lock competition. They could go fast with their self lru_lock.

After move memcg charge before lru inserting, page isolation could serialize page's memcg, then per memcg lruvec lock is stable and could replace per node lru lock.

In isolate_migratepages_block(), compact_unlock_should_abort and lock_page_lruvec_irqsave are open coded to work with compact_control. Also add a debug func in locking which may give some clues if there are sth out of hands.

Daniel Jordan's testing show 62% improvement on modified readtwice case on his 2P * 10 core * 2 HT broadwell box.
https://lore.kernel.org/lkml/[email protected]/

Hugh Dickins helped on the patch polish, thanks!

[[email protected]: fix comment typo]
[[email protected]: use page_memcg()]
Cc Rong Chen <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Yang Shi <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Wei Yang <[email protected]>

---

 include/linux/memcontrol.h |   58 +++++++++++++++++
 include/linux/mmzone.h     |    3 
 mm/compaction.c            |   56 ++++++++++------
 mm/huge_memory.c           |   11 +--
 mm/memcontrol.c            |   78 ++++++++++++++++++++++-
 mm/mlock.c                 |   22 ++++--
 mm/mmzone.c                |    1 
 mm/page_alloc.c            |    1 
 mm/swap.c                  |  116 +++++++++++++++++------------------
 mm/vmscan.c                |   55 +++++++---------
 10 files changed, 275 insertions(+), 126 deletions(-)
include/linux/memcontrol.h renamed from include/linux/memcontrol.h~mm-lru-replace-pgdat-lru_lock-with-lruvec-lock +58 -0
@@ -491,6 +491,19 @@ struct mem_cgroup *get_mem_cgroup_from_m
 
 struct mem_cgroup *get_mem_cgroup_from_page(struct page *page);
 
+struct lruvec *lock_page_lruvec(struct page *page);
+struct lruvec *lock_page_lruvec_irq(struct page *page);
+struct lruvec *lock_page_lruvec_irqsave(struct page *page,
+						unsigned long *flags);
+
+#ifdef CONFIG_DEBUG_VM
+void lruvec_memcg_debug(struct lruvec *lruvec, struct page *page);
+#else
+static inline void lruvec_memcg_debug(struct lruvec *lruvec, struct page *page)
+{
+}
+#endif
+
 static inline
 struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css){
 	return css ? container_of(css, struct mem_cgroup, css) : NULL;
@@ -996,6 +1009,31 @@ static inline void mem_cgroup_put(struct
 {
 }
 
+static inline struct lruvec *lock_page_lruvec(struct page *page)
+{
+	struct pglist_data *pgdat = page_pgdat(page);
+
+	spin_lock(&pgdat->__lruvec.lru_lock);
+	return &pgdat->__lruvec;
+}
+
+static inline struct lruvec *lock_page_lruvec_irq(struct page *page)
+{
+	struct pglist_data *pgdat = page_pgdat(page);
+
+	spin_lock_irq(&pgdat->__lruvec.lru_lock);
+	return &pgdat->__lruvec;
+}
+
+static inline struct lruvec *lock_page_lruvec_irqsave(struct page *page,
+		unsigned long *flagsp)
+{
+	struct pglist_data *pgdat = page_pgdat(page);
+
+	spin_lock_irqsave(&pgdat->__lruvec.lru_lock, *flagsp);
+	return &pgdat->__lruvec;
+}
+
 static inline struct mem_cgroup *
 mem_cgroup_iter(struct mem_cgroup *root,
 		struct mem_cgroup *prev,
@@ -1215,6 +1253,10 @@ static inline
 void count_memcg_event_mm(struct mm_struct *mm, enum vm_event_item idx)
 {
 }
+
+static inline void lruvec_memcg_debug(struct lruvec *lruvec, struct page *page)
+{
+}
 #endif /* CONFIG_MEMCG */
 
 /* idx can be of type enum memcg_stat_item or node_stat_item */
@@ -1296,6 +1338,22 @@ static inline struct lruvec *parent_lruv
 	return mem_cgroup_lruvec(memcg, lruvec_pgdat(lruvec));
 }
 
+static inline void unlock_page_lruvec(struct lruvec *lruvec)
+{
+	spin_unlock(&lruvec->lru_lock);
+}
+
+static inline void unlock_page_lruvec_irq(struct lruvec *lruvec)
+{
+	spin_unlock_irq(&lruvec->lru_lock);
+}
+
+static inline void unlock_page_lruvec_irqrestore(struct lruvec *lruvec,
+		unsigned long flags)
+{
+	spin_unlock_irqrestore(&lruvec->lru_lock, flags);
+}
+
 #ifdef CONFIG_CGROUP_WRITEBACK
 
 struct wb_domain *mem_cgroup_wb_domain(struct bdi_writeback *wb);
include/linux/mmzone.h renamed from include/linux/mmzone.h~mm-lru-replace-pgdat-lru_lock-with-lruvec-lock +2 -1
@@ -276,6 +276,8 @@ enum lruvec_flags {
 
 struct lruvec {
 	struct list_head		lists[NR_LRU_LISTS];
+	/* per lruvec lru_lock for memcg */
+	spinlock_t			lru_lock;
 	/*
 	 * These track the cost of reclaiming one LRU - file or anon -
 	 * over the other. As the observed cost of reclaiming one LRU
@@ -782,7 +784,6 @@ typedef struct pglist_data {
 
 	/* Write-intensive fields used by page reclaim */
 	ZONE_PADDING(_pad1_)
-	spinlock_t		lru_lock;
 
 #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT
 	/*
mm/compaction.c renamed from mm/compaction.c~mm-lru-replace-pgdat-lru_lock-with-lruvec-lock +36 -20

105 lines not on this page · raw message

mm/huge_memory.c renamed from mm/huge_memory.c~mm-lru-replace-pgdat-lru_lock-with-lruvec-lock +4 -7
@@ -2365,7 +2365,7 @@ static void lru_add_page_tail(struct pag
 	VM_BUG_ON_PAGE(!PageHead(head), head);
 	VM_BUG_ON_PAGE(PageCompound(tail), head);
 	VM_BUG_ON_PAGE(PageLRU(tail), head);
-	lockdep_assert_held(&lruvec_pgdat(lruvec)->lru_lock);
+	lockdep_assert_held(&lruvec->lru_lock);
 
 	if (list) {
 		/* page reclaim is reclaiming a huge page */
@@ -2449,7 +2449,6 @@ static void __split_huge_page(struct pag
 		pgoff_t end)
 {
 	struct page *head = compound_head(page);
-	pg_data_t *pgdat = page_pgdat(head);
 	struct lruvec *lruvec;
 	struct address_space *swap_cache = NULL;
 	unsigned long offset = 0;
@@ -2467,10 +2466,8 @@ static void __split_huge_page(struct pag
 		xa_lock(&swap_cache->i_pages);
 	}
 
-	/* prevent PageLRU to go away from under us, and freeze lru stats */
-	spin_lock(&pgdat->lru_lock);
-
-	lruvec = mem_cgroup_page_lruvec(head, pgdat);
+	/* lock lru list/PageCompound, ref freezed by page_ref_freeze */
+	lruvec = lock_page_lruvec(head);
 
 	for (i = nr - 1; i >= 1; i--) {
 		__split_huge_page_tail(head, i, lruvec, list);
@@ -2491,7 +2488,7 @@ static void __split_huge_page(struct pag
 	}
 
 	ClearPageCompound(head);
-	spin_unlock(&pgdat->lru_lock);
+	unlock_page_lruvec(lruvec);
 	/* Caller disabled irqs, so they are still disabled here */
 
 	split_page_owner(head, nr);
mm/memcontrol.c renamed from mm/memcontrol.c~mm-lru-replace-pgdat-lru_lock-with-lruvec-lock +75 -3

103 lines not on this page · raw message

mm/mlock.c renamed from mm/mlock.c~mm-lru-replace-pgdat-lru_lock-with-lruvec-lock +16 -6
@@ -262,12 +262,12 @@ static void __munlock_pagevec(struct pag
 	int nr = pagevec_count(pvec);
 	int delta_munlocked = -nr;
 	struct pagevec pvec_putback;
+	struct lruvec *lruvec = NULL;
 	int pgrescued = 0;
 
 	pagevec_init(&pvec_putback);
 
 	/* Phase 1: page isolation */
-	spin_lock_irq(&zone->zone_pgdat->lru_lock);
 	for (i = 0; i < nr; i++) {
 		struct page *page = pvec->pages[i];
 
@@ -277,10 +277,16 @@ static void __munlock_pagevec(struct pag
 			 * so we can spare the get_page() here.
 			 */
 			if (TestClearPageLRU(page)) {
-				struct lruvec *lruvec;
+				struct lruvec *new_lruvec;
+
+				new_lruvec = mem_cgroup_page_lruvec(page,
+						page_pgdat(page));
+				if (new_lruvec != lruvec) {
+					if (lruvec)
+						unlock_page_lruvec_irq(lruvec);
+					lruvec = lock_page_lruvec_irq(page);
+				}
 
-				lruvec = mem_cgroup_page_lruvec(page,
-							page_pgdat(page));
 				del_page_from_lru_list(page, lruvec,
 							page_lru(page));
 				continue;
@@ -299,8 +305,12 @@ static void __munlock_pagevec(struct pag
 		pagevec_add(&pvec_putback, pvec->pages[i]);
 		pvec->pages[i] = NULL;
 	}
-	__mod_zone_page_state(zone, NR_MLOCK, delta_munlocked);
-	spin_unlock_irq(&zone->zone_pgdat->lru_lock);
+	if (lruvec) {
+		__mod_zone_page_state(zone, NR_MLOCK, delta_munlocked);
+		unlock_page_lruvec_irq(lruvec);
+	} else if (delta_munlocked) {
+		mod_zone_page_state(zone, NR_MLOCK, delta_munlocked);
+	}
 
 	/* Now we can release pins of pages that we are not munlocking */
 	pagevec_release(&pvec_putback);
mm/mmzone.c renamed from mm/mmzone.c~mm-lru-replace-pgdat-lru_lock-with-lruvec-lock +1 -0
@@ -77,6 +77,7 @@ void lruvec_init(struct lruvec *lruvec)
 	enum lru_list lru;
 
 	memset(lruvec, 0, sizeof(struct lruvec));
+	spin_lock_init(&lruvec->lru_lock);
 
 	for_each_lru(lru)
 		INIT_LIST_HEAD(&lruvec->lists[lru]);
mm/page_alloc.c renamed from mm/page_alloc.c~mm-lru-replace-pgdat-lru_lock-with-lruvec-lock +0 -1
@@ -6870,7 +6870,6 @@ static void __meminit pgdat_init_interna
 	init_waitqueue_head(&pgdat->pfmemalloc_wait);
 
 	pgdat_page_ext_init(pgdat);
-	spin_lock_init(&pgdat->lru_lock);
 	lruvec_init(&pgdat->__lruvec);
 }
 
mm/swap.c renamed from mm/swap.c~mm-lru-replace-pgdat-lru_lock-with-lruvec-lock +58 -58

225 lines not on this page · raw message

mm/vmscan.c renamed from mm/vmscan.c~mm-lru-replace-pgdat-lru_lock-with-lruvec-lock +25 -30

169 lines not on this page · raw message

_

author maintainer
#35 lore raw

PATCH 18/19 mm/lru: introduce relock_page_lruvec()

In reply to Andrew Morton

From: Alexander Duyck <[email protected]>
Subject: mm/lru: introduce relock_page_lruvec()

Add relock_page_lruvec() to replace repeated same code, no functional change.

When testing for relock we can avoid the need for RCU locking if we simply compare the page pgdat and memcg pointers versus those that the lruvec is holding. By doing this we can avoid the extra pointer walks and accesses of the memory cgroup.

In addition we can avoid the checks entirely if lruvec is currently NULL.

[[email protected]: use page_memcg()]
Cc Thomas Gleixner <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 include/linux/memcontrol.h |   52 +++++++++++++++++++++++++++++++++++
 mm/mlock.c                 |   11 -------
 mm/swap.c                  |   31 ++++----------------
 mm/vmscan.c                |   12 +-------
 4 files changed, 61 insertions(+), 45 deletions(-)
include/linux/memcontrol.h renamed from include/linux/memcontrol.h~mm-lru-introduce-the-relock_page_lruvec-function +52 -0
@@ -485,6 +485,22 @@ out:
 
 struct lruvec *mem_cgroup_page_lruvec(struct page *, struct pglist_data *);
 
+static inline bool lruvec_holds_page_lru_lock(struct page *page,
+					      struct lruvec *lruvec)
+{
+	pg_data_t *pgdat = page_pgdat(page);
+	const struct mem_cgroup *memcg;
+	struct mem_cgroup_per_node *mz;
+
+	if (mem_cgroup_disabled())
+		return lruvec == &pgdat->__lruvec;
+
+	mz = container_of(lruvec, struct mem_cgroup_per_node, lruvec);
+	memcg = page_memcg(page) ? : root_mem_cgroup;
+
+	return lruvec->pgdat == pgdat && mz->memcg == memcg;
+}
+
 struct mem_cgroup *mem_cgroup_from_task(struct task_struct *p);
 
 struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm);
@@ -984,6 +1000,14 @@ static inline struct lruvec *mem_cgroup_
 	return &pgdat->__lruvec;
 }
 
+static inline bool lruvec_holds_page_lru_lock(struct page *page,
+					      struct lruvec *lruvec)
+{
+	pg_data_t *pgdat = page_pgdat(page);
+
+	return lruvec == &pgdat->__lruvec;
+}
+
 static inline struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *memcg)
 {
 	return NULL;
@@ -1354,6 +1378,34 @@ static inline void unlock_page_lruvec_ir
 	spin_unlock_irqrestore(&lruvec->lru_lock, flags);
 }
 
+/* Don't lock again iff page's lruvec locked */
+static inline struct lruvec *relock_page_lruvec_irq(struct page *page,
+		struct lruvec *locked_lruvec)
+{
+	if (locked_lruvec) {
+		if (lruvec_holds_page_lru_lock(page, locked_lruvec))
+			return locked_lruvec;
+
+		unlock_page_lruvec_irq(locked_lruvec);
+	}
+
+	return lock_page_lruvec_irq(page);
+}
+
+/* Don't lock again iff page's lruvec locked */
+static inline struct lruvec *relock_page_lruvec_irqsave(struct page *page,
+		struct lruvec *locked_lruvec, unsigned long *flags)
+{
+	if (locked_lruvec) {
+		if (lruvec_holds_page_lru_lock(page, locked_lruvec))
+			return locked_lruvec;
+
+		unlock_page_lruvec_irqrestore(locked_lruvec, *flags);
+	}
+
+	return lock_page_lruvec_irqsave(page, flags);
+}
+
 #ifdef CONFIG_CGROUP_WRITEBACK
 
 struct wb_domain *mem_cgroup_wb_domain(struct bdi_writeback *wb);
mm/mlock.c renamed from mm/mlock.c~mm-lru-introduce-the-relock_page_lruvec-function +1 -10
@@ -277,16 +277,7 @@ static void __munlock_pagevec(struct pag
 			 * so we can spare the get_page() here.
 			 */
 			if (TestClearPageLRU(page)) {
-				struct lruvec *new_lruvec;
-
-				new_lruvec = mem_cgroup_page_lruvec(page,
-						page_pgdat(page));
-				if (new_lruvec != lruvec) {
-					if (lruvec)
-						unlock_page_lruvec_irq(lruvec);
-					lruvec = lock_page_lruvec_irq(page);
-				}
-
+				lruvec = relock_page_lruvec_irq(page, lruvec);
 				del_page_from_lru_list(page, lruvec,
 							page_lru(page));
 				continue;
mm/swap.c renamed from mm/swap.c~mm-lru-introduce-the-relock_page_lruvec-function +6 -25
@@ -210,19 +210,12 @@ static void pagevec_lru_move_fn(struct p
 
 	for (i = 0; i < pagevec_count(pvec); i++) {
 		struct page *page = pvec->pages[i];
-		struct lruvec *new_lruvec;
 
 		/* block memcg migration during page moving between lru */
 		if (!TestClearPageLRU(page))
 			continue;
 
-		new_lruvec = mem_cgroup_page_lruvec(page, page_pgdat(page));
-		if (lruvec != new_lruvec) {
-			if (lruvec)
-				unlock_page_lruvec_irqrestore(lruvec, flags);
-			lruvec = lock_page_lruvec_irqsave(page, &flags);
-		}
-
+		lruvec = relock_page_lruvec_irqsave(page, lruvec, &flags);
 		(*move_fn)(page, lruvec);
 
 		SetPageLRU(page);
@@ -918,17 +911,12 @@ void release_pages(struct page **pages,
 		}
 
 		if (PageLRU(page)) {
-			struct lruvec *new_lruvec;
+			struct lruvec *prev_lruvec = lruvec;
 
-			new_lruvec = mem_cgroup_page_lruvec(page,
-							page_pgdat(page));
-			if (new_lruvec != lruvec) {
-				if (lruvec)
-					unlock_page_lruvec_irqrestore(lruvec,
-									flags);
+			lruvec = relock_page_lruvec_irqsave(page, lruvec,
+									&flags);
+			if (prev_lruvec != lruvec)
 				lock_batch = 0;
-				lruvec = lock_page_lruvec_irqsave(page, &flags);
-			}
 
 			VM_BUG_ON_PAGE(!PageLRU(page), page);
 			__ClearPageLRU(page);
@@ -1033,15 +1021,8 @@ void __pagevec_lru_add(struct pagevec *p
 
 	for (i = 0; i < pagevec_count(pvec); i++) {
 		struct page *page = pvec->pages[i];
-		struct lruvec *new_lruvec;
-
-		new_lruvec = mem_cgroup_page_lruvec(page, page_pgdat(page));
-		if (lruvec != new_lruvec) {
-			if (lruvec)
-				unlock_page_lruvec_irqrestore(lruvec, flags);
-			lruvec = lock_page_lruvec_irqsave(page, &flags);
-		}
 
+		lruvec = relock_page_lruvec_irqsave(page, lruvec, &flags);
 		__pagevec_lru_add_fn(page, lruvec);
 	}
 	if (lruvec)
mm/vmscan.c renamed from mm/vmscan.c~mm-lru-introduce-the-relock_page_lruvec-function +2 -10
@@ -1883,8 +1883,7 @@ static unsigned noinline_for_stack move_
 		 * All pages were isolated from the same lruvec (and isolation
 		 * inhibits memcg migration).
 		 */
-		VM_BUG_ON_PAGE(mem_cgroup_page_lruvec(page, page_pgdat(page))
-							!= lruvec, page);
+		VM_BUG_ON_PAGE(!lruvec_holds_page_lru_lock(page, lruvec), page);
 		lru = page_lru(page);
 		nr_pages = thp_nr_pages(page);
 
@@ -4273,7 +4272,6 @@ void check_move_unevictable_pages(struct
 	for (i = 0; i < pvec->nr; i++) {
 		struct page *page = pvec->pages[i];
 		int nr_pages;
-		struct lruvec *new_lruvec;
 
 		if (PageTransTail(page))
 			continue;
@@ -4285,13 +4283,7 @@ void check_move_unevictable_pages(struct
 		if (!TestClearPageLRU(page))
 			continue;
 
-		new_lruvec = mem_cgroup_page_lruvec(page, page_pgdat(page));
-		if (lruvec != new_lruvec) {
-			if (lruvec)
-				unlock_page_lruvec_irq(lruvec);
-			lruvec = lock_page_lruvec_irq(page);
-		}
-
+		lruvec = relock_page_lruvec_irq(page, lruvec);
 		if (page_evictable(page) && PageUnevictable(page)) {
 			enum lru_list lru = page_lru_base_type(page);
 

_

author maintainer
#36 lore raw

PATCH 18/19 mm/lru: introduce relock_page_lruvec()

In reply to Andrew Morton

From: Alexander Duyck <[email protected]>
Subject: mm/lru: introduce relock_page_lruvec()

Add relock_page_lruvec() to replace repeated same code, no functional change.

When testing for relock we can avoid the need for RCU locking if we simply compare the page pgdat and memcg pointers versus those that the lruvec is holding. By doing this we can avoid the extra pointer walks and accesses of the memory cgroup.

In addition we can avoid the checks entirely if lruvec is currently NULL.

[[email protected]: use page_memcg()]
Cc Thomas Gleixner <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Jann Horn <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 include/linux/memcontrol.h |   52 +++++++++++++++++++++++++++++++++++
 mm/mlock.c                 |   11 -------
 mm/swap.c                  |   31 ++++----------------
 mm/vmscan.c                |   12 +-------
 4 files changed, 61 insertions(+), 45 deletions(-)
include/linux/memcontrol.h renamed from include/linux/memcontrol.h~mm-lru-introduce-the-relock_page_lruvec-function +52 -0
@@ -485,6 +485,22 @@ out:
 
 struct lruvec *mem_cgroup_page_lruvec(struct page *, struct pglist_data *);
 
+static inline bool lruvec_holds_page_lru_lock(struct page *page,
+					      struct lruvec *lruvec)
+{
+	pg_data_t *pgdat = page_pgdat(page);
+	const struct mem_cgroup *memcg;
+	struct mem_cgroup_per_node *mz;
+
+	if (mem_cgroup_disabled())
+		return lruvec == &pgdat->__lruvec;
+
+	mz = container_of(lruvec, struct mem_cgroup_per_node, lruvec);
+	memcg = page_memcg(page) ? : root_mem_cgroup;
+
+	return lruvec->pgdat == pgdat && mz->memcg == memcg;
+}
+
 struct mem_cgroup *mem_cgroup_from_task(struct task_struct *p);
 
 struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm);
@@ -984,6 +1000,14 @@ static inline struct lruvec *mem_cgroup_
 	return &pgdat->__lruvec;
 }
 
+static inline bool lruvec_holds_page_lru_lock(struct page *page,
+					      struct lruvec *lruvec)
+{
+	pg_data_t *pgdat = page_pgdat(page);
+
+	return lruvec == &pgdat->__lruvec;
+}
+
 static inline struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *memcg)
 {
 	return NULL;
@@ -1354,6 +1378,34 @@ static inline void unlock_page_lruvec_ir
 	spin_unlock_irqrestore(&lruvec->lru_lock, flags);
 }
 
+/* Don't lock again iff page's lruvec locked */
+static inline struct lruvec *relock_page_lruvec_irq(struct page *page,
+		struct lruvec *locked_lruvec)
+{
+	if (locked_lruvec) {
+		if (lruvec_holds_page_lru_lock(page, locked_lruvec))
+			return locked_lruvec;
+
+		unlock_page_lruvec_irq(locked_lruvec);
+	}
+
+	return lock_page_lruvec_irq(page);
+}
+
+/* Don't lock again iff page's lruvec locked */
+static inline struct lruvec *relock_page_lruvec_irqsave(struct page *page,
+		struct lruvec *locked_lruvec, unsigned long *flags)
+{
+	if (locked_lruvec) {
+		if (lruvec_holds_page_lru_lock(page, locked_lruvec))
+			return locked_lruvec;
+
+		unlock_page_lruvec_irqrestore(locked_lruvec, *flags);
+	}
+
+	return lock_page_lruvec_irqsave(page, flags);
+}
+
 #ifdef CONFIG_CGROUP_WRITEBACK
 
 struct wb_domain *mem_cgroup_wb_domain(struct bdi_writeback *wb);
mm/mlock.c renamed from mm/mlock.c~mm-lru-introduce-the-relock_page_lruvec-function +1 -10
@@ -277,16 +277,7 @@ static void __munlock_pagevec(struct pag
 			 * so we can spare the get_page() here.
 			 */
 			if (TestClearPageLRU(page)) {
-				struct lruvec *new_lruvec;
-
-				new_lruvec = mem_cgroup_page_lruvec(page,
-						page_pgdat(page));
-				if (new_lruvec != lruvec) {
-					if (lruvec)
-						unlock_page_lruvec_irq(lruvec);
-					lruvec = lock_page_lruvec_irq(page);
-				}
-
+				lruvec = relock_page_lruvec_irq(page, lruvec);
 				del_page_from_lru_list(page, lruvec,
 							page_lru(page));
 				continue;
mm/swap.c renamed from mm/swap.c~mm-lru-introduce-the-relock_page_lruvec-function +6 -25
@@ -210,19 +210,12 @@ static void pagevec_lru_move_fn(struct p
 
 	for (i = 0; i < pagevec_count(pvec); i++) {
 		struct page *page = pvec->pages[i];
-		struct lruvec *new_lruvec;
 
 		/* block memcg migration during page moving between lru */
 		if (!TestClearPageLRU(page))
 			continue;
 
-		new_lruvec = mem_cgroup_page_lruvec(page, page_pgdat(page));
-		if (lruvec != new_lruvec) {
-			if (lruvec)
-				unlock_page_lruvec_irqrestore(lruvec, flags);
-			lruvec = lock_page_lruvec_irqsave(page, &flags);
-		}
-
+		lruvec = relock_page_lruvec_irqsave(page, lruvec, &flags);
 		(*move_fn)(page, lruvec);
 
 		SetPageLRU(page);
@@ -918,17 +911,12 @@ void release_pages(struct page **pages,
 		}
 
 		if (PageLRU(page)) {
-			struct lruvec *new_lruvec;
+			struct lruvec *prev_lruvec = lruvec;
 
-			new_lruvec = mem_cgroup_page_lruvec(page,
-							page_pgdat(page));
-			if (new_lruvec != lruvec) {
-				if (lruvec)
-					unlock_page_lruvec_irqrestore(lruvec,
-									flags);
+			lruvec = relock_page_lruvec_irqsave(page, lruvec,
+									&flags);
+			if (prev_lruvec != lruvec)
 				lock_batch = 0;
-				lruvec = lock_page_lruvec_irqsave(page, &flags);
-			}
 
 			VM_BUG_ON_PAGE(!PageLRU(page), page);
 			__ClearPageLRU(page);
@@ -1033,15 +1021,8 @@ void __pagevec_lru_add(struct pagevec *p
 
 	for (i = 0; i < pagevec_count(pvec); i++) {
 		struct page *page = pvec->pages[i];
-		struct lruvec *new_lruvec;
-
-		new_lruvec = mem_cgroup_page_lruvec(page, page_pgdat(page));
-		if (lruvec != new_lruvec) {
-			if (lruvec)
-				unlock_page_lruvec_irqrestore(lruvec, flags);
-			lruvec = lock_page_lruvec_irqsave(page, &flags);
-		}
 
+		lruvec = relock_page_lruvec_irqsave(page, lruvec, &flags);
 		__pagevec_lru_add_fn(page, lruvec);
 	}
 	if (lruvec)
mm/vmscan.c renamed from mm/vmscan.c~mm-lru-introduce-the-relock_page_lruvec-function +2 -10
@@ -1883,8 +1883,7 @@ static unsigned noinline_for_stack move_
 		 * All pages were isolated from the same lruvec (and isolation
 		 * inhibits memcg migration).
 		 */
-		VM_BUG_ON_PAGE(mem_cgroup_page_lruvec(page, page_pgdat(page))
-							!= lruvec, page);
+		VM_BUG_ON_PAGE(!lruvec_holds_page_lru_lock(page, lruvec), page);
 		lru = page_lru(page);
 		nr_pages = thp_nr_pages(page);
 
@@ -4273,7 +4272,6 @@ void check_move_unevictable_pages(struct
 	for (i = 0; i < pvec->nr; i++) {
 		struct page *page = pvec->pages[i];
 		int nr_pages;
-		struct lruvec *new_lruvec;
 
 		if (PageTransTail(page))
 			continue;
@@ -4285,13 +4283,7 @@ void check_move_unevictable_pages(struct
 		if (!TestClearPageLRU(page))
 			continue;
 
-		new_lruvec = mem_cgroup_page_lruvec(page, page_pgdat(page));
-		if (lruvec != new_lruvec) {
-			if (lruvec)
-				unlock_page_lruvec_irq(lruvec);
-			lruvec = lock_page_lruvec_irq(page);
-		}
-
+		lruvec = relock_page_lruvec_irq(page, lruvec);
 		if (page_evictable(page) && PageUnevictable(page)) {
 			enum lru_list lru = page_lru_base_type(page);
 

_

author maintainer
#38 lore raw

PATCH 19/19 mm/lru: revise the comments of lru_lock

In reply to Andrew Morton

From: Hugh Dickins <[email protected]>
Subject: mm/lru: revise the comments of lru_lock

Since we changed the pgdat->lru_lock to lruvec->lru_lock, it's time to fix the incorrect comments in code. Also fixed some zone->lru_lock comment error from ancient time. etc.

I struggled to understand the comment above move_pages_to_lru() (surely it never calls page_referenced()), and eventually realized that most of it had got separated from shrink_active_list(): move that comment back.

Cc Tejun Heo <[email protected]>
Cc Andrey Ryabinin <[email protected]>
Cc Jann Horn <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc "Chen, Rong A" <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mika Penttilä <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vladimir Davydov <[email protected]>
Cc Wei Yang <[email protected]>
Cc Yang Shi <[email protected]>

---

 Documentation/admin-guide/cgroup-v1/memcg_test.rst |   15 ----
 Documentation/admin-guide/cgroup-v1/memory.rst     |   23 ++----
 Documentation/trace/events-kmem.rst                |    2 
 Documentation/vm/unevictable-lru.rst               |   22 ++---
 include/linux/mm_types.h                           |    2 
 include/linux/mmzone.h                             |    3 
 mm/filemap.c                                       |    4 -
 mm/rmap.c                                          |    4 -
 mm/vmscan.c                                        |   41 ++++++-----
 9 files changed, 51 insertions(+), 65 deletions(-)
Documentation/admin-guide/cgroup-v1/memcg_test.rst renamed from Documentation/admin-guide/cgroup-v1/memcg_test.rst~mm-lru-revise-the-comments-of-lru_lock +3 -12
@@ -133,18 +133,9 @@ Under below explanation, we assume CONFI
 
 8. LRU
 ======
-        Each memcg has its own private LRU. Now, its handling is under global
-	VM's control (means that it's handled under global pgdat->lru_lock).
-	Almost all routines around memcg's LRU is called by global LRU's
-	list management functions under pgdat->lru_lock.
-
-	A special function is mem_cgroup_isolate_pages(). This scans
-	memcg's private LRU and call __isolate_lru_page() to extract a page
-	from LRU.
-
-	(By __isolate_lru_page(), the page is removed from both of global and
-	private LRU.)
-
+	Each memcg has its own vector of LRUs (inactive anon, active anon,
+	inactive file, active file, unevictable) of pages from each node,
+	each LRU handled under a single lru_lock for that memcg and node.
 
 9. Typical Tests.
 =================
Documentation/admin-guide/cgroup-v1/memory.rst renamed from Documentation/admin-guide/cgroup-v1/memory.rst~mm-lru-revise-the-comments-of-lru_lock +10 -13
@@ -287,20 +287,17 @@ When oom event notifier is registered, e
 2.6 Locking
 -----------
 
-   lock_page_cgroup()/unlock_page_cgroup() should not be called under
-   the i_pages lock.
+Lock order is as follows:
 
-   Other lock order is following:
-
-   PG_locked.
-     mm->page_table_lock
-         pgdat->lru_lock
-	   lock_page_cgroup.
-
-  In many cases, just lock_page_cgroup() is called.
-
-  per-zone-per-cgroup LRU (cgroup's private LRU) is just guarded by
-  pgdat->lru_lock, it has no lock of its own.
+  Page lock (PG_locked bit of page->flags)
+    mm->page_table_lock or split pte_lock
+      lock_page_memcg (memcg->move_lock)
+        mapping->i_pages lock
+          lruvec->lru_lock.
+
+Per-node-per-memcgroup LRU (cgroup's private LRU) is guarded by
+lruvec->lru_lock; PG_lru bit of page->flags is cleared before
+isolating a page from its LRU under lruvec->lru_lock.
 
 2.7 Kernel Memory Extension (CONFIG_MEMCG_KMEM)
 -----------------------------------------------
Documentation/trace/events-kmem.rst renamed from Documentation/trace/events-kmem.rst~mm-lru-revise-the-comments-of-lru_lock +1 -1
@@ -69,7 +69,7 @@ When pages are freed in batch, the also
 Broadly speaking, pages are taken off the LRU lock in bulk and
 freed in batch with a page list. Significant amounts of activity here could
 indicate that the system is under memory pressure and can also indicate
-contention on the zone->lru_lock.
+contention on the lruvec->lru_lock.
 
 4. Per-CPU Allocator Activity
 =============================
Documentation/vm/unevictable-lru.rst renamed from Documentation/vm/unevictable-lru.rst~mm-lru-revise-the-comments-of-lru_lock +8 -14
@@ -33,7 +33,7 @@ reclaim in Linux.  The problems have bee
 memory x86_64 systems.
 
 To illustrate this with an example, a non-NUMA x86_64 platform with 128GB of
-main memory will have over 32 million 4k pages in a single zone.  When a large
+main memory will have over 32 million 4k pages in a single node.  When a large
 fraction of these pages are not evictable for any reason [see below], vmscan
 will spend a lot of time scanning the LRU lists looking for the small fraction
 of pages that are evictable.  This can result in a situation where all CPUs are
@@ -55,7 +55,7 @@ unevictable, either by definition or by
 The Unevictable Page List
 -------------------------
 
-The Unevictable LRU infrastructure consists of an additional, per-zone, LRU list
+The Unevictable LRU infrastructure consists of an additional, per-node, LRU list
 called the "unevictable" list and an associated page flag, PG_unevictable, to
 indicate that the page is being managed on the unevictable list.
 
@@ -84,15 +84,9 @@ The unevictable list does not differenti
 swap-backed pages.  This differentiation is only important while the pages are,
 in fact, evictable.
 
-The unevictable list benefits from the "arrayification" of the per-zone LRU
+The unevictable list benefits from the "arrayification" of the per-node LRU
 lists and statistics originally proposed and posted by Christoph Lameter.
 
-The unevictable list does not use the LRU pagevec mechanism. Rather,
-unevictable pages are placed directly on the page's zone's unevictable list
-under the zone lru_lock.  This allows us to prevent the stranding of pages on
-the unevictable list when one task has the page isolated from the LRU and other
-tasks are changing the "evictability" state of the page.
-
 
 Memory Control Group Interaction
 --------------------------------
@@ -101,8 +95,8 @@ The unevictable LRU facility interacts w
 memory controller; see Documentation/admin-guide/cgroup-v1/memory.rst] by extending the
 lru_list enum.
 
-The memory controller data structure automatically gets a per-zone unevictable
-list as a result of the "arrayification" of the per-zone LRU lists (one per
+The memory controller data structure automatically gets a per-node unevictable
+list as a result of the "arrayification" of the per-node LRU lists (one per
 lru_list enum element).  The memory controller tracks the movement of pages to
 and from the unevictable list.
 
@@ -196,7 +190,7 @@ for the sake of expediency, to leave a u
 active/inactive LRU lists for vmscan to deal with.  vmscan checks for such
 pages in all of the shrink_{active|inactive|page}_list() functions and will
 "cull" such pages that it encounters: that is, it diverts those pages to the
-unevictable list for the zone being scanned.
+unevictable list for the node being scanned.
 
 There may be situations where a page is mapped into a VM_LOCKED VMA, but the
 page is not marked as PG_mlocked.  Such pages will make it all the way to
@@ -328,7 +322,7 @@ If the page was NOT already mlocked, mlo
 page from the LRU, as it is likely on the appropriate active or inactive list
 at that time.  If the isolate_lru_page() succeeds, mlock_vma_page() will put
 back the page - by calling putback_lru_page() - which will notice that the page
-is now mlocked and divert the page to the zone's unevictable list.  If
+is now mlocked and divert the page to the node's unevictable list.  If
 mlock_vma_page() is unable to isolate the page from the LRU, vmscan will handle
 it later if and when it attempts to reclaim the page.
 
@@ -603,7 +597,7 @@ Some examples of these unevictable pages
      unevictable list in mlock_vma_page().
 
 shrink_inactive_list() also diverts any unevictable pages that it finds on the
-inactive lists to the appropriate zone's unevictable list.
+inactive lists to the appropriate node's unevictable list.
 
 shrink_inactive_list() should only see SHM_LOCK'd pages that became SHM_LOCK'd
 after shrink_active_list() had moved them to the inactive list, or pages mapped
include/linux/mm_types.h renamed from include/linux/mm_types.h~mm-lru-revise-the-comments-of-lru_lock +1 -1
@@ -79,7 +79,7 @@ struct page {
 		struct {	/* Page cache and anonymous pages */
 			/**
 			 * @lru: Pageout list, eg. active_list protected by
-			 * pgdat->lru_lock.  Sometimes used as a generic list
+			 * lruvec->lru_lock.  Sometimes used as a generic list
 			 * by the page owner.
 			 */
 			struct list_head lru;
include/linux/mmzone.h renamed from include/linux/mmzone.h~mm-lru-revise-the-comments-of-lru_lock +1 -2
@@ -113,8 +113,7 @@ static inline bool free_area_empty(struc
 struct pglist_data;
 
 /*
- * zone->lock and the zone lru_lock are two of the hottest locks in the kernel.
- * So add a wild amount of padding here to ensure that they fall into separate
+ * Add a wild amount of padding here to ensure datas fall into separate
  * cachelines.  There are very few zone structures in the machine, so space
  * consumption is not a concern here.
  */
mm/filemap.c renamed from mm/filemap.c~mm-lru-revise-the-comments-of-lru_lock +2 -2
@@ -102,8 +102,8 @@
  *    ->swap_lock		(try_to_unmap_one)
  *    ->private_lock		(try_to_unmap_one)
  *    ->i_pages lock		(try_to_unmap_one)
- *    ->pgdat->lru_lock		(follow_page->mark_page_accessed)
- *    ->pgdat->lru_lock		(check_pte_range->isolate_lru_page)
+ *    ->lruvec->lru_lock	(follow_page->mark_page_accessed)
+ *    ->lruvec->lru_lock	(check_pte_range->isolate_lru_page)
  *    ->private_lock		(page_remove_rmap->set_page_dirty)
  *    ->i_pages lock		(page_remove_rmap->set_page_dirty)
  *    bdi.wb->list_lock		(page_remove_rmap->set_page_dirty)
mm/rmap.c renamed from mm/rmap.c~mm-lru-revise-the-comments-of-lru_lock +2 -2
@@ -28,12 +28,12 @@
  *           hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
  *           anon_vma->rwsem
  *             mm->page_table_lock or pte_lock
- *               pgdat->lru_lock (in mark_page_accessed, isolate_lru_page)
  *               swap_lock (in swap_duplicate, swap_info_get)
  *                 mmlist_lock (in mmput, drain_mmlist and others)
  *                 mapping->private_lock (in __set_page_dirty_buffers)
- *                   mem_cgroup_{begin,end}_page_stat (memcg->move_lock)
+ *                   lock_page_memcg move_lock (in __set_page_dirty_buffers)
  *                     i_pages lock (widely used)
+ *                       lruvec->lru_lock (in lock_page_lruvec_irq)
  *                 inode->i_lock (in set_page_dirty's __mark_inode_dirty)
  *                 bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty)
  *                   sb_lock (within inode_lock in fs/fs-writeback.c)
mm/vmscan.c renamed from mm/vmscan.c~mm-lru-revise-the-comments-of-lru_lock +23 -18

68 lines not on this page · raw message

_

#39 lore raw
In reply to Andrew Morton
Quoted 4 lines · Andrew Morton

- more MM work: a memcg scalability improvememt

19 patches, based on 148842c98a24e508aecb929718818fbf4c2a6ff3.

With your re-send, I get all patches, but they don't actually apply cleanly.

Is that base correct?

I get

error: patch failed: mm/huge_memory.c:2750
error: mm/huge_memory.c: patch does not apply
Patch failed at 0004 mm/thp: narrow lru locking

for that patch "[patch 04/19] mm/thp: narrow lru locking", and that's definitely true: the patch fragment has

@@ -2750,7 +2751,7 @@ int split_huge_page_to_list(struct page

                                __dec_lruvec_page_state(head, NR_FILE_THPS);
                }

- __split_huge_page(page, list, end, flags);

+               __split_huge_page(page, list, end);
                ret = 0;
        } else {
                if (IS_ENABLED(CONFIG_DEBUG_VM) && mapcount) {

but that __dec_lruvec_page_state() conversion was done by your previous commit series.

So I have the feeling that what you actually mean by "base" isn't actually really the base for that series at all..

I will try to apply it on top of my merge of your previous series instead.

Linus

#40 lore raw
Quoted 2 lines · Linus Torvalds

I will try to apply it on top of my merge of your previous series instead.

Yes, then it applies cleanly. So apparently we just have different concepts of what really constitutes a "base" for applying your series.

Linus

author maintainer
#41 lore raw
Quoted 8 lines · On Tue, 15 Dec 2020 14:49:24 -0800 Linus Torvalds…
Quoted 2 lines · Linus Torvalds

I will try to apply it on top of my merge of your previous series instead.

Yes, then it applies cleanly. So apparently we just have different concepts of what really constitutes a "base" for applying your series.

oop, sorry, yes, the "based on" thing was wrong because I had two series in flight simultaneously. I've never tried that before..