Skip to contentSkip to conversation

mm: account lazily freed anon pages in NR_FILE_PAGES

PATCH In review Yafang Shao 4 participants 9 messages Nov 5, 2020 – Nov 6, 2020

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mm: account lazily freed anon pages in NR_FILE_PAGES

The memory utilization (Used / Total) is used to monitor the memory pressure by us. If it is too high, it means the system may be OOM sooner or later when swap is off, then we will make adjustment on this system.

However, this method is broken since MADV_FREE is introduced, because these lazily free anonymous can be reclaimed under memory pressure while they are still accounted in NR_ANON_MAPPED.

Furthermore, since commit f7ad2a6cb9f7 ("mm: move MADV_FREE pages into LRU_INACTIVE_FILE list"), these lazily free anonymous pages are moved from anon lru list into file lru list. That means (Inactive(file) + Active(file)) may be much larger than Cached in /proc/meminfo. That makes our users confused.

So we'd better account the lazily freed anonoymous pages in NR_FILE_PAGES as well.

Cc Minchan Kim <[email protected]>
Cc Johannes Weiner <[email protected]>
Cc Michal Hocko <[email protected]>
---
 mm/memcontrol.c | 11 +++++++++--
 mm/rmap.c       | 26 ++++++++++++++++++--------
 mm/swap.c       |  2 ++
 mm/vmscan.c     |  2 ++
 4 files changed, 31 insertions(+), 10 deletions(-)
mm/memcontrol.c +9 -2
index 3dcbf24d2227..217a6f10fa8d 100644
@@ -5659,8 +5659,15 @@ static int mem_cgroup_move_account(struct page *page,
 
 	if (PageAnon(page)) {
 		if (page_mapped(page)) {
-			__mod_lruvec_state(from_vec, NR_ANON_MAPPED, -nr_pages);
-			__mod_lruvec_state(to_vec, NR_ANON_MAPPED, nr_pages);
+			if (!PageSwapBacked(page) && !PageSwapCache(page) &&
+			    !PageUnevictable(page)) {
+				__mod_lruvec_state(from_vec, NR_FILE_PAGES, -nr_pages);
+				__mod_lruvec_state(to_vec, NR_FILE_PAGES, nr_pages);
+			} else {
+				__mod_lruvec_state(from_vec, NR_ANON_MAPPED, -nr_pages);
+				__mod_lruvec_state(to_vec, NR_ANON_MAPPED, nr_pages);
+			}
+
 			if (PageTransHuge(page)) {
 				__mod_lruvec_state(from_vec, NR_ANON_THPS,
 						   -nr_pages);
mm/rmap.c +18 -8
index 1b84945d655c..690ca7ff2392 100644
@@ -1312,8 +1312,13 @@ static void page_remove_anon_compound_rmap(struct page *page)
 	if (unlikely(PageMlocked(page)))
 		clear_page_mlock(page);
 
-	if (nr)
-		__mod_lruvec_page_state(page, NR_ANON_MAPPED, -nr);
+	if (nr) {
+		if (PageLRU(page) && PageAnon(page) && !PageSwapBacked(page) &&
+		    !PageSwapCache(page) && !PageUnevictable(page))
+			__mod_lruvec_page_state(page, NR_FILE_PAGES, -nr);
+		else
+			__mod_lruvec_page_state(page, NR_ANON_MAPPED, -nr);
+	}
 }
 
 /**
@@ -1341,12 +1346,17 @@ void page_remove_rmap(struct page *page, bool compound)
 	if (!atomic_add_negative(-1, &page->_mapcount))
 		goto out;
 
-	/*
-	 * We use the irq-unsafe __{inc|mod}_zone_page_stat because
-	 * these counters are not modified in interrupt context, and
-	 * pte lock(a spinlock) is held, which implies preemption disabled.
-	 */
-	__dec_lruvec_page_state(page, NR_ANON_MAPPED);
+	if (PageLRU(page) && PageAnon(page) && !PageSwapBacked(page) &&
+	    !PageSwapCache(page) && !PageUnevictable(page)) {
+		__dec_lruvec_page_state(page, NR_FILE_PAGES);
+	} else {
+		/*
+		 * We use the irq-unsafe __{inc|mod}_zone_page_stat because
+		 * these counters are not modified in interrupt context, and
+		 * pte lock(a spinlock) is held, which implies preemption disabled.
+		 */
+		__dec_lruvec_page_state(page, NR_ANON_MAPPED);
+	}
 
 	if (unlikely(PageMlocked(page)))
 		clear_page_mlock(page);
mm/swap.c +2 -0
index 47a47681c86b..340c5276a0f3 100644
@@ -601,6 +601,7 @@ static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec,
 
 		del_page_from_lru_list(page, lruvec,
 				       LRU_INACTIVE_ANON + active);
+		__mod_lruvec_state(lruvec, NR_ANON_MAPPED, -nr_pages);
 		ClearPageActive(page);
 		ClearPageReferenced(page);
 		/*
@@ -610,6 +611,7 @@ static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec,
 		 */
 		ClearPageSwapBacked(page);
 		add_page_to_lru_list(page, lruvec, LRU_INACTIVE_FILE);
+		__mod_lruvec_state(lruvec, NR_FILE_PAGES, nr_pages);
 
 		__count_vm_events(PGLAZYFREE, nr_pages);
 		__count_memcg_events(lruvec_memcg(lruvec), PGLAZYFREE,
mm/vmscan.c +2 -0
index 1b8f0e059767..4821124c70f7 100644
@@ -1428,6 +1428,8 @@ static unsigned int shrink_page_list(struct list_head *page_list,
 				goto keep_locked;
 			}
 
+			mod_lruvec_page_state(page, NR_ANON_MAPPED, nr_pages);
+			mod_lruvec_page_state(page, NR_FILE_PAGES, -nr_pages);
 			count_vm_event(PGLAZYFREED);
 			count_memcg_page_event(page, PGLAZYFREED);
 		} else if (!mapping || !__remove_mapping(mapping, page, true,
Signature · 1 line
2.18.4
#1 lore raw
Quoted 8 lines · Yafang Shao

The memory utilization (Used / Total) is used to monitor the memory pressure by us. If it is too high, it means the system may be OOM sooner or later when swap is off, then we will make adjustment on this system.

However, this method is broken since MADV_FREE is introduced, because these lazily free anonymous can be reclaimed under memory pressure while they are still accounted in NR_ANON_MAPPED.

Quoted 8 lines

Furthermore, since commit f7ad2a6cb9f7 ("mm: move MADV_FREE pages into LRU_INACTIVE_FILE list"), these lazily free anonymous pages are moved from anon lru list into file lru list. That means (Inactive(file) + Active(file)) may be much larger than Cached in /proc/meminfo. That makes our users confused.

So we'd better account the lazily freed anonoymous pages in NR_FILE_PAGES as well.

Can you simply subtract lazyfree pages in the userspace? I am afraid your patch just makes the situation even more muddy. NR_ANON_MAPPED is really meant to tell how many anonymous pages are mapped. And MADV_FREE pages are mapped until they are freed. NR_*_FILE are reflecting size of LRU lists and NR_FILE_PAGES reflects the number of page cache pages but madvfree pages are not a page cache. They are aged together with file pages but they are not the same thing. Same like shmem pages are page cache that is living on anon LRUs.

Confusing? Tricky? Yes, likely. But I do not think we want to bend those counters even further.

Quoted 113 lines
Cc Minchan Kim <[email protected]>
Cc Johannes Weiner <[email protected]>
Cc Michal Hocko <[email protected]>
---
 mm/memcontrol.c | 11 +++++++++--
 mm/rmap.c       | 26 ++++++++++++++++++--------
 mm/swap.c       |  2 ++
 mm/vmscan.c     |  2 ++
 4 files changed, 31 insertions(+), 10 deletions(-)
mm/memcontrol.c +9 -2
index 3dcbf24d2227..217a6f10fa8d 100644
@@ -5659,8 +5659,15 @@ static int mem_cgroup_move_account(struct page *page,
 
 	if (PageAnon(page)) {
 		if (page_mapped(page)) {
-			__mod_lruvec_state(from_vec, NR_ANON_MAPPED, -nr_pages);
-			__mod_lruvec_state(to_vec, NR_ANON_MAPPED, nr_pages);
+			if (!PageSwapBacked(page) && !PageSwapCache(page) &&
+			    !PageUnevictable(page)) {
+				__mod_lruvec_state(from_vec, NR_FILE_PAGES, -nr_pages);
+				__mod_lruvec_state(to_vec, NR_FILE_PAGES, nr_pages);
+			} else {
+				__mod_lruvec_state(from_vec, NR_ANON_MAPPED, -nr_pages);
+				__mod_lruvec_state(to_vec, NR_ANON_MAPPED, nr_pages);
+			}
+
 			if (PageTransHuge(page)) {
 				__mod_lruvec_state(from_vec, NR_ANON_THPS,
 						   -nr_pages);
mm/rmap.c +18 -8
index 1b84945d655c..690ca7ff2392 100644
@@ -1312,8 +1312,13 @@ static void page_remove_anon_compound_rmap(struct page *page)
 	if (unlikely(PageMlocked(page)))
 		clear_page_mlock(page);
 
-	if (nr)
-		__mod_lruvec_page_state(page, NR_ANON_MAPPED, -nr);
+	if (nr) {
+		if (PageLRU(page) && PageAnon(page) && !PageSwapBacked(page) &&
+		    !PageSwapCache(page) && !PageUnevictable(page))
+			__mod_lruvec_page_state(page, NR_FILE_PAGES, -nr);
+		else
+			__mod_lruvec_page_state(page, NR_ANON_MAPPED, -nr);
+	}
 }
 
 /**
@@ -1341,12 +1346,17 @@ void page_remove_rmap(struct page *page, bool compound)
 	if (!atomic_add_negative(-1, &page->_mapcount))
 		goto out;
 
-	/*
-	 * We use the irq-unsafe __{inc|mod}_zone_page_stat because
-	 * these counters are not modified in interrupt context, and
-	 * pte lock(a spinlock) is held, which implies preemption disabled.
-	 */
-	__dec_lruvec_page_state(page, NR_ANON_MAPPED);
+	if (PageLRU(page) && PageAnon(page) && !PageSwapBacked(page) &&
+	    !PageSwapCache(page) && !PageUnevictable(page)) {
+		__dec_lruvec_page_state(page, NR_FILE_PAGES);
+	} else {
+		/*
+		 * We use the irq-unsafe __{inc|mod}_zone_page_stat because
+		 * these counters are not modified in interrupt context, and
+		 * pte lock(a spinlock) is held, which implies preemption disabled.
+		 */
+		__dec_lruvec_page_state(page, NR_ANON_MAPPED);
+	}
 
 	if (unlikely(PageMlocked(page)))
 		clear_page_mlock(page);
mm/swap.c +2 -0
index 47a47681c86b..340c5276a0f3 100644
@@ -601,6 +601,7 @@ static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec,
 
 		del_page_from_lru_list(page, lruvec,
 				       LRU_INACTIVE_ANON + active);
+		__mod_lruvec_state(lruvec, NR_ANON_MAPPED, -nr_pages);
 		ClearPageActive(page);
 		ClearPageReferenced(page);
 		/*
@@ -610,6 +611,7 @@ static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec,
 		 */
 		ClearPageSwapBacked(page);
 		add_page_to_lru_list(page, lruvec, LRU_INACTIVE_FILE);
+		__mod_lruvec_state(lruvec, NR_FILE_PAGES, nr_pages);
 
 		__count_vm_events(PGLAZYFREE, nr_pages);
 		__count_memcg_events(lruvec_memcg(lruvec), PGLAZYFREE,
mm/vmscan.c +2 -0
index 1b8f0e059767..4821124c70f7 100644
@@ -1428,6 +1428,8 @@ static unsigned int shrink_page_list(struct list_head *page_list,
 				goto keep_locked;
 			}
 
+			mod_lruvec_page_state(page, NR_ANON_MAPPED, nr_pages);
+			mod_lruvec_page_state(page, NR_FILE_PAGES, -nr_pages);
 			count_vm_event(PGLAZYFREED);
 			count_memcg_page_event(page, PGLAZYFREED);
 		} else if (!mapping || !__remove_mapping(mapping, page, true,
Signature · 1 line
2.18.4
Signature · 2 lines
Michal Hocko
SUSE Labs
#2 lore raw
Quoted 16 lines · Michal Hocko
Quoted 14 lines · Yafang Shao

The memory utilization (Used / Total) is used to monitor the memory pressure by us. If it is too high, it means the system may be OOM sooner or later when swap is off, then we will make adjustment on this system.

However, this method is broken since MADV_FREE is introduced, because these lazily free anonymous can be reclaimed under memory pressure while they are still accounted in NR_ANON_MAPPED.

Furthermore, since commit f7ad2a6cb9f7 ("mm: move MADV_FREE pages into LRU_INACTIVE_FILE list"), these lazily free anonymous pages are moved from anon lru list into file lru list. That means (Inactive(file) + Active(file)) may be much larger than Cached in /proc/meminfo. That makes our users confused.

Quoted 4 lines
Quoted 2 lines

So we'd better account the lazily freed anonoymous pages in NR_FILE_PAGES as well.

Can you simply subtract lazyfree pages in the userspace?

Could you pls. tell me how to subtract lazyfree pages in the userspace? Pls. note that we can't use (pglazyfree - pglazyfreed) because pglazyfreed is only counted in the regular reclaim path while the process exit path is not counted, that means we have to introduce another counter like LazyPage....

Quoted 129 lines

I am afraid your
patch just makes the situation even more muddy. NR_ANON_MAPPED is really meant to tell how many anonymous pages are mapped. And MADV_FREE pages are mapped until they are freed. NR_*_FILE are reflecting size of LRU lists and NR_FILE_PAGES reflects the number of page cache pages but madvfree pages are not a page cache. They are aged together with file pages but they are not the same thing. Same like shmem pages are page cache that is living on anon LRUs.

Confusing? Tricky? Yes, likely. But I do not think we want to bend those counters even further.

Quoted 111 lines
Cc Minchan Kim <[email protected]>
Cc Johannes Weiner <[email protected]>
Cc Michal Hocko <[email protected]>
---
 mm/memcontrol.c | 11 +++++++++--
 mm/rmap.c       | 26 ++++++++++++++++++--------
 mm/swap.c       |  2 ++
 mm/vmscan.c     |  2 ++
 4 files changed, 31 insertions(+), 10 deletions(-)
mm/memcontrol.c +9 -2
index 3dcbf24d2227..217a6f10fa8d 100644
@@ -5659,8 +5659,15 @@ static int mem_cgroup_move_account(struct page *page,
 
      if (PageAnon(page)) {
              if (page_mapped(page)) {
-                     __mod_lruvec_state(from_vec, NR_ANON_MAPPED, -nr_pages);
-                     __mod_lruvec_state(to_vec, NR_ANON_MAPPED, nr_pages);
+                     if (!PageSwapBacked(page) && !PageSwapCache(page) &&
+                         !PageUnevictable(page)) {
+                             __mod_lruvec_state(from_vec, NR_FILE_PAGES, -nr_pages);
+                             __mod_lruvec_state(to_vec, NR_FILE_PAGES, nr_pages);
+                     } else {
+                             __mod_lruvec_state(from_vec, NR_ANON_MAPPED, -nr_pages);
+                             __mod_lruvec_state(to_vec, NR_ANON_MAPPED, nr_pages);
+                     }
+
                      if (PageTransHuge(page)) {
                              __mod_lruvec_state(from_vec, NR_ANON_THPS,
                                                 -nr_pages);
mm/rmap.c +18 -8
index 1b84945d655c..690ca7ff2392 100644
@@ -1312,8 +1312,13 @@ static void page_remove_anon_compound_rmap(struct page *page)
      if (unlikely(PageMlocked(page)))
              clear_page_mlock(page);
 
-     if (nr)
-             __mod_lruvec_page_state(page, NR_ANON_MAPPED, -nr);
+     if (nr) {
+             if (PageLRU(page) && PageAnon(page) && !PageSwapBacked(page) &&
+                 !PageSwapCache(page) && !PageUnevictable(page))
+                     __mod_lruvec_page_state(page, NR_FILE_PAGES, -nr);
+             else
+                     __mod_lruvec_page_state(page, NR_ANON_MAPPED, -nr);
+     }
 }
 
 /**
@@ -1341,12 +1346,17 @@ void page_remove_rmap(struct page *page, bool compound)
      if (!atomic_add_negative(-1, &page->_mapcount))
              goto out;
 
-     /*
-      * We use the irq-unsafe __{inc|mod}_zone_page_stat because
-      * these counters are not modified in interrupt context, and
-      * pte lock(a spinlock) is held, which implies preemption disabled.
-      */
-     __dec_lruvec_page_state(page, NR_ANON_MAPPED);
+     if (PageLRU(page) && PageAnon(page) && !PageSwapBacked(page) &&
+         !PageSwapCache(page) && !PageUnevictable(page)) {
+             __dec_lruvec_page_state(page, NR_FILE_PAGES);
+     } else {
+             /*
+              * We use the irq-unsafe __{inc|mod}_zone_page_stat because
+              * these counters are not modified in interrupt context, and
+              * pte lock(a spinlock) is held, which implies preemption disabled.
+              */
+             __dec_lruvec_page_state(page, NR_ANON_MAPPED);
+     }
 
      if (unlikely(PageMlocked(page)))
              clear_page_mlock(page);
mm/swap.c +2 -0
index 47a47681c86b..340c5276a0f3 100644
@@ -601,6 +601,7 @@ static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec,
 
              del_page_from_lru_list(page, lruvec,
                                     LRU_INACTIVE_ANON + active);
+             __mod_lruvec_state(lruvec, NR_ANON_MAPPED, -nr_pages);
              ClearPageActive(page);
              ClearPageReferenced(page);
              /*
@@ -610,6 +611,7 @@ static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec,
               */
              ClearPageSwapBacked(page);
              add_page_to_lru_list(page, lruvec, LRU_INACTIVE_FILE);
+             __mod_lruvec_state(lruvec, NR_FILE_PAGES, nr_pages);
 
              __count_vm_events(PGLAZYFREE, nr_pages);
              __count_memcg_events(lruvec_memcg(lruvec), PGLAZYFREE,
mm/vmscan.c +2 -1
index 1b8f0e059767..4821124c70f7 100644
@@ -1428,6 +1428,8 @@ static unsigned int shrink_page_list(struct list_head *page_list,
                              goto keep_locked;
                      }
 
+                     mod_lruvec_page_state(page, NR_ANON_MAPPED, nr_pages);
+                     mod_lruvec_page_state(page, NR_FILE_PAGES, -nr_pages);
                      count_vm_event(PGLAZYFREED);
                      count_memcg_page_event(page, PGLAZYFREED);
              } else if (!mapping || !__remove_mapping(mapping, page, true,
--
Quoted 2 lines

2.18.4

--
Michal Hocko
SUSE Labs

Signature · 2 lines
Thanks
Yafang
#3 lore raw
Quoted 20 lines · Yafang Shao
Quoted 19 lines · Michal Hocko
Quoted 16 lines · Yafang Shao

The memory utilization (Used / Total) is used to monitor the memory pressure by us. If it is too high, it means the system may be OOM sooner or later when swap is off, then we will make adjustment on this system.

However, this method is broken since MADV_FREE is introduced, because these lazily free anonymous can be reclaimed under memory pressure while they are still accounted in NR_ANON_MAPPED.

Furthermore, since commit f7ad2a6cb9f7 ("mm: move MADV_FREE pages into LRU_INACTIVE_FILE list"), these lazily free anonymous pages are moved from anon lru list into file lru list. That means (Inactive(file) + Active(file)) may be much larger than Cached in /proc/meminfo. That makes our users confused.

So we'd better account the lazily freed anonoymous pages in NR_FILE_PAGES as well.

Quoted 7 lines
Quoted 1 line

Can you simply subtract lazyfree pages in the userspace?

Could you pls. tell me how to subtract lazyfree pages in the userspace? Pls. note that we can't use (pglazyfree - pglazyfreed) because pglazyfreed is only counted in the regular reclaim path while the process exit path is not counted, that means we have to introduce another counter like LazyPage....

OK, I see your concern. I thought that we do update counters on a regular unmap. I do not see any reason why we shouldn't. It is indeed bad that we cannot tell the current number of lazy free pages by no means. Was this deliberate Minchan?

Signature · 2 lines
Michal Hocko
SUSE Labs
#4 lore raw
Quoted 29 lines · Michal Hocko
Quoted 20 lines · Yafang Shao
Quoted 19 lines · Michal Hocko
Quoted 16 lines · Yafang Shao

The memory utilization (Used / Total) is used to monitor the memory pressure by us. If it is too high, it means the system may be OOM sooner or later when swap is off, then we will make adjustment on this system.

However, this method is broken since MADV_FREE is introduced, because these lazily free anonymous can be reclaimed under memory pressure while they are still accounted in NR_ANON_MAPPED.

Furthermore, since commit f7ad2a6cb9f7 ("mm: move MADV_FREE pages into LRU_INACTIVE_FILE list"), these lazily free anonymous pages are moved from anon lru list into file lru list. That means (Inactive(file) + Active(file)) may be much larger than Cached in /proc/meminfo. That makes our users confused.

So we'd better account the lazily freed anonoymous pages in NR_FILE_PAGES as well.

Quoted 7 lines
Quoted 1 line

Can you simply subtract lazyfree pages in the userspace?

Could you pls. tell me how to subtract lazyfree pages in the userspace? Pls. note that we can't use (pglazyfree - pglazyfreed) because pglazyfreed is only counted in the regular reclaim path while the process exit path is not counted, that means we have to introduce another counter like LazyPage....

Quoted 4 lines

OK, I see your concern. I thought that we do update counters on a regular unmap. I do not see any reason why we shouldn't. It is indeed bad that we cannot tell the current number of lazy free pages by no means. Was this deliberate Minchan?

http://lkml.kernel.org/r/[email protected] has explained this. I completely forgot about the fact that those pages can be reused and lose their madvise status and we will learn about that only when checking the page.

Thanks Johannes for clarification.

Signature · 2 lines
Michal Hocko
SUSE Labs
#5 lore raw
In reply to Yafang Shao
Quoted 3 lines · Yafang Shao

The memory utilization (Used / Total) is used to monitor the memory pressure by us. If it is too high, it means the system may be OOM sooner or later when swap is off, then we will make adjustment on this system.

Hmm I would say that any system looking just at memory utilization (Used / Total) and not looking at file lru size is flawed. There's a reason MemAvailable exists, and does count file lru sizes.

Quoted 9 lines

However, this method is broken since MADV_FREE is introduced, because these lazily free anonymous can be reclaimed under memory pressure while they are still accounted in NR_ANON_MAPPED.

Furthermore, since commit f7ad2a6cb9f7 ("mm: move MADV_FREE pages into LRU_INACTIVE_FILE list"), these lazily free anonymous pages are moved from anon lru list into file lru list. That means (Inactive(file) + Active(file)) may be much larger than Cached in /proc/meminfo. That makes our users confused.

Yeah the counters are tricky for multiple reasons as Michal said...

Quoted 114 lines

So we'd better account the lazily freed anonoymous pages in NR_FILE_PAGES as well.

Cc Minchan Kim <[email protected]>
Cc Johannes Weiner <[email protected]>
Cc Michal Hocko <[email protected]>
---
  mm/memcontrol.c | 11 +++++++++--
  mm/rmap.c       | 26 ++++++++++++++++++--------
  mm/swap.c       |  2 ++
  mm/vmscan.c     |  2 ++
  4 files changed, 31 insertions(+), 10 deletions(-)
mm/memcontrol.c +9 -2
index 3dcbf24d2227..217a6f10fa8d 100644
@@ -5659,8 +5659,15 @@ static int mem_cgroup_move_account(struct page *page,
 
  	if (PageAnon(page)) {
  		if (page_mapped(page)) {
-			__mod_lruvec_state(from_vec, NR_ANON_MAPPED, -nr_pages);
-			__mod_lruvec_state(to_vec, NR_ANON_MAPPED, nr_pages);
+			if (!PageSwapBacked(page) && !PageSwapCache(page) &&
+			    !PageUnevictable(page)) {
+				__mod_lruvec_state(from_vec, NR_FILE_PAGES, -nr_pages);
+				__mod_lruvec_state(to_vec, NR_FILE_PAGES, nr_pages);
+			} else {
+				__mod_lruvec_state(from_vec, NR_ANON_MAPPED, -nr_pages);
+				__mod_lruvec_state(to_vec, NR_ANON_MAPPED, nr_pages);
+			}
+
  			if (PageTransHuge(page)) {
  				__mod_lruvec_state(from_vec, NR_ANON_THPS,
  						   -nr_pages);
mm/rmap.c +18 -8
index 1b84945d655c..690ca7ff2392 100644
@@ -1312,8 +1312,13 @@ static void page_remove_anon_compound_rmap(struct page *page)
  	if (unlikely(PageMlocked(page)))
  		clear_page_mlock(page);
 
-	if (nr)
-		__mod_lruvec_page_state(page, NR_ANON_MAPPED, -nr);
+	if (nr) {
+		if (PageLRU(page) && PageAnon(page) && !PageSwapBacked(page) &&
+		    !PageSwapCache(page) && !PageUnevictable(page))
+			__mod_lruvec_page_state(page, NR_FILE_PAGES, -nr);
+		else
+			__mod_lruvec_page_state(page, NR_ANON_MAPPED, -nr);
+	}
  }
 
  /**
@@ -1341,12 +1346,17 @@ void page_remove_rmap(struct page *page, bool compound)
  	if (!atomic_add_negative(-1, &page->_mapcount))
  		goto out;
 
-	/*
-	 * We use the irq-unsafe __{inc|mod}_zone_page_stat because
-	 * these counters are not modified in interrupt context, and
-	 * pte lock(a spinlock) is held, which implies preemption disabled.
-	 */
-	__dec_lruvec_page_state(page, NR_ANON_MAPPED);
+	if (PageLRU(page) && PageAnon(page) && !PageSwapBacked(page) &&
+	    !PageSwapCache(page) && !PageUnevictable(page)) {
+		__dec_lruvec_page_state(page, NR_FILE_PAGES);
+	} else {
+		/*
+		 * We use the irq-unsafe __{inc|mod}_zone_page_stat because
+		 * these counters are not modified in interrupt context, and
+		 * pte lock(a spinlock) is held, which implies preemption disabled.
+		 */
+		__dec_lruvec_page_state(page, NR_ANON_MAPPED);
+	}
 
  	if (unlikely(PageMlocked(page)))
  		clear_page_mlock(page);
mm/swap.c +2 -0
index 47a47681c86b..340c5276a0f3 100644
@@ -601,6 +601,7 @@ static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec,
 
  		del_page_from_lru_list(page, lruvec,
  				       LRU_INACTIVE_ANON + active);
+		__mod_lruvec_state(lruvec, NR_ANON_MAPPED, -nr_pages);
  		ClearPageActive(page);
  		ClearPageReferenced(page);
  		/*
@@ -610,6 +611,7 @@ static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec,
  		 */
  		ClearPageSwapBacked(page);
  		add_page_to_lru_list(page, lruvec, LRU_INACTIVE_FILE);
+		__mod_lruvec_state(lruvec, NR_FILE_PAGES, nr_pages);
 
  		__count_vm_events(PGLAZYFREE, nr_pages);
  		__count_memcg_events(lruvec_memcg(lruvec), PGLAZYFREE,
mm/vmscan.c +2 -0
index 1b8f0e059767..4821124c70f7 100644
@@ -1428,6 +1428,8 @@ static unsigned int shrink_page_list(struct list_head *page_list,
  				goto keep_locked;
  			}
 
+			mod_lruvec_page_state(page, NR_ANON_MAPPED, nr_pages);
+			mod_lruvec_page_state(page, NR_FILE_PAGES, -nr_pages);
  			count_vm_event(PGLAZYFREED);
  			count_memcg_page_event(page, PGLAZYFREED);
  		} else if (!mapping || !__remove_mapping(mapping, page, true,
#6 lore raw
Quoted 10 lines · Vlastimil Babka
Quoted 3 lines · Yafang Shao

The memory utilization (Used / Total) is used to monitor the memory pressure by us. If it is too high, it means the system may be OOM sooner or later when swap is off, then we will make adjustment on this system.

Hmm I would say that any system looking just at memory utilization (Used / Total) and not looking at file lru size is flawed.
There's a reason MemAvailable exists, and does count file lru sizes.

Right, the file lru size is counted in MemAvailable. MemAvailable and Used are two different metrics used by us. Both of them are useful, but the Used is not reliable anymore...

Quoted 127 lines
Quoted 9 lines

However, this method is broken since MADV_FREE is introduced, because these lazily free anonymous can be reclaimed under memory pressure while they are still accounted in NR_ANON_MAPPED.

Furthermore, since commit f7ad2a6cb9f7 ("mm: move MADV_FREE pages into LRU_INACTIVE_FILE list"), these lazily free anonymous pages are moved from anon lru list into file lru list. That means (Inactive(file) + Active(file)) may be much larger than Cached in /proc/meminfo. That makes our users confused.

Yeah the counters are tricky for multiple reasons as Michal said...

Quoted 114 lines

So we'd better account the lazily freed anonoymous pages in NR_FILE_PAGES as well.

Cc Minchan Kim <[email protected]>
Cc Johannes Weiner <[email protected]>
Cc Michal Hocko <[email protected]>
---
  mm/memcontrol.c | 11 +++++++++--
  mm/rmap.c       | 26 ++++++++++++++++++--------
  mm/swap.c       |  2 ++
  mm/vmscan.c     |  2 ++
  4 files changed, 31 insertions(+), 10 deletions(-)
mm/memcontrol.c +9 -2
index 3dcbf24d2227..217a6f10fa8d 100644
@@ -5659,8 +5659,15 @@ static int mem_cgroup_move_account(struct page *page,
 
      if (PageAnon(page)) {
              if (page_mapped(page)) {
-                     __mod_lruvec_state(from_vec, NR_ANON_MAPPED, -nr_pages);
-                     __mod_lruvec_state(to_vec, NR_ANON_MAPPED, nr_pages);
+                     if (!PageSwapBacked(page) && !PageSwapCache(page) &&
+                         !PageUnevictable(page)) {
+                             __mod_lruvec_state(from_vec, NR_FILE_PAGES, -nr_pages);
+                             __mod_lruvec_state(to_vec, NR_FILE_PAGES, nr_pages);
+                     } else {
+                             __mod_lruvec_state(from_vec, NR_ANON_MAPPED, -nr_pages);
+                             __mod_lruvec_state(to_vec, NR_ANON_MAPPED, nr_pages);
+                     }
+
                      if (PageTransHuge(page)) {
                              __mod_lruvec_state(from_vec, NR_ANON_THPS,
                                                 -nr_pages);
mm/rmap.c +18 -8
index 1b84945d655c..690ca7ff2392 100644
@@ -1312,8 +1312,13 @@ static void page_remove_anon_compound_rmap(struct page *page)
      if (unlikely(PageMlocked(page)))
              clear_page_mlock(page);
 
-     if (nr)
-             __mod_lruvec_page_state(page, NR_ANON_MAPPED, -nr);
+     if (nr) {
+             if (PageLRU(page) && PageAnon(page) && !PageSwapBacked(page) &&
+                 !PageSwapCache(page) && !PageUnevictable(page))
+                     __mod_lruvec_page_state(page, NR_FILE_PAGES, -nr);
+             else
+                     __mod_lruvec_page_state(page, NR_ANON_MAPPED, -nr);
+     }
  }
 
  /**
@@ -1341,12 +1346,17 @@ void page_remove_rmap(struct page *page, bool compound)
      if (!atomic_add_negative(-1, &page->_mapcount))
              goto out;
 
-     /*
-      * We use the irq-unsafe __{inc|mod}_zone_page_stat because
-      * these counters are not modified in interrupt context, and
-      * pte lock(a spinlock) is held, which implies preemption disabled.
-      */
-     __dec_lruvec_page_state(page, NR_ANON_MAPPED);
+     if (PageLRU(page) && PageAnon(page) && !PageSwapBacked(page) &&
+         !PageSwapCache(page) && !PageUnevictable(page)) {
+             __dec_lruvec_page_state(page, NR_FILE_PAGES);
+     } else {
+             /*
+              * We use the irq-unsafe __{inc|mod}_zone_page_stat because
+              * these counters are not modified in interrupt context, and
+              * pte lock(a spinlock) is held, which implies preemption disabled.
+              */
+             __dec_lruvec_page_state(page, NR_ANON_MAPPED);
+     }
 
      if (unlikely(PageMlocked(page)))
              clear_page_mlock(page);
mm/swap.c +2 -0
index 47a47681c86b..340c5276a0f3 100644
@@ -601,6 +601,7 @@ static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec,
 
              del_page_from_lru_list(page, lruvec,
                                     LRU_INACTIVE_ANON + active);
+             __mod_lruvec_state(lruvec, NR_ANON_MAPPED, -nr_pages);
              ClearPageActive(page);
              ClearPageReferenced(page);
              /*
@@ -610,6 +611,7 @@ static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec,
               */
              ClearPageSwapBacked(page);
              add_page_to_lru_list(page, lruvec, LRU_INACTIVE_FILE);
+             __mod_lruvec_state(lruvec, NR_FILE_PAGES, nr_pages);
 
              __count_vm_events(PGLAZYFREE, nr_pages);
              __count_memcg_events(lruvec_memcg(lruvec), PGLAZYFREE,
mm/vmscan.c +2 -0
index 1b8f0e059767..4821124c70f7 100644
@@ -1428,6 +1428,8 @@ static unsigned int shrink_page_list(struct list_head *page_list,
                              goto keep_locked;
                      }
 
+                     mod_lruvec_page_state(page, NR_ANON_MAPPED, nr_pages);
+                     mod_lruvec_page_state(page, NR_FILE_PAGES, -nr_pages);
                      count_vm_event(PGLAZYFREED);
                      count_memcg_page_event(page, PGLAZYFREED);
              } else if (!mapping || !__remove_mapping(mapping, page, true,
Signature · 2 lines
Thanks
Yafang
maintainer
#7 lore raw
In reply to Yafang Shao
Quoted 8 lines · Yafang Shao

The memory utilization (Used / Total) is used to monitor the memory pressure by us. If it is too high, it means the system may be OOM sooner or later when swap is off, then we will make adjustment on this system.

However, this method is broken since MADV_FREE is introduced, because these lazily free anonymous can be reclaimed under memory pressure while they are still accounted in NR_ANON_MAPPED.

Quoted 8 lines

Furthermore, since commit f7ad2a6cb9f7 ("mm: move MADV_FREE pages into LRU_INACTIVE_FILE list"), these lazily free anonymous pages are moved from anon lru list into file lru list. That means (Inactive(file) + Active(file)) may be much larger than Cached in /proc/meminfo. That makes our users confused.

So we'd better account the lazily freed anonoymous pages in NR_FILE_PAGES as well.

What about the share of pages that have been reused? After all, the idea behind deferred reclaim is cheap reuse of already allocated and faulted in pages.

Anywhere between 0% and 100% of MADV_FREEd pages may be dirty and need swap-out to reclaim. That means even after this patch, your formula would still have an error margin of 100%.

The tradeoff with saving the reuse fault and relying on the MMU is that the kernel simply *cannot do* lazy free accounting. Userspace needs to do it. E.g. if a malloc implementation or similar uses MADV_FREE, it has to keep track of what is and isn't used and make those stats available.

If that's not practical, I don't see an alternative to trapping minor faults upon page reuse, eating the additional TLB flush, and doing the accounting properly inside the kernel.

mm/rmap.c static void page_remove_anon_compound_rmap(struct page *page) · line 1320

Quoted 12 lines
mm/rmap.c +6 -2
@@ -1312,8 +1312,13 @@ static void page_remove_anon_compound_rmap(struct page *page)
 	if (unlikely(PageMlocked(page)))
 		clear_page_mlock(page);
 
-	if (nr)
-		__mod_lruvec_page_state(page, NR_ANON_MAPPED, -nr);
+	if (nr) {
+		if (PageLRU(page) && PageAnon(page) && !PageSwapBacked(page) &&
+		    !PageSwapCache(page) && !PageUnevictable(page))
+			__mod_lruvec_page_state(page, NR_FILE_PAGES, -nr);
+		else
+			__mod_lruvec_page_state(page, NR_ANON_MAPPED, -nr);

I don't think this would work. The page can be temporarily off-LRU for compaction, migration, reclaim etc. and then you'd misaccount it here.

#8 lore raw
Quoted 16 lines · Johannes Weiner
Quoted 14 lines · Yafang Shao

The memory utilization (Used / Total) is used to monitor the memory pressure by us. If it is too high, it means the system may be OOM sooner or later when swap is off, then we will make adjustment on this system.

However, this method is broken since MADV_FREE is introduced, because these lazily free anonymous can be reclaimed under memory pressure while they are still accounted in NR_ANON_MAPPED.

Furthermore, since commit f7ad2a6cb9f7 ("mm: move MADV_FREE pages into LRU_INACTIVE_FILE list"), these lazily free anonymous pages are moved from anon lru list into file lru list. That means (Inactive(file) + Active(file)) may be much larger than Cached in /proc/meminfo. That makes our users confused.

Quoted 7 lines
Quoted 2 lines

So we'd better account the lazily freed anonoymous pages in NR_FILE_PAGES as well.

What about the share of pages that have been reused? After all, the idea behind deferred reclaim is cheap reuse of already allocated and faulted in pages.

I missed the reuse case. Thanks for the explanation.

Quoted 11 lines

Anywhere between 0% and 100% of MADV_FREEd pages may be dirty and need swap-out to reclaim. That means even after this patch, your formula would still have an error margin of 100%.

The tradeoff with saving the reuse fault and relying on the MMU is that the kernel simply *cannot do* lazy free accounting. Userspace needs to do it. E.g. if a malloc implementation or similar uses MADV_FREE, it has to keep track of what is and isn't used and make those stats available.

If that's not practical,

That is not practical. The process which uses MADV_FREE can keep track of it, but other processes like monitor tools have no easier way to keep track of it. We can't give the userspace trouble.

Quoted 4 lines

I don't see an alternative to trapping minor
faults upon page reuse, eating the additional TLB flush, and doing the accounting properly inside the kernel.

I will try to analyze the details and find whether there is some way to track it in the kernel.

Quoted 4 lines
Quoted 4 lines
mm/rmap.c +0 -0
@@ -1312,8 +1312,13 @@ static void page_remove_anon_compound_rmap(struct page *page)
      if (unlikely(PageMlocked(page)))
              clear_page_mlock(page);
 
Quoted 11 lines

mm/rmap.c static void page_remove_anon_compound_rmap(struct page *page) · line 1320

Quoted 8 lines
mm/rmap.c +6 -2
-     if (nr)
-             __mod_lruvec_page_state(page, NR_ANON_MAPPED, -nr);
+     if (nr) {
+             if (PageLRU(page) && PageAnon(page) && !PageSwapBacked(page) &&
+                 !PageSwapCache(page) && !PageUnevictable(page))
+                     __mod_lruvec_page_state(page, NR_FILE_PAGES, -nr);
+             else
+                     __mod_lruvec_page_state(page, NR_ANON_MAPPED, -nr);

I don't think this would work. The page can be temporarily off-LRU for compaction, migration, reclaim etc. and then you'd misaccount it here.

Right, thanks for the clarification.

Signature · 2 lines
Thanks
Yafang