mm: khugepaged: fix NR_FILE_PAGES and NR_SHMEM in collapse_file()

In META's fleet, we observed high-level cgroups showing zero file memcg stats while their descendants had non-zero values. Investigation using drgn revealed that these parent cgroups actually had negative file stats, aggregated from their children.

This issue became more frequent after deploying thp-always more widely, pointing to a correlation with THP file collapsing. The root cause is that collapse_file() assumes old folios and the new THP belong to the same node and memcg. When this assumption breaks, stats become skewed. The bug affects not just memcg stats but also per-numa stats, and not just NR_FILE_PAGES but also NR_SHMEM.

The assumption breaks in scenarios such as:

1. Small folios allocated on one node while the THP gets allocated on a different node.

2. A package downloader running in one cgroup populates the page cache, while a job in a different cgroup executes the downloaded binary.

3. A file shared between processes in different cgroups, where one process faults in the pages and khugepaged (or madvise(COLLAPSE)) collapses them on behalf of the other.

Fix the accounting by explicitly incrementing stats for the new THP and decrementing stats for the old folios being replaced.

Fixes f3f0e1d2150b ("khugepaged: add support of collapse for tmpfs/shmem pages")

Signed-off-by Shakeel Butt <[email protected]>

---

In META's fleet, we observed high-level cgroups showing zero file memcg stats while their descendants had non-zero values. Investigation using drgn revealed that these parent cgroups actually had negative file stats, aggregated from their children.

This issue became more frequent after deploying thp-always more widely, pointing to a correlation with THP file collapsing. The root cause is that collapse_file() assumes old folios and the new THP belong to the same node and memcg. When this assumption breaks, stats become skewed. The bug affects not just memcg stats but also per-numa stats, and not just NR_FILE_PAGES but also NR_SHMEM.

The assumption breaks in scenarios such as:

1. Small folios allocated on one node while the THP gets allocated on a different node.

2. A package downloader running in one cgroup populates the page cache, while a job in a different cgroup executes the downloaded binary.

3. A file shared between processes in different cgroups, where one process faults in the pages and khugepaged (or madvise(COLLAPSE)) collapses them on behalf of the other.

Fix the accounting by explicitly incrementing stats for the new THP and decrementing stats for the old folios being replaced.

Fixes f3f0e1d2150b ("khugepaged: add support of collapse for tmpfs/shmem pages")

Signed-off-by Shakeel Butt <[email protected]>

---

 mm/khugepaged.c | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)

mm/khugepaged.c +4 -7

index 1d994b6c58c6..fa1e57fd2c46 100644
@@ -2195,16 +2195,13 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,
 		xas_lock_irq(&xas);
 	}
 
-	if (is_shmem)
+	if (is_shmem) {
+		lruvec_stat_mod_folio(new_folio, NR_SHMEM, HPAGE_PMD_NR);
 		lruvec_stat_mod_folio(new_folio, NR_SHMEM_THPS, HPAGE_PMD_NR);
-	else
+	} else {
 		lruvec_stat_mod_folio(new_folio, NR_FILE_THPS, HPAGE_PMD_NR);
-
-	if (nr_none) {
-		lruvec_stat_mod_folio(new_folio, NR_FILE_PAGES, nr_none);
-		/* nr_none is always 0 for non-shmem. */
-		lruvec_stat_mod_folio(new_folio, NR_SHMEM, nr_none);
 	}
+	lruvec_stat_mod_folio(new_folio, NR_FILE_PAGES, HPAGE_PMD_NR);
 
 	/*
 	 * Mark new_folio as uptodate before inserting it into the

mm/khugepaged.c +0 -0

 		folio->mapping = NULL;
 		folio_clear_active(folio);
 		folio_clear_unevictable(folio);

Quoted 24 lines · Shakeel Butt

In META's fleet, we observed high-level cgroups showing zero file memcg stats while their descendants had non-zero values. Investigation using drgn revealed that these parent cgroups actually had negative file stats, aggregated from their children.

This issue became more frequent after deploying thp-always more widely, pointing to a correlation with THP file collapsing. The root cause is that collapse_file() assumes old folios and the new THP belong to the same node and memcg. When this assumption breaks, stats become skewed. The bug affects not just memcg stats but also per-numa stats, and not just NR_FILE_PAGES but also NR_SHMEM.

The assumption breaks in scenarios such as:

1. Small folios allocated on one node while the THP gets allocated on a different node.

2. A package downloader running in one cgroup populates the page cache, while a job in a different cgroup executes the downloaded binary.

3. A file shared between processes in different cgroups, where one process faults in the pages and khugepaged (or madvise(COLLAPSE)) collapses them on behalf of the other.

Quoted 6 lines

Fix the accounting by explicitly incrementing stats for the new THP and decrementing stats for the old folios being replaced.

Fixes f3f0e1d2150b ("khugepaged: add support of collapse for tmpfs/shmem pages")

Signed-off-by Shakeel Butt <[email protected]>

---

Thanks.

Reviewed-by Dev Jain <[email protected]>

Quoted 29 lines

  mm/khugepaged.c | 16 +++++++++-------
  1 file changed, 9 insertions(+), 7 deletions(-)

mm/khugepaged.c +4 -7

index 1d994b6c58c6..fa1e57fd2c46 100644
@@ -2195,16 +2195,13 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,
  		xas_lock_irq(&xas);
  	}
 
-	if (is_shmem)
+	if (is_shmem) {
+		lruvec_stat_mod_folio(new_folio, NR_SHMEM, HPAGE_PMD_NR);
  		lruvec_stat_mod_folio(new_folio, NR_SHMEM_THPS, HPAGE_PMD_NR);
-	else
+	} else {
  		lruvec_stat_mod_folio(new_folio, NR_FILE_THPS, HPAGE_PMD_NR);
-
-	if (nr_none) {
-		lruvec_stat_mod_folio(new_folio, NR_FILE_PAGES, nr_none);
-		/* nr_none is always 0 for non-shmem. */
-		lruvec_stat_mod_folio(new_folio, NR_SHMEM, nr_none);
  	}
+	lruvec_stat_mod_folio(new_folio, NR_FILE_PAGES, HPAGE_PMD_NR);
 
  	/*
  	 * Mark new_folio as uptodate before inserting it into the

mm/khugepaged.c static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, · line 2242

Quoted 8 lines

mm/khugepaged.c +5 -0

@@ -2238,6 +2235,11 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,
  	 */
  	list_for_each_entry_safe(folio, tmp, &pagelist, lru) {
  		list_del(&folio->lru);
+		lruvec_stat_mod_folio(folio, NR_FILE_PAGES,
+				      -folio_nr_pages(folio));
+		if (is_shmem)
+			lruvec_stat_mod_folio(folio, NR_SHMEM,
+					      -folio_nr_pages(folio));

I notice here that we don't need to do accounting for NR_SHMEM_THPS or NR_FILE_THPS - but the following bit:

if (folio_order(folio) == HPAGE_PMD_ORDER && folio->index == start)

in the khugepaged code, seems to suggest that we can reach this stat accounting path with a PMD order old folio, if folio->index != start. But this condition should not be possible; a folio is always order-aligned within the file, which means the folio->index here is PMD-aligned. The entry of collapse_file() asserts that start is also PMD-aligned (guaranteed

by thp_vma_allowable_order in khugepaged_scan_mm_slot). Therefore start must equal folio->index.

If I am not missing something here, I'll send a patch to convert this to a VM_WARN_ON.

Quoted 3 lines

mm/khugepaged.c +0 -0

  		folio->mapping = NULL;
  		folio_clear_active(folio);
  		folio_clear_unevictable(folio);

In META's fleet, we observed high-level cgroups showing zero file memcg stats while their descendants had non-zero values. Investigation using drgn revealed that these parent cgroups actually had negative file stats, aggregated from their children.

This issue became more frequent after deploying thp-always more widely, pointing to a correlation with THP file collapsing. The root cause is that collapse_file() assumes old folios and the new THP belong to the same node and memcg. When this assumption breaks, stats become skewed. The bug affects not just memcg stats but also per-numa stats, and not just NR_FILE_PAGES but also NR_SHMEM.

The assumption breaks in scenarios such as:

1. Small folios allocated on one node while the THP gets allocated on a different node.

2. A package downloader running in one cgroup populates the page cache, while a job in a different cgroup executes the downloaded binary.

3. A file shared between processes in different cgroups, where one process faults in the pages and khugepaged (or madvise(COLLAPSE)) collapses them on behalf of the other.

Fix the accounting by explicitly incrementing stats for the new THP and decrementing stats for the old folios being replaced.

Fixes f3f0e1d2150b ("khugepaged: add support of collapse for tmpfs/shmem pages")

Signed-off-by Shakeel Butt <[email protected]>

---
 mm/khugepaged.c | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)

In META's fleet, we observed high-level cgroups showing zero file memcg stats while their descendants had non-zero values. Investigation using drgn revealed that these parent cgroups actually had negative file stats, aggregated from their children.

This issue became more frequent after deploying thp-always more widely, pointing to a correlation with THP file collapsing. The root cause is that collapse_file() assumes old folios and the new THP belong to the same node and memcg. When this assumption breaks, stats become skewed. The bug affects not just memcg stats but also per-numa stats, and not just NR_FILE_PAGES but also NR_SHMEM.

The assumption breaks in scenarios such as:

1. Small folios allocated on one node while the THP gets allocated on a different node.

2. A package downloader running in one cgroup populates the page cache, while a job in a different cgroup executes the downloaded binary.

3. A file shared between processes in different cgroups, where one process faults in the pages and khugepaged (or madvise(COLLAPSE)) collapses them on behalf of the other.

Fix the accounting by explicitly incrementing stats for the new THP and decrementing stats for the old folios being replaced.

Fixes f3f0e1d2150b ("khugepaged: add support of collapse for tmpfs/shmem pages")

Signed-off-by Shakeel Butt <[email protected]>

In META's fleet, we observed high-level cgroups showing zero file memcg stats while their descendants had non-zero values. Investigation using drgn revealed that these parent cgroups actually had negative file stats, aggregated from their children.

This issue became more frequent after deploying thp-always more widely, pointing to a correlation with THP file collapsing. The root cause is that collapse_file() assumes old folios and the new THP belong to the same node and memcg. When this assumption breaks, stats become skewed. The bug affects not just memcg stats but also per-numa stats, and not just NR_FILE_PAGES but also NR_SHMEM.

The assumption breaks in scenarios such as:

1. Small folios allocated on one node while the THP gets allocated on a different node.

2. A package downloader running in one cgroup populates the page cache, while a job in a different cgroup executes the downloaded binary.

3. A file shared between processes in different cgroups, where one process faults in the pages and khugepaged (or madvise(COLLAPSE)) collapses them on behalf of the other.

Fix the accounting by explicitly incrementing stats for the new THP and decrementing stats for the old folios being replaced.

Fixes f3f0e1d2150b ("khugepaged: add support of collapse for tmpfs/shmem pages")

Signed-off-by Shakeel Butt <[email protected]>

In META's fleet, we observed high-level cgroups showing zero file memcg stats while their descendants had non-zero values. Investigation using drgn revealed that these parent cgroups actually had negative file stats, aggregated from their children.

This issue became more frequent after deploying thp-always more widely, pointing to a correlation with THP file collapsing. The root cause is that collapse_file() assumes old folios and the new THP belong to the same node and memcg. When this assumption breaks, stats become skewed. The bug affects not just memcg stats but also per-numa stats, and not just NR_FILE_PAGES but also NR_SHMEM.

The assumption breaks in scenarios such as:

1. Small folios allocated on one node while the THP gets allocated on a different node.

2. A package downloader running in one cgroup populates the page cache, while a job in a different cgroup executes the downloaded binary.

3. A file shared between processes in different cgroups, where one process faults in the pages and khugepaged (or madvise(COLLAPSE)) collapses them on behalf of the other.

Fix the accounting by explicitly incrementing stats for the new THP and decrementing stats for the old folios being replaced.

Fixes f3f0e1d2150b ("khugepaged: add support of collapse for tmpfs/shmem pages")

As the bug is 10 years old I think I'll queue this for 6.20(?)-rc1 with cc:stable. Just to get it a bit more time-under-test before -stable kernels pick it up. Sound OK?

In META's fleet, we observed high-level cgroups showing zero file memcg stats while their descendants had non-zero values. Investigation using drgn revealed that these parent cgroups actually had negative file stats, aggregated from their children.

This issue became more frequent after deploying thp-always more widely, pointing to a correlation with THP file collapsing. The root cause is that collapse_file() assumes old folios and the new THP belong to the same node and memcg. When this assumption breaks, stats become skewed. The bug affects not just memcg stats but also per-numa stats, and not just NR_FILE_PAGES but also NR_SHMEM.

The assumption breaks in scenarios such as:

1. Small folios allocated on one node while the THP gets allocated on a different node.

2. A package downloader running in one cgroup populates the page cache, while a job in a different cgroup executes the downloaded binary.

3. A file shared between processes in different cgroups, where one process faults in the pages and khugepaged (or madvise(COLLAPSE)) collapses them on behalf of the other.

Fix the accounting by explicitly incrementing stats for the new THP and decrementing stats for the old folios being replaced.

Fixes f3f0e1d2150b ("khugepaged: add support of collapse for tmpfs/shmem pages")

Signed-off-by Shakeel Butt <[email protected]>

---

Quoted 24 lines · Shakeel Butt

In META's fleet, we observed high-level cgroups showing zero file memcg stats while their descendants had non-zero values. Investigation using drgn revealed that these parent cgroups actually had negative file stats, aggregated from their children.

This issue became more frequent after deploying thp-always more widely, pointing to a correlation with THP file collapsing. The root cause is that collapse_file() assumes old folios and the new THP belong to the same node and memcg. When this assumption breaks, stats become skewed. The bug affects not just memcg stats but also per-numa stats, and not just NR_FILE_PAGES but also NR_SHMEM.

The assumption breaks in scenarios such as:

1. Small folios allocated on one node while the THP gets allocated on a different node.

2. A package downloader running in one cgroup populates the page cache, while a job in a different cgroup executes the downloaded binary.

3. A file shared between processes in different cgroups, where one process faults in the pages and khugepaged (or madvise(COLLAPSE)) collapses them on behalf of the other.

Quoted 6 lines

Fix the accounting by explicitly incrementing stats for the new THP and decrementing stats for the old folios being replaced.

Fixes f3f0e1d2150b ("khugepaged: add support of collapse for tmpfs/shmem pages")

Signed-off-by Shakeel Butt <[email protected]>

---

Acked-by David Hildenbrand (Red Hat) <[email protected]>