Skip to contentSkip to conversation

mm-remap-unused-subpages-to-shared-zeropage-when-splitting-isolated-thp.patch removed from -mm tree

merged mm-stable Andrew Morton 1 participant 1 message Sep 9, 2024

lore
Share & tools
lore

Andrew Morton

author tree notice Applied
#0 lore raw

The quilt patch titled
Subject: mm: remap unused subpages to shared zeropage when splitting isolated thp
has been removed from the -mm tree. Its filename was
mm-remap-unused-subpages-to-shared-zeropage-when-splitting-isolated-thp.patch

This patch was dropped because it was merged into the mm-stable branch of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

------------------------------------------------------
From: Yu Zhao <[email protected]>
Subject: mm: remap unused subpages to shared zeropage when splitting isolated thp
Date: Fri, 30 Aug 2024 11:03:36 +0100

Patch series "mm: split underused THPs", v5.

The current upstream default policy for THP is always. However, Meta uses madvise in production as the current THP=always policy vastly overprovisions THPs in sparsely accessed memory areas, resulting in excessive memory pressure and premature OOM killing. Using madvise + relying on khugepaged has certain drawbacks over THP=always. Using madvise hints mean THPs aren't "transparent" and require userspace changes. Waiting for khugepaged to scan memory and collapse pages into THP can be slow and unpredictable in terms of performance (i.e. you dont know when the collapse will happen), while production environments require predictable performance. If there is enough memory available, its better for both performance and predictability to have a THP from fault time, i.e. THP=always rather than wait for khugepaged to collapse it, and deal with sparsely populated THPs when the system is running out of memory.

This patch series is an attempt to mitigate the issue of running out of memory when THP is always enabled. During runtime whenever a THP is being faulted in or collapsed by khugepaged, the THP is added to a list. Whenever memory reclaim happens, the kernel runs the deferred_split shrinker which goes through the list and checks if the THP was underused, i.e. how many of the base 4K pages of the entire THP were zero-filled. If this number goes above a certain threshold, the shrinker will attempt to split that THP. Then at remap time, the pages that were zero-filled are mapped to the shared zeropage, hence saving memory. This method avoids the downside of wasting memory in areas where THP is sparsely filled when THP is always enabled, while still providing the upside THPs like reduced TLB misses without having to use madvise.

Meta production workloads that were CPU bound (>99% CPU utilzation) were tested with THP shrinker. The results after 2 hours are as follows:

                            | THP=madvise |  THP=always   | THP=always
                            |             |               | + shrinker series
                            |             |               | + max_ptes_none=409
-----------------------------------------------------------------------------
Performance improvement     |      -      |    +1.8%      |     +1.7%
(over THP=madvise)          |             |               |

-----------------------------------------------------------------------------
Memory usage | 54.6G | 58.8G (+7.7%) | 55.9G (+2.4%)
-----------------------------------------------------------------------------
max_ptes_none=409 means that any THP that has more than 409 out of 512 (80%) zero filled filled pages will be split.

To test out the patches, the below commands without the shrinker will invoke OOM killer immediately and kill stress, but will not fail with the shrinker:

echo 450 > /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none mkdir /sys/fs/cgroup/test
echo $$ > /sys/fs/cgroup/test/cgroup.procs
echo 20M > /sys/fs/cgroup/test/memory.max
echo 0 > /sys/fs/cgroup/test/memory.swap.max

# allocate twice memory.max for each stress worker and touch 40/512 of
# each THP, i.e. vm-stride 50K.
# With the shrinker, max_ptes_none of 470 and below won't invoke OOM
# killer.
# Without the shrinker, OOM killer is invoked immediately irrespective
# of max_ptes_none value and kills stress.
stress --vm 1 --vm-bytes 40M --vm-stride 50K

This patch (of 5):

Here being unused means containing only zeros and inaccessible to userspace. When splitting an isolated thp under reclaim or migration, the unused subpages can be mapped to the shared zeropage, hence saving memory. This is particularly helpful when the internal fragmentation of a thp is high, i.e. it has many untouched subpages.

This is also a prerequisite for THP low utilization shrinker which will be introduced in later patches, where underutilized THPs are split, and the zero-filled pages are freed saving memory.

Cc Alexander Zhu <[email protected]>
Cc Barry Song <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Domenico Cerasuolo <[email protected]>
Cc Johannes Weiner <[email protected]>
Cc Jonathan Corbet <[email protected]>
Cc Kairui Song <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Mike Rapoport <[email protected]>
Cc Nico Pache <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Roman Gushchin <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Shuang Zhai <[email protected]>
Cc Hugh Dickins <[email protected]>

---

 include/linux/rmap.h |    7 +++
 mm/huge_memory.c     |    8 ++--
 mm/migrate.c         |   72 +++++++++++++++++++++++++++++++++++------
 mm/migrate_device.c  |    4 +-
 4 files changed, 75 insertions(+), 16 deletions(-)
include/linux/rmap.h renamed from include/linux/rmap.h~mm-remap-unused-subpages-to-shared-zeropage-when-splitting-isolated-thp +6 -1
@@ -745,7 +745,12 @@ int folio_mkclean(struct folio *);
 int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff,
 		      struct vm_area_struct *vma);
 
-void remove_migration_ptes(struct folio *src, struct folio *dst, bool locked);
+enum rmp_flags {
+	RMP_LOCKED		= 1 << 0,
+	RMP_USE_SHARED_ZEROPAGE	= 1 << 1,
+};
+
+void remove_migration_ptes(struct folio *src, struct folio *dst, int flags);
 
 /*
  * rmap_walk_control: To control rmap traversing for specific needs
mm/huge_memory.c renamed from mm/huge_memory.c~mm-remap-unused-subpages-to-shared-zeropage-when-splitting-isolated-thp +4 -4
@@ -3004,7 +3004,7 @@ bool unmap_huge_pmd_locked(struct vm_are
 	return false;
 }
 
-static void remap_page(struct folio *folio, unsigned long nr)
+static void remap_page(struct folio *folio, unsigned long nr, int flags)
 {
 	int i = 0;
 
@@ -3012,7 +3012,7 @@ static void remap_page(struct folio *fol
 	if (!folio_test_anon(folio))
 		return;
 	for (;;) {
-		remove_migration_ptes(folio, folio, true);
+		remove_migration_ptes(folio, folio, RMP_LOCKED | flags);
 		i += folio_nr_pages(folio);
 		if (i >= nr)
 			break;
@@ -3222,7 +3222,7 @@ static void __split_huge_page(struct pag
 
 	if (nr_dropped)
 		shmem_uncharge(folio->mapping->host, nr_dropped);
-	remap_page(folio, nr);
+	remap_page(folio, nr, PageAnon(head) ? RMP_USE_SHARED_ZEROPAGE : 0);
 
 	/*
 	 * set page to its compound_head when split to non order-0 pages, so
@@ -3498,7 +3498,7 @@ fail:
 		if (mapping)
 			xas_unlock(&xas);
 		local_irq_enable();
-		remap_page(folio, folio_nr_pages(folio));
+		remap_page(folio, folio_nr_pages(folio), 0);
 		ret = -EAGAIN;
 	}
 
mm/migrate.c renamed from mm/migrate.c~mm-remap-unused-subpages-to-shared-zeropage-when-splitting-isolated-thp +63 -9

124 lines not on this page · raw message

mm/migrate_device.c renamed from mm/migrate_device.c~mm-remap-unused-subpages-to-shared-zeropage-when-splitting-isolated-thp +2 -2
@@ -424,7 +424,7 @@ static unsigned long migrate_device_unma
 			continue;
 
 		folio = page_folio(page);
-		remove_migration_ptes(folio, folio, false);
+		remove_migration_ptes(folio, folio, 0);
 
 		src_pfns[i] = 0;
 		folio_unlock(folio);
@@ -840,7 +840,7 @@ void migrate_device_finalize(unsigned lo
 			dst = src;
 		}
 
-		remove_migration_ptes(src, dst, false);
+		remove_migration_ptes(src, dst, 0);
 		folio_unlock(src);
 
 		if (folio_is_zone_device(src))

_

Patches currently in -mm which might be from [email protected] are