Skip to contentSkip to conversation

mm: support mTHP swap-in for zRAM-like swapfile

PATCH In review Barry Song 17 participants 60 messages Jul 26, 2024 – Aug 27, 2024

Subsystems (7)
lore
Share & tools
lore

Barry Song

author
#0 lore raw

v5 cover mm: support mTHP swap-in for zRAM-like swapfile

From: Barry Song <[email protected]>

In an embedded system like Android, more than half of anonymous memory is actually stored in swap devices such as zRAM. For instance, when an app is switched to the background, most of its memory might be swapped out.

Currently, we have mTHP features, but unfortunately, without support for large folio swap-ins, once those large folios are swapped out, we lose them immediately because mTHP is a one-way ticket.

This is unacceptable and reduces mTHP to merely a toy on systems with significant swap utilization.

This patch introduces mTHP swap-in support. For now, we limit mTHP swap-ins to contiguous swaps that were likely swapped out from mTHP as a whole.

Additionally, the current implementation only covers the SWAP_SYNCHRONOUS case. This is the simplest and most common use case, benefiting millions of Android phones and similar devices with minimal implementation cost. In this straightforward scenario, large folios are always exclusive, eliminating the need to handle complex rmap and swapcache issues.

It offers several benefits:
1. Enables bidirectional mTHP swapping, allowing retrieval of mTHP after swap-out and swap-in.
2. Eliminates fragmentation in swap slots and supports successful THP_SWPOUT without fragmentation. Based on the observed data [1] on Chris's and Ryan's THP swap allocation optimization, aligned swap-in plays a crucial role in the success of THP_SWPOUT.
3. Enables zRAM/zsmalloc to compress and decompress mTHP, reducing CPU usage and enhancing compression ratios significantly. We have another patchset to enable mTHP compression and decompression in zsmalloc/zRAM[2].

Using the readahead mechanism to decide whether to swap in mTHP doesn't seem to be an optimal approach. There's a critical distinction between pagecache and anonymous pages: pagecache can be evicted and later retrieved from disk, potentially becoming a mTHP upon retrieval, whereas anonymous pages must always reside in memory or swapfile. If we swap in small folios and identify adjacent memory suitable for swapping in as mTHP, those pages that have been converted to small folios may never transition to mTHP. The process of converting mTHP into small folios remains irreversible. This introduces the risk of losing all mTHP through several swap-out and swap-in cycles, let alone losing the benefits of defragmentation, improved compression ratios, and reduced CPU usage based on mTHP compression/decompression.

Conversely, in deploying mTHP on millions of real-world products with this feature in OPPO's out-of-tree code[3], we haven't observed any significant increase in memory footprint for 64KiB mTHP based on CONT-PTE on ARM64.

[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] OnePlusOSS / android_kernel_oneplus_sm8550
https://github.com/OnePlusOSS/android_kernel_oneplus_sm8550/tree/oneplus/sm8550_u_14.0.0_oneplus11

-v5:
* Add swap-in control policy according to Ying's proposal. Right now only "always" and "never" are supported, later we can extend to "auto";
* Fix the comment regarding zswap_never_enabled() according to Yosry;
* Filter out unaligned swp entries earlier;
* add mem_cgroup_swapin_uncharge_swap_nr() helper

-v4:
https://lore.kernel.org/linux-mm/[email protected]/

Many parts of v3 have been merged into the mm tree with the help on reviewing from Ryan, David, Ying and Chris etc. Thank you very much!
This is the final part to allocate large folios and map them.

* Use Yosry's zswap_never_enabled(), notice there is a bug. I put the bug fix in this v4 RFC though it should be fixed in Yosry's patch
* lots of code improvement (drop large stack, hold ptl etc) according to Yosry's and Ryan's feedback
* rebased on top of the latest mm-unstable and utilized some new helpers introduced recently.

-v3:
https://lore.kernel.org/linux-mm/[email protected]/
* avoid over-writing err in __swap_duplicate_nr, pointed out by Yosry, thanks!
* fix the issue folio is charged twice for do_swap_page, separating alloc_anon_folio and alloc_swap_folio as they have many differences now on
* memcg charing
* clearing allocated folio or not

-v2:
https://lore.kernel.org/linux-mm/[email protected]/
* lots of code cleanup according to Chris's comments, thanks!
* collect Chris's ack tags, thanks!
* address David's comment on moving to use folio_add_new_anon_rmap for !folio_test_anon in do_swap_page, thanks!
* remove the MADV_PAGEOUT patch from this series as Ryan will intergrate it into swap-out series
* Apply Kairui's work of "mm/swap: fix race when skipping swapcache" on large folios swap-in as well
* fixed corrupted data(zero-filled data) in two races: zswap and a part of entries are in swapcache while some others are not in by checking SWAP_HAS_CACHE while swapping in a large folio

-v1:
https://lore.kernel.org/all/[email protected]/#t

Barry Song (3):
mm: swap: introduce swapcache_prepare_nr and swapcache_clear_nr for
large folios swap-in
mm: Introduce mem_cgroup_swapin_uncharge_swap_nr() helper for large
folios swap-in
mm: Introduce per-thpsize swapin control policy

Chuanhua Han (1):
mm: support large folios swapin as a whole for zRAM-like swapfile

 Documentation/admin-guide/mm/transhuge.rst |   6 +
 include/linux/huge_mm.h                    |   1 +
 include/linux/memcontrol.h                 |  12 ++
 include/linux/swap.h                       |   9 +-
 mm/huge_memory.c                           |  44 +++++
 mm/memory.c                                | 212 ++++++++++++++++++---
 mm/swap.h                                  |  10 +-
 mm/swapfile.c                              | 102 ++++++----
 8 files changed, 329 insertions(+), 67 deletions(-)
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH v5 1/4 mm: swap: introduce swapcache_prepare_nr and swapcache_clear_nr for large folios swap-in

From: Barry Song <[email protected]>

Commit 13ddaf26be32 ("mm/swap: fix race when skipping swapcache") supports one entry only, to support large folio swap-in, we need to handle multiple swap entries.

To optimize stack usage, we iterate twice in __swap_duplicate_nr(): the first time to verify that all entries are valid, and the second time to apply the modifications to the entries.

---
 include/linux/swap.h |   9 +++-
 mm/swap.h            |  10 ++++-
 mm/swapfile.c        | 102 ++++++++++++++++++++++++++-----------------
 3 files changed, 77 insertions(+), 44 deletions(-)
include/linux/swap.h +7 -2
index ba7ea95d1c57..f1b28fd04533 100644
@@ -480,7 +480,7 @@ extern int get_swap_pages(int n, swp_entry_t swp_entries[], int order);
 extern int add_swap_count_continuation(swp_entry_t, gfp_t);
 extern void swap_shmem_alloc(swp_entry_t);
 extern int swap_duplicate(swp_entry_t);
-extern int swapcache_prepare(swp_entry_t);
+extern int swapcache_prepare_nr(swp_entry_t entry, int nr);
 extern void swap_free_nr(swp_entry_t entry, int nr_pages);
 extern void swapcache_free_entries(swp_entry_t *entries, int n);
 extern void free_swap_and_cache_nr(swp_entry_t entry, int nr);
@@ -554,7 +554,7 @@ static inline int swap_duplicate(swp_entry_t swp)
 	return 0;
 }
 
-static inline int swapcache_prepare(swp_entry_t swp)
+static inline int swapcache_prepare_nr(swp_entry_t swp, int nr)
 {
 	return 0;
 }
@@ -612,6 +612,11 @@ static inline void swap_free(swp_entry_t entry)
 	swap_free_nr(entry, 1);
 }
 
+static inline int swapcache_prepare(swp_entry_t entry)
+{
+	return swapcache_prepare_nr(entry, 1);
+}
+
 #ifdef CONFIG_MEMCG
 static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg)
 {
mm/swap.h +8 -2
index baa1fa946b34..81ff7eb0be9c 100644
@@ -59,7 +59,7 @@ void __delete_from_swap_cache(struct folio *folio,
 void delete_from_swap_cache(struct folio *folio);
 void clear_shadow_from_swap_cache(int type, unsigned long begin,
 				  unsigned long end);
-void swapcache_clear(struct swap_info_struct *si, swp_entry_t entry);
+void swapcache_clear_nr(struct swap_info_struct *si, swp_entry_t entry, int nr);
 struct folio *swap_cache_get_folio(swp_entry_t entry,
 		struct vm_area_struct *vma, unsigned long addr);
 struct folio *filemap_get_incore_folio(struct address_space *mapping,
@@ -120,7 +120,7 @@ static inline int swap_writepage(struct page *p, struct writeback_control *wbc)
 	return 0;
 }
 
-static inline void swapcache_clear(struct swap_info_struct *si, swp_entry_t entry)
+static inline void swapcache_clear_nr(struct swap_info_struct *si, swp_entry_t entry, int nr)
 {
 }
 
@@ -172,4 +172,10 @@ static inline unsigned int folio_swap_flags(struct folio *folio)
 	return 0;
 }
 #endif /* CONFIG_SWAP */
+
+static inline void swapcache_clear(struct swap_info_struct *si, swp_entry_t entry)
+{
+	swapcache_clear_nr(si, entry, 1);
+}
+
 #endif /* _MM_SWAP_H */
mm/swapfile.c +62 -40

158 lines not on this page · raw message

Signature · 1 line
2.34.1
reviewer Reviewed-by on v5 1/4
#2 lore raw

Re: [PATCH v5 1/4] mm: swap: introduce swapcache_prepare_nr and swapcache_clear_nr for large folios swap-in

Hi Barry,

Quoted 11 lines · Barry Song

From: Barry Song <[email protected]>

Commit 13ddaf26be32 ("mm/swap: fix race when skipping swapcache") supports one entry only, to support large folio swap-in, we need to handle multiple swap entries.

To optimize stack usage, we iterate twice in __swap_duplicate_nr(): the first time to verify that all entries are valid, and the second time to apply the modifications to the entries.

LGTM. Feel free to add:

By the way, my shmem swap patchset[1] also relies on this patch, so I wonder if it's possible to merge this patch into the mm-unstable branch first (if other patches still need discussion), to make it easier for me to rebase and resend my patch set? Thanks.

[1] https://lore.kernel.org/all/[email protected]/

#3 lore raw

Re: [PATCH v5 1/4] mm: swap: introduce swapcache_prepare_nr and swapcache_clear_nr for large folios swap-in

In reply to Barry Song

include/linux/swap.h static inline void swap_free(swp_entry_t entry) · line 618

Quoted 4 lines · Barry Song
include/linux/swap.h +4 -0
+static inline int swapcache_prepare(swp_entry_t entry)
+{
+	return swapcache_prepare_nr(entry, 1);
+}

Same comment as 2/4 -- there are only two callers of swapcache_prepre(). Just make that take the 'nr' argument and change both callers to pass 1.

author
#4 lore raw

Re: [PATCH v5 1/4] mm: swap: introduce swapcache_prepare_nr and swapcache_clear_nr for large folios swap-in

Quoted 9 lines · Matthew Wilcox

include/linux/swap.h static inline void swap_free(swp_entry_t entry) · line 618

Quoted 4 lines · Barry Song
include/linux/swap.h +4 -0
+static inline int swapcache_prepare(swp_entry_t entry)
+{
+     return swapcache_prepare_nr(entry, 1);
+}

Same comment as 2/4 -- there are only two callers of swapcache_prepre(). Just make that take the 'nr' argument and change both callers to pass 1.

make sense to me. As Baolin also needs this patch for shmem, I'm going to separate this one from this series and send a new version with the suggested change so that Andrew can pull it earlier.

Thanks
Barry

author
#5 lore raw

PATCH v5 2/4 mm: Introduce mem_cgroup_swapin_uncharge_swap_nr() helper for large folios swap-in

In reply to Barry Song

From: Barry Song <[email protected]>

With large folios swap-in, we might need to uncharge multiple entries all together, it is better to introduce a helper for that.

---
 include/linux/memcontrol.h | 12 ++++++++++++
 1 file changed, 12 insertions(+)
include/linux/memcontrol.h +12 -0
index 1b79760af685..55958cbce61b 100644
@@ -684,6 +684,14 @@ int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm,
 				  gfp_t gfp, swp_entry_t entry);
 void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry);
 
+static inline void mem_cgroup_swapin_uncharge_swap_nr(swp_entry_t entry, int nr)
+{
+	int i;
+
+	for (i = 0; i < nr; i++, entry.val++)
+		mem_cgroup_swapin_uncharge_swap(entry);
+}
+
 void __mem_cgroup_uncharge(struct folio *folio);
 
 /**
@@ -1185,6 +1193,10 @@ static inline void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry)
 {
 }
 
+static inline void mem_cgroup_swapin_uncharge_swap_nr(swp_entry_t entry, int nr)
+{
+}
+
 static inline void mem_cgroup_uncharge(struct folio *folio)
 {
 }
Signature · 1 line
2.34.1
#6 lore raw

Re: [PATCH v5 2/4] mm: Introduce mem_cgroup_swapin_uncharge_swap_nr() helper for large folios swap-in

Quoted 16 lines · Barry Song

From: Barry Song <[email protected]>

With large folios swap-in, we might need to uncharge multiple entries all together, it is better to introduce a helper for that.

---
 include/linux/memcontrol.h | 12 ++++++++++++
 1 file changed, 12 insertions(+)
include/linux/memcontrol.h +0 -0
index 1b79760af685..55958cbce61b 100644

include/linux/memcontrol.h int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm, · line 692

Quoted 9 lines
include/linux/memcontrol.h +6 -0
@@ -684,6 +684,14 @@ int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm,
                                  gfp_t gfp, swp_entry_t entry);
 void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry);
 
+static inline void mem_cgroup_swapin_uncharge_swap_nr(swp_entry_t entry, int nr)
+{
+       int i;
+
+       for (i = 0; i < nr; i++, entry.val++)
+               mem_cgroup_swapin_uncharge_swap(entry);

mem_cgroup_swapin_uncharge_swap() calls mem_cgroup_uncharge_swap() which already takes in nr_pages, but we currently only pass 1. Would it be better if we just make mem_cgroup_swapin_uncharge_swap() take in nr_pages as well and pass it along to mem_cgroup_uncharge_swap(), instead of calling it in a loop?

This would batch the page counter, stats updates, and refcount updates in mem_cgroup_uncharge_swap(). You may be able to observe a bit of a performance gain with this.

Quoted 19 lines
include/linux/memcontrol.h +6 -1
+}
+
 void __mem_cgroup_uncharge(struct folio *folio);
 
 /**
@@ -1185,6 +1193,10 @@ static inline void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry)
 {
 }
 
+static inline void mem_cgroup_swapin_uncharge_swap_nr(swp_entry_t entry, int nr)
+{
+}
+
 static inline void mem_cgroup_uncharge(struct folio *folio)
 {
 }
--

2.34.1

author
#7 lore raw

Re: [PATCH v5 2/4] mm: Introduce mem_cgroup_swapin_uncharge_swap_nr() helper for large folios swap-in

Quoted 34 lines · Yosry Ahmed
Quoted 16 lines · Barry Song

From: Barry Song <[email protected]>

With large folios swap-in, we might need to uncharge multiple entries all together, it is better to introduce a helper for that.

---
 include/linux/memcontrol.h | 12 ++++++++++++
 1 file changed, 12 insertions(+)
include/linux/memcontrol.h +0 -0
index 1b79760af685..55958cbce61b 100644

include/linux/memcontrol.h int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm, · line 692

Quoted 9 lines
include/linux/memcontrol.h +6 -0
@@ -684,6 +684,14 @@ int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm,
                                  gfp_t gfp, swp_entry_t entry);
 void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry);
 
+static inline void mem_cgroup_swapin_uncharge_swap_nr(swp_entry_t entry, int nr)
+{
+       int i;
+
+       for (i = 0; i < nr; i++, entry.val++)
+               mem_cgroup_swapin_uncharge_swap(entry);

mem_cgroup_swapin_uncharge_swap() calls mem_cgroup_uncharge_swap() which already takes in nr_pages, but we currently only pass 1. Would it be better if we just make mem_cgroup_swapin_uncharge_swap() take in nr_pages as well and pass it along to mem_cgroup_uncharge_swap(), instead of calling it in a loop?

Quoted 3 lines

This would batch the page counter, stats updates, and refcount updates in mem_cgroup_uncharge_swap(). You may be able to observe a bit of a performance gain with this.

Good suggestion. I'll send the v6 version below after waiting for some comments on the other patches.

From 92dfbf300fd51b427d2a6833226d1b777e0b5fee Mon Sep 17 00:00:00 2001
From: Barry Song <[email protected]>
Date: Fri, 26 Jul 2024 14:33:54 +1200
Subject: [PATCH v6 2/4] mm: Introduce mem_cgroup_swapin_uncharge_swap_nr()
helper for large folios swap-in

With large folios swap-in, we might need to uncharge multiple entries all together, it is better to introduce a helper for that.

---
 include/linux/memcontrol.h | 10 ++++++++--
 mm/memcontrol.c            |  7 ++++---
 2 files changed, 12 insertions(+), 5 deletions(-)
include/linux/memcontrol.h +8 -2
index 1b79760af685..f5dd1e34654a 100644
@@ -682,7 +682,8 @@ int mem_cgroup_hugetlb_try_charge(struct mem_cgroup *memcg, gfp_t gfp,
 
 int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm,
 				  gfp_t gfp, swp_entry_t entry);
-void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry);
+
+void mem_cgroup_swapin_uncharge_swap_nr(swp_entry_t entry, unsigned int nr_pages);
 
 void __mem_cgroup_uncharge(struct folio *folio);
 
@@ -1181,7 +1182,7 @@ static inline int mem_cgroup_swapin_charge_folio(struct folio *folio,
 	return 0;
 }
 
-static inline void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry)
+static inline void mem_cgroup_swapin_uncharge_swap_nr(swp_entry_t entry, int nr)
 {
 }
 
@@ -1796,6 +1797,11 @@ static inline void count_objcg_event(struct obj_cgroup *objcg,
 
 #endif /* CONFIG_MEMCG */
 
+static inline void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry)
+{
+	mem_cgroup_swapin_uncharge_swap_nr(entry, 1);
+}
+
 #if defined(CONFIG_MEMCG) && defined(CONFIG_ZSWAP)
 bool obj_cgroup_may_zswap(struct obj_cgroup *objcg);
 void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size);
mm/memcontrol.c +4 -3
index eb92c21615eb..25657d6a133f 100644
@@ -4573,14 +4573,15 @@ int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm,
 
 /*
  * mem_cgroup_swapin_uncharge_swap - uncharge swap slot
- * @entry: swap entry for which the page is charged
+ * @entry: the first swap entry for which the pages are charged
+ * @nr_pages: number of pages which will be uncharged
  *
  * Call this function after successfully adding the charged page to swapcache.
  *
  * Note: This function assumes the page for which swap slot is being uncharged
  * is order 0 page.
  */
-void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry)
+void mem_cgroup_swapin_uncharge_swap_nr(swp_entry_t entry, unsigned int nr_pages)
 {
 	/*
 	 * Cgroup1's unified memory+swap counter has been charged with the
@@ -4600,7 +4601,7 @@ void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry)
 		 * let's not wait for it.  The page already received a
 		 * memory+swap charge, drop the swap entry duplicate.
 		 */
-		mem_cgroup_uncharge_swap(entry, 1);
+		mem_cgroup_uncharge_swap(entry, nr_pages);
 	}
 }
 
Signature · 1 line
2.34.1
Quoted 20 lines
Quoted 20 lines
include/linux/memcontrol.h +6 -1
+}
+
 void __mem_cgroup_uncharge(struct folio *folio);
 
 /**
@@ -1185,6 +1193,10 @@ static inline void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry)
 {
 }
 
+static inline void mem_cgroup_swapin_uncharge_swap_nr(swp_entry_t entry, int nr)
+{
+}
+
 static inline void mem_cgroup_uncharge(struct folio *folio)
 {
 }
--

2.34.1

#8 lore raw

Re: [PATCH v5 2/4] mm: Introduce mem_cgroup_swapin_uncharge_swap_nr() helper for large folios swap-in

Quoted 3 lines · Barry Song
include/linux/memcontrol.h +2 -1
-void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry);
+
+void mem_cgroup_swapin_uncharge_swap_nr(swp_entry_t entry, unsigned int nr_pages);

[...]

include/linux/memcontrol.h static inline void count_objcg_event(struct obj_cgroup *objcg, · line 1803

Quoted 4 lines
include/linux/memcontrol.h +4 -0
+static inline void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry)
+{
+	mem_cgroup_swapin_uncharge_swap_nr(entry, 1);
+}

There are only two callers of mem_cgroup_swapin_uncharge_swap! Just add an argument to mem_cgroup_swapin_uncharge_swap() and change the two callers. It would be _less_ code than this extra wrapper, and certainly less confusing.

author
#9 lore raw

Re: [PATCH v5 2/4] mm: Introduce mem_cgroup_swapin_uncharge_swap_nr() helper for large folios swap-in

Quoted 6 lines · Matthew Wilcox
Quoted 3 lines · Barry Song
include/linux/memcontrol.h +2 -1
-void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry);
+
+void mem_cgroup_swapin_uncharge_swap_nr(swp_entry_t entry, unsigned int nr_pages);

[...]

Quoted 9 lines

include/linux/memcontrol.h static inline void count_objcg_event(struct obj_cgroup *objcg, · line 1803

Quoted 4 lines
include/linux/memcontrol.h +4 -0
+static inline void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry)
+{
+     mem_cgroup_swapin_uncharge_swap_nr(entry, 1);
+}

There are only two callers of mem_cgroup_swapin_uncharge_swap! Just add an argument to mem_cgroup_swapin_uncharge_swap() and change the two callers. It would be _less_ code than this extra wrapper, and certainly less confusing.

sounds good to me. I can totally drop this wrapper - mem_cgroup_swapin_uncharge_swap()
in v6.

Thanks
Barry

author
#10 lore raw

PATCH v5 3/4 mm: support large folios swapin as a whole for zRAM-like swapfile

In reply to Barry Song

From: Chuanhua Han <[email protected]>

In an embedded system like Android, more than half of anonymous memory is actually stored in swap devices such as zRAM. For instance, when an app is switched to the background, most of its memory might be swapped out.

Currently, we have mTHP features, but unfortunately, without support for large folio swap-ins, once those large folios are swapped out, we lose them immediately because mTHP is a one-way ticket.

This patch introduces mTHP swap-in support. For now, we limit mTHP swap-ins to contiguous swaps that were likely swapped out from mTHP as a whole.

Additionally, the current implementation only covers the SWAP_SYNCHRONOUS case. This is the simplest and most common use case, benefiting millions of Android phones and similar devices with minimal implementation cost. In this straightforward scenario, large folios are always exclusive, eliminating the need to handle complex rmap and swapcache issues.

It offers several benefits:
1. Enables bidirectional mTHP swapping, allowing retrieval of mTHP after swap-out and swap-in.
2. Eliminates fragmentation in swap slots and supports successful THP_SWPOUT without fragmentation.
3. Enables zRAM/zsmalloc to compress and decompress mTHP, reducing CPU usage and enhancing compression ratios significantly.

Deploying this on millions of actual products, we haven't observed any noticeable increase in memory footprint for 64KiB mTHP based on CONT-PTE on ARM64.

---
 mm/memory.c | 211 ++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 188 insertions(+), 23 deletions(-)
mm/memory.c +188 -23
index 833d2cad6eb2..14048e9285d4 100644
@@ -3986,6 +3986,152 @@ static vm_fault_t handle_pte_marker(struct vm_fault *vmf)
 	return VM_FAULT_SIGBUS;
 }
 
+/*
+ * check a range of PTEs are completely swap entries with
+ * contiguous swap offsets and the same SWAP_HAS_CACHE.
+ * ptep must be first one in the range
+ */
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+static bool can_swapin_thp(struct vm_fault *vmf, pte_t *ptep, int nr_pages)
+{
+	struct swap_info_struct *si;
+	unsigned long addr;
+	swp_entry_t entry;
+	pgoff_t offset;
+	char has_cache;
+	int idx, i;
+	pte_t pte;
+
+	addr = ALIGN_DOWN(vmf->address, nr_pages * PAGE_SIZE);
+	idx = (vmf->address - addr) / PAGE_SIZE;
+	pte = ptep_get(ptep);
+
+	if (!pte_same(pte, pte_move_swp_offset(vmf->orig_pte, -idx)))
+		return false;
+	entry = pte_to_swp_entry(pte);
+	offset = swp_offset(entry);
+	if (swap_pte_batch(ptep, nr_pages, pte) != nr_pages)
+		return false;
+
+	si = swp_swap_info(entry);
+	has_cache = si->swap_map[offset] & SWAP_HAS_CACHE;
+	for (i = 1; i < nr_pages; i++) {
+		/*
+		 * while allocating a large folio and doing swap_read_folio for the
+		 * SWP_SYNCHRONOUS_IO path, which is the case the being faulted pte
+		 * doesn't have swapcache. We need to ensure all PTEs have no cache
+		 * as well, otherwise, we might go to swap devices while the content
+		 * is in swapcache
+		 */
+		if ((si->swap_map[offset + i] & SWAP_HAS_CACHE) != has_cache)
+			return false;
+	}
+
+	return true;
+}
+
+static inline unsigned long thp_swap_suitable_orders(pgoff_t swp_offset,
+		unsigned long addr, unsigned long orders)
+{
+	int order, nr;
+
+	order = highest_order(orders);
+
+	/*
+	 * To swap-in a THP with nr pages, we require its first swap_offset
+	 * is aligned with nr. This can filter out most invalid entries.
+	 */
+	while (orders) {
+		nr = 1 << order;
+		if ((addr >> PAGE_SHIFT) % nr == swp_offset % nr)
+			break;
+		order = next_order(&orders, order);
+	}
+
+	return orders;
+}
+#else
+static inline bool can_swapin_thp(struct vm_fault *vmf, pte_t *ptep, int nr_pages)
+{
+	return false;
+}
+#endif
+
+static struct folio *alloc_swap_folio(struct vm_fault *vmf)
+{
+	struct vm_area_struct *vma = vmf->vma;
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+	unsigned long orders;
+	struct folio *folio;
+	unsigned long addr;
+	swp_entry_t entry;
+	spinlock_t *ptl;
+	pte_t *pte;
+	gfp_t gfp;
+	int order;
+
+	/*
+	 * If uffd is active for the vma we need per-page fault fidelity to
+	 * maintain the uffd semantics.
+	 */
+	if (unlikely(userfaultfd_armed(vma)))
+		goto fallback;
+
+	/*
+	 * A large swapped out folio could be partially or fully in zswap. We
+	 * lack handling for such cases, so fallback to swapping in order-0
+	 * folio.
+	 */
+	if (!zswap_never_enabled())
+		goto fallback;
+
+	entry = pte_to_swp_entry(vmf->orig_pte);
+	/*
+	 * Get a list of all the (large) orders below PMD_ORDER that are enabled
+	 * and suitable for swapping THP.
+	 */
+	orders = thp_vma_allowable_orders(vma, vma->vm_flags,
+			TVA_IN_PF | TVA_ENFORCE_SYSFS, BIT(PMD_ORDER) - 1);
+	orders = thp_vma_suitable_orders(vma, vmf->address, orders);
+	orders = thp_swap_suitable_orders(swp_offset(entry), vmf->address, orders);
+
+	if (!orders)
+		goto fallback;
+
+	pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address & PMD_MASK, &ptl);
+	if (unlikely(!pte))
+		goto fallback;
+
+	/*
+	 * For do_swap_page, find the highest order where the aligned range is
+	 * completely swap entries with contiguous swap offsets.
+	 */
+	order = highest_order(orders);
+	while (orders) {
+		addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
+		if (can_swapin_thp(vmf, pte + pte_index(addr), 1 << order))
+			break;
+		order = next_order(&orders, order);
+	}
+
+	pte_unmap_unlock(pte, ptl);
+
+	/* Try allocating the highest of the remaining orders. */
+	gfp = vma_thp_gfp_mask(vma);
+	while (orders) {
+		addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
+		folio = vma_alloc_folio(gfp, order, vma, addr, true);
+		if (folio)
+			return folio;
+		order = next_order(&orders, order);
+	}
+
+fallback:
+#endif
+	return vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, vmf->address, false);
+}
+
+
 /*
  * We enter with non-exclusive mmap_lock (to exclude vma changes,
  * but allow concurrent faults), and pte mapped but not yet locked.
@@ -4074,35 +4220,37 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 	if (!folio) {
 		if (data_race(si->flags & SWP_SYNCHRONOUS_IO) &&
 		    __swap_count(entry) == 1) {
-			/*
-			 * Prevent parallel swapin from proceeding with
-			 * the cache flag. Otherwise, another thread may
-			 * finish swapin first, free the entry, and swapout
-			 * reusing the same entry. It's undetectable as
-			 * pte_same() returns true due to entry reuse.
-			 */
-			if (swapcache_prepare(entry)) {
-				/* Relax a bit to prevent rapid repeated page faults */
-				schedule_timeout_uninterruptible(1);
-				goto out;
-			}
-			need_clear_cache = true;
-
 			/* skip swapcache */
-			folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0,
-						vma, vmf->address, false);
+			folio = alloc_swap_folio(vmf);
 			page = &folio->page;
 			if (folio) {
 				__folio_set_locked(folio);
 				__folio_set_swapbacked(folio);
 
+				nr_pages = folio_nr_pages(folio);
+				if (folio_test_large(folio))
+					entry.val = ALIGN_DOWN(entry.val, nr_pages);
+				/*
+				 * Prevent parallel swapin from proceeding with
+				 * the cache flag. Otherwise, another thread may
+				 * finish swapin first, free the entry, and swapout
+				 * reusing the same entry. It's undetectable as
+				 * pte_same() returns true due to entry reuse.
+				 */
+				if (swapcache_prepare_nr(entry, nr_pages)) {
+					/* Relax a bit to prevent rapid repeated page faults */
+					schedule_timeout_uninterruptible(1);
+					goto out_page;
+				}
+				need_clear_cache = true;
+
 				if (mem_cgroup_swapin_charge_folio(folio,
 							vma->vm_mm, GFP_KERNEL,
 							entry)) {
 					ret = VM_FAULT_OOM;
 					goto out_page;
 				}
-				mem_cgroup_swapin_uncharge_swap(entry);
+				mem_cgroup_swapin_uncharge_swap_nr(entry, nr_pages);
 
 				shadow = get_shadow_from_swap_cache(entry);
 				if (shadow)
@@ -4209,6 +4357,22 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 		goto out_nomap;
 	}
 
+	/* allocated large folios for SWP_SYNCHRONOUS_IO */
+	if (folio_test_large(folio) && !folio_test_swapcache(folio)) {
+		unsigned long nr = folio_nr_pages(folio);
+		unsigned long folio_start = ALIGN_DOWN(vmf->address, nr * PAGE_SIZE);
+		unsigned long idx = (vmf->address - folio_start) / PAGE_SIZE;
+		pte_t *folio_ptep = vmf->pte - idx;
+
+		if (!can_swapin_thp(vmf, folio_ptep, nr))
+			goto out_nomap;
+
+		page_idx = idx;
+		address = folio_start;
+		ptep = folio_ptep;
+		goto check_folio;
+	}
+
 	nr_pages = 1;
 	page_idx = 0;
 	address = vmf->address;
@@ -4340,11 +4504,12 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 		folio_add_lru_vma(folio, vma);
 	} else if (!folio_test_anon(folio)) {
 		/*
-		 * We currently only expect small !anon folios, which are either
-		 * fully exclusive or fully shared. If we ever get large folios
-		 * here, we have to be careful.
+		 * We currently only expect small !anon folios which are either
+		 * fully exclusive or fully shared, or new allocated large folios
+		 * which are fully exclusive. If we ever get large folios within
+		 * swapcache here, we have to be careful.
 		 */
-		VM_WARN_ON_ONCE(folio_test_large(folio));
+		VM_WARN_ON_ONCE(folio_test_large(folio) && folio_test_swapcache(folio));
 		VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
 		folio_add_new_anon_rmap(folio, vma, address, rmap_flags);
 	} else {
@@ -4387,7 +4552,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 out:
 	/* Clear the swap cache pin for direct swapin after PTL unlock */
 	if (need_clear_cache)
-		swapcache_clear(si, entry);
+		swapcache_clear_nr(si, entry, nr_pages);
 	if (si)
 		put_swap_device(si);
 	return ret;
@@ -4403,7 +4568,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 		folio_put(swapcache);
 	}
 	if (need_clear_cache)
-		swapcache_clear(si, entry);
+		swapcache_clear_nr(si, entry, nr_pages);
 	if (si)
 		put_swap_device(si);
 	return ret;
Signature · 1 line
2.34.1
#11 lore raw

Re: [PATCH v5 3/4] mm: support large folios swapin as a whole for zRAM-like swapfile

mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4225

Quoted 4 lines · Barry Song
mm/memory.c +1 -2
-			folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0,
-						vma, vmf->address, false);
+			folio = alloc_swap_folio(vmf);
 			page = &folio->page;

This is no longer correct. You need to set 'page' to the precise page that is being faulted rather than the first page of the folio. It was fine before because it always allocated a single-page folio, but now it must use folio_page() or folio_file_page() (whichever has the correct semantics for you).

Also you need to fix your test suite to notice this bug. I suggest doing that first so that you know whether you've got the calculation correct.

author
#12 lore raw

Re: [PATCH v5 3/4] mm: support large folios swapin as a whole for zRAM-like swapfile

Quoted 13 lines · Matthew Wilcox

mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4225

Quoted 4 lines · Barry Song
mm/memory.c +1 -2
-                     folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0,
-                                             vma, vmf->address, false);
+                     folio = alloc_swap_folio(vmf);
                      page = &folio->page;

This is no longer correct. You need to set 'page' to the precise page that is being faulted rather than the first page of the folio. It was fine before because it always allocated a single-page folio, but now it must use folio_page() or folio_file_page() (whichever has the correct semantics for you).

Quoted 3 lines

Also you need to fix your test suite to notice this bug. I suggest doing that first so that you know whether you've got the calculation correct.

I don't understand why the code is designed in the way the page is the first page of this folio. Otherwise, we need lots of changes later while mapping the folio in ptes and rmap.

Quoted 1 line

Thanks
Barry

author
#13 lore raw

Re: [PATCH v5 3/4] mm: support large folios swapin as a whole for zRAM-like swapfile

Quoted 15 lines · Barry Song
Quoted 13 lines · Matthew Wilcox

mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4225

Quoted 4 lines · Barry Song
mm/memory.c +1 -2
-                     folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0,
-                                             vma, vmf->address, false);
+                     folio = alloc_swap_folio(vmf);
                      page = &folio->page;

This is no longer correct. You need to set 'page' to the precise page that is being faulted rather than the first page of the folio. It was fine before because it always allocated a single-page folio, but now it must use folio_page() or folio_file_page() (whichever has the correct semantics for you).

Quoted 7 lines
Quoted 3 lines

Also you need to fix your test suite to notice this bug. I suggest doing that first so that you know whether you've got the calculation correct.

I don't understand why the code is designed in the way the page is the first page of this folio. Otherwise, we need lots of changes later while mapping the folio in ptes and rmap.

For both accessing large folios in the swapcache and allocating new large folios, the page points to the first page of the folio. we are mapping the whole folio not the specific page.

for swapcache cases, you can find the same thing here,

        if (folio_test_large(folio) && folio_test_swapcache(folio)) {
                ...
                entry = folio->swap;
                page = &folio->page;
        }

Thanks
Barry

#14 lore raw

Re: [PATCH v5 3/4] mm: support large folios swapin as a whole for zRAM-like swapfile

Quoted 16 lines · Barry Song
Quoted 15 lines · Barry Song
Quoted 13 lines · Matthew Wilcox

mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4225

Quoted 4 lines · Barry Song
mm/memory.c +1 -2
-                     folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0,
-                                             vma, vmf->address, false);
+                     folio = alloc_swap_folio(vmf);
                      page = &folio->page;

This is no longer correct. You need to set 'page' to the precise page that is being faulted rather than the first page of the folio. It was fine before because it always allocated a single-page folio, but now it must use folio_page() or folio_file_page() (whichever has the correct semantics for you).

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

Also you need to fix your test suite to notice this bug. I suggest doing that first so that you know whether you've got the calculation correct.

I don't understand why the code is designed in the way the page is the first page of this folio. Otherwise, we need lots of changes later while mapping the folio in ptes and rmap.

What?

        folio = swap_cache_get_folio(entry, vma, vmf->address);
        if (folio)
                page = folio_file_page(folio, swp_offset(entry));

page is the precise page, not the first page of the folio.

Quoted 3 lines

For both accessing large folios in the swapcache and allocating new large folios, the page points to the first page of the folio. we are mapping the whole folio not the specific page.

But what address are we mapping the whole folio at?

Quoted 7 lines

for swapcache cases, you can find the same thing here,

        if (folio_test_large(folio) && folio_test_swapcache(folio)) {
                ...
                entry = folio->swap;
                page = &folio->page;
        }

Yes, but you missed some important lines from your quote:

                page_idx = idx;
                address = folio_start;
                ptep = folio_ptep;
                nr_pages = nr;

We deliberate adjust the address so that, yes, we're mapping the entire folio, but we're mapping it at an address that means that the page we actually faulted on ends up at the address that we faulted on.

author
#15 lore raw

Re: [PATCH v5 3/4] mm: support large folios swapin as a whole for zRAM-like swapfile

Quoted 26 lines · Matthew Wilcox
Quoted 16 lines · Barry Song
Quoted 15 lines · Barry Song
Quoted 13 lines · Matthew Wilcox

mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4225

Quoted 4 lines · Barry Song
mm/memory.c +1 -2
-                     folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0,
-                                             vma, vmf->address, false);
+                     folio = alloc_swap_folio(vmf);
                      page = &folio->page;

This is no longer correct. You need to set 'page' to the precise page that is being faulted rather than the first page of the folio. It was fine before because it always allocated a single-page folio, but now it must use folio_page() or folio_file_page() (whichever has the correct semantics for you).

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

Also you need to fix your test suite to notice this bug. I suggest doing that first so that you know whether you've got the calculation correct.

I don't understand why the code is designed in the way the page is the first page of this folio. Otherwise, we need lots of changes later while mapping the folio in ptes and rmap.

Quoted 7 lines

What?

        folio = swap_cache_get_folio(entry, vma, vmf->address);
        if (folio)
                page = folio_file_page(folio, swp_offset(entry));

page is the precise page, not the first page of the folio.

this is the case we may get a large folio in swapcache but we result in mapping only one subpage due to the condition to map the whole folio is not met. if we meet the condition, we are going to set page to the head instead and map the whole mTHP:

        if (folio_test_large(folio) && folio_test_swapcache(folio)) {
                int nr = folio_nr_pages(folio);
                unsigned long idx = folio_page_idx(folio, page);
                unsigned long folio_start = address - idx * PAGE_SIZE;
                unsigned long folio_end = folio_start + nr * PAGE_SIZE;
                pte_t *folio_ptep;
                pte_t folio_pte;

                if (unlikely(folio_start < max(address & PMD_MASK,
vma->vm_start)))
                        goto check_folio;
                if (unlikely(folio_end > pmd_addr_end(address, vma->vm_end)))
                        goto check_folio;

                folio_ptep = vmf->pte - idx;
                folio_pte = ptep_get(folio_ptep);
                if (!pte_same(folio_pte,
pte_move_swp_offset(vmf->orig_pte, -idx)) ||
                    swap_pte_batch(folio_ptep, nr, folio_pte) != nr)
                        goto check_folio;

                page_idx = idx;
                address = folio_start;
                ptep = folio_ptep;
                nr_pages = nr;
                entry = folio->swap;
                page = &folio->page;
        }
Quoted 17 lines
Quoted 4 lines

For both accessing large folios in the swapcache and allocating new large folios, the page points to the first page of the folio. we are mapping the whole folio not the specific page.

But what address are we mapping the whole folio at?

Quoted 7 lines

for swapcache cases, you can find the same thing here,

        if (folio_test_large(folio) && folio_test_swapcache(folio)) {
                ...
                entry = folio->swap;
                page = &folio->page;
        }

Yes, but you missed some important lines from your quote:

Quoted 8 lines
                page_idx = idx;
                address = folio_start;
                ptep = folio_ptep;
                nr_pages = nr;

We deliberate adjust the address so that, yes, we're mapping the entire folio, but we're mapping it at an address that means that the page we actually faulted on ends up at the address that we faulted on.

for this zRAM case, it is a new allocated large folio, only while all conditions are met, we will allocate and map the whole folio. you can check can_swapin_thp() and thp_swap_suitable_orders().

static bool can_swapin_thp(struct vm_fault *vmf, pte_t *ptep, int nr_pages)

{
        struct swap_info_struct *si;
        unsigned long addr;
        swp_entry_t entry;
        pgoff_t offset;
        char has_cache;
        int idx, i;
        pte_t pte;

        addr = ALIGN_DOWN(vmf->address, nr_pages * PAGE_SIZE);
        idx = (vmf->address - addr) / PAGE_SIZE;
        pte = ptep_get(ptep);

        if (!pte_same(pte, pte_move_swp_offset(vmf->orig_pte, -idx)))
                return false;
        entry = pte_to_swp_entry(pte);
        offset = swp_offset(entry);
        if (swap_pte_batch(ptep, nr_pages, pte) != nr_pages)
                return false;

        si = swp_swap_info(entry);
        has_cache = si->swap_map[offset] & SWAP_HAS_CACHE;
        for (i = 1; i < nr_pages; i++) {
                /*
                 * while allocating a large folio and doing

swap_read_folio for the
* SWP_SYNCHRONOUS_IO path, which is the case the
being faulted pte
* doesn't have swapcache. We need to ensure all PTEs
have no cache
* as well, otherwise, we might go to swap devices
while the content

                 * is in swapcache
                 */
                if ((si->swap_map[offset + i] & SWAP_HAS_CACHE) != has_cache)
                        return false;
        }

        return true;
}

and

static struct folio *alloc_swap_folio(struct vm_fault *vmf)

{
        ....
        entry = pte_to_swp_entry(vmf->orig_pte);
        /*
         * Get a list of all the (large) orders below PMD_ORDER that are enabled
         * and suitable for swapping THP.
         */
        orders = thp_vma_allowable_orders(vma, vma->vm_flags,
                        TVA_IN_PF | TVA_IN_SWAPIN | TVA_ENFORCE_SYSFS,
                        BIT(PMD_ORDER) - 1);
        orders = thp_vma_suitable_orders(vma, vmf->address, orders);
        orders = thp_swap_suitable_orders(swp_offset(entry),

vmf->address, orders);
....
}

static inline unsigned long thp_swap_suitable_orders(pgoff_t swp_offset,
unsigned long addr, unsigned long orders)

{
        int order, nr;

        order = highest_order(orders);

        /*
         * To swap-in a THP with nr pages, we require its first swap_offset
         * is aligned with nr. This can filter out most invalid entries.
         */
        while (orders) {
                nr = 1 << order;
                if ((addr >> PAGE_SHIFT) % nr == swp_offset % nr)
                        break;
                order = next_order(&orders, order);
        }

        return orders;
}

A mTHP is swapped out at aligned swap offset. and we only swap in aligned mTHP. if somehow one mTHP is mremap() to unaligned address, we won't swap them in as a large folio. For swapcache case, we are still checking unaligned mTHP, but for new allocated mTHP, it is a different story. There is totally no necessity to support unaligned mTHP and there is no possibility to support unless something is marked in swap devices to say there was a mTHP.

Thanks
Barry

#16 lore raw

Re: [PATCH v5 3/4] mm: support large folios swapin as a whole for zRAM-like swapfile

Quoted 4 lines · Barry Song

for this zRAM case, it is a new allocated large folio, only while all conditions are met, we will allocate and map the whole folio. you can check can_swapin_thp() and thp_swap_suitable_orders().

YOU ARE DOING THIS WRONGLY!

All of you anonymous memory people are utterly fixated on TLBs AND THIS IS WRONG. Yes, TLB performance is important, particularly with crappy ARM designs, which I know a lot of you are paid to work on. But you seem to think this is the only consideration, and you're making bad design choices as a result. It's overly complicated, and you're leaving performance on the table.

Look back at the results Ryan showed in the early days of working on large anonymous folios. Half of the performance win on his system came from using larger TLBs. But the other half came from _reduced software overhead_. The LRU lock is a huge problem, and using large folios cuts the length of the LRU list, hence LRU lock hold time.

Your _own_ data on how hard it is to get hold of a large folio due to fragmentation should be enough to convince you that the more large folios in the system, the better the whole system runs. We should not decline to allocate large folios just because they can't be mapped with a single TLB!

author
#17 lore raw

Re: [PATCH v5 3/4] mm: support large folios swapin as a whole for zRAM-like swapfile

Quoted 22 lines · Matthew Wilcox
Quoted 4 lines · Barry Song

for this zRAM case, it is a new allocated large folio, only while all conditions are met, we will allocate and map the whole folio. you can check can_swapin_thp() and thp_swap_suitable_orders().

YOU ARE DOING THIS WRONGLY!

All of you anonymous memory people are utterly fixated on TLBs AND THIS IS WRONG. Yes, TLB performance is important, particularly with crappy ARM designs, which I know a lot of you are paid to work on. But you seem to think this is the only consideration, and you're making bad design choices as a result. It's overly complicated, and you're leaving performance on the table.

Look back at the results Ryan showed in the early days of working on large anonymous folios. Half of the performance win on his system came from using larger TLBs. But the other half came from _reduced software overhead_. The LRU lock is a huge problem, and using large folios cuts the length of the LRU list, hence LRU lock hold time.

Quoted 4 lines

Your _own_ data on how hard it is to get hold of a large folio due to fragmentation should be enough to convince you that the more large folios in the system, the better the whole system runs. We should not decline to allocate large folios just because they can't be mapped with a single TLB!

I am not convinced. for a new allocated large folio, even alloc_anon_folio() of do_anonymous_page() does the exactly same thing

alloc_anon_folio()

{
        /*
         * Get a list of all the (large) orders below PMD_ORDER that are enabled
         * for this vma. Then filter out the orders that can't be allocated over
         * the faulting address and still be fully contained in the vma.
         */
        orders = thp_vma_allowable_orders(vma, vma->vm_flags,
                        TVA_IN_PF | TVA_ENFORCE_SYSFS, BIT(PMD_ORDER) - 1);
        orders = thp_vma_suitable_orders(vma, vmf->address, orders);

}

you are not going to allocate a mTHP for an unaligned address for a new PF.
Please point out where it is wrong.

Thanks
Barry

author
#18 lore raw

Re: [PATCH v5 3/4] mm: support large folios swapin as a whole for zRAM-like swapfile

Quoted 43 lines · Barry Song
Quoted 22 lines · Matthew Wilcox
Quoted 4 lines · Barry Song

for this zRAM case, it is a new allocated large folio, only while all conditions are met, we will allocate and map the whole folio. you can check can_swapin_thp() and thp_swap_suitable_orders().

YOU ARE DOING THIS WRONGLY!

All of you anonymous memory people are utterly fixated on TLBs AND THIS IS WRONG. Yes, TLB performance is important, particularly with crappy ARM designs, which I know a lot of you are paid to work on. But you seem to think this is the only consideration, and you're making bad design choices as a result. It's overly complicated, and you're leaving performance on the table.

Look back at the results Ryan showed in the early days of working on large anonymous folios. Half of the performance win on his system came from using larger TLBs. But the other half came from _reduced software overhead_. The LRU lock is a huge problem, and using large folios cuts the length of the LRU list, hence LRU lock hold time.

Quoted 4 lines

Your _own_ data on how hard it is to get hold of a large folio due to fragmentation should be enough to convince you that the more large folios in the system, the better the whole system runs. We should not decline to allocate large folios just because they can't be mapped with a single TLB!

I am not convinced. for a new allocated large folio, even alloc_anon_folio() of do_anonymous_page() does the exactly same thing

alloc_anon_folio()

{
        /*
         * Get a list of all the (large) orders below PMD_ORDER that are enabled
         * for this vma. Then filter out the orders that can't be allocated over
         * the faulting address and still be fully contained in the vma.
         */
        orders = thp_vma_allowable_orders(vma, vma->vm_flags,
                        TVA_IN_PF | TVA_ENFORCE_SYSFS, BIT(PMD_ORDER) - 1);
        orders = thp_vma_suitable_orders(vma, vmf->address, orders);
Quoted 5 lines

}

you are not going to allocate a mTHP for an unaligned address for a new PF.
Please point out where it is wrong.

Let's assume we have a folio with the virtual address as 0x500000000000 ~ 0x500000000000 + 64KB
if it is swapped out to 0x10000 ~ 0x100000 + 64KB.

The current code will swap it in as a mTHP if page fault occurs in any address within (0x500000000000 ~ 0x500000000000 + 64KB)

In this case, the mTHP enjoys both decreased TLB and reduced overhead such as LRU lock etc. So it sounds we have nothing lost in this case.

But if the folio is mremap-ed to an unaligned address like:
(0x600000000000 + 16KB ~ 0x600000000000 + 80KB) and its swap offset is still (0x10000 ~ 0x100000 + 64KB).

The current code won't swap in them as mTHP. Sounds like a loss?

If this is the performance problem you are trying to address, my point is that it is not worth increasing the complexity for this stage though this might be doable. We once tracked hundreds of phones running apps randomly for a couple of days, and we didn't encounter such a case. So this is pretty much a corner case.

If your concern is more than this, for example, if you want to swap in large folios even when swaps are completely not contiguous, this is a different story. I agree this is a potential optimization direction to go, but in that case, you still need to find an aligned boundary to handle page faults just like do_anonymous_page(), otherwise, you may result in all kinds of pointless intersections where PFs can cover the address ranges of other PFs, then make the PTEs check such as pte_range_none() completely dis-ordered:

static struct folio *alloc_anon_folio(struct vm_fault *vmf)

{
        ....

        /*
         * Find the highest order where the aligned range is completely
         * pte_none(). Note that all remaining orders will be completely
         * pte_none().
         */
        order = highest_order(orders);
        while (orders) {
                addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
                if (pte_range_none(pte + pte_index(addr), 1 << order))
                        break;
                order = next_order(&orders, order);
        }
}
Quoted 3 lines

Thanks
Barry

reviewer
#19 lore raw

Re: [PATCH v5 3/4] mm: support large folios swapin as a whole for zRAM-like swapfile

In reply to Matthew Wilcox
Quoted 7 lines · Matthew Wilcox
Quoted 4 lines · Barry Song

for this zRAM case, it is a new allocated large folio, only while all conditions are met, we will allocate and map the whole folio. you can check can_swapin_thp() and thp_swap_suitable_orders().

YOU ARE DOING THIS WRONGLY!

I've only scanned the preceeding thread, but I think you're talking about the design descision to only allocate large folios that are naturally aligned in virtual address space, and you're arguing to remove that restriction? The main reason we gave ourselves that constraint for anon mTHP was because allowing it would create the possibility of wandering off the end of the PTE table and add significant complexity to manage neighbouring PTE tables and their respective PTLs.

If the proposal is to start doing this, then I don't agree with that approach.

Quoted 8 lines

All of you anonymous memory people are utterly fixated on TLBs AND THIS IS WRONG. Yes, TLB performance is important, particularly with crappy ARM designs, which I know a lot of you are paid to work on. But you seem to think this is the only consideration, and you're making bad design choices as a result. It's overly complicated, and you're leaving performance on the table.

Quoted 4 lines

Look back at the results Ryan showed in the early days of working on large anonymous folios. Half of the performance win on his system came from using larger TLBs. But the other half came from _reduced software overhead_.

I would just point out that I think the results you are referring to are for the kernel compilation workload, and yes this is indeed what I observed. But kernel compilation is a bit of an outlier since it does a huge amount of fork/exec so the kernel spends a lot of time fiddling with page tables and faulting. The vast majority of the reduced sw overhead is due to significantly reducing the number of faults because we map more pages per fault. But in my experience this workload is a bit of an outlier; most workloads that I've tested with at least tend to set up their memory at the start and its static forever more, which means that those workloads benefit mostly from the TLB benefits - there are very few existing SW overheads to actually reduce.

Quoted 2 lines

The LRU lock is a huge problem, and using large folios cuts the length of the LRU list, hence LRU lock hold time.

I'm sure this is true and you have lots more experience and data than me. And it makes intuitive sense. But I've never personally seen this in any of the workloads that I've benchmarked.

Thanks,
Ryan

Quoted 6 lines

Your _own_ data on how hard it is to get hold of a large folio due to fragmentation should be enough to convince you that the more large folios in the system, the better the whole system runs. We should not decline to allocate large folios just because they can't be mapped with a single TLB!

#20 lore raw

Re: [PATCH v5 3/4] mm: support large folios swapin as a whole for zRAM-like swapfile

In reply to Matthew Wilcox
Quoted 13 lines · Matthew Wilcox <[email protected]> 于2024年7月29日周一 11:51写道:

mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4225

Quoted 4 lines · Barry Song
mm/memory.c +1 -2
-                     folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0,
-                                             vma, vmf->address, false);
+                     folio = alloc_swap_folio(vmf);
                      page = &folio->page;

This is no longer correct. You need to set 'page' to the precise page that is being faulted rather than the first page of the folio. It was fine before because it always allocated a single-page folio, but now it must use folio_page() or folio_file_page() (whichever has the correct semantics for you).

Quoted 6 lines

Also you need to fix your test suite to notice this bug. I suggest doing that first so that you know whether you've got the calculation correct.

This is no problem now, we support large folios swapin as a whole, so the head page is used here instead of the page that is being faulted. You can also refer to the current code context, now support large folios swapin as a whole, and previously only support small page swapin is not the same.

Signature · 2 lines
Thanks,
Chuanhua
#21 lore raw

Re: [PATCH v5 3/4] mm: support large folios swapin as a whole for zRAM-like swapfile

Quoted 20 lines · Chuanhua Han
Quoted 13 lines · Matthew Wilcox <[email protected]> 于2024年7月29日周一 11:51写道:

mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4225

Quoted 4 lines · Barry Song
mm/memory.c +1 -2
-                     folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0,
-                                             vma, vmf->address, false);
+                     folio = alloc_swap_folio(vmf);
                      page = &folio->page;

This is no longer correct. You need to set 'page' to the precise page that is being faulted rather than the first page of the folio. It was fine before because it always allocated a single-page folio, but now it must use folio_page() or folio_file_page() (whichever has the correct semantics for you).

Quoted 6 lines

Also you need to fix your test suite to notice this bug. I suggest doing that first so that you know whether you've got the calculation correct.

Quoted 5 lines

This is no problem now, we support large folios swapin as a whole, so the head page is used here instead of the page that is being faulted. You can also refer to the current code context, now support large folios swapin as a whole, and previously only support small page swapin is not the same.

You have completely failed to understand the problem. Let's try it this way:

We take a page fault at address 0x123456789000.
If part of a 16KiB folio, that's page 1 of the folio at 0x123456788000. If you now map page 0 of the folio at 0x123456789000, you've given the user the wrong page! That looks like data corruption.

The code in

        if (folio_test_large(folio) && folio_test_swapcache(folio)) {

as Barry pointed out will save you -- but what if those conditions fail? What if the mmap has been mremap()ed and the folio now crosses a PMD boundary? mk_pte() will now be called on the wrong page.

author
#22 lore raw

Re: [PATCH v5 3/4] mm: support large folios swapin as a whole for zRAM-like swapfile

Quoted 36 lines · Matthew Wilcox
Quoted 20 lines · Chuanhua Han
Quoted 13 lines · Matthew Wilcox <[email protected]> 于2024年7月29日周一 11:51写道:

mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4225

Quoted 4 lines · Barry Song
mm/memory.c +1 -2
-                     folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0,
-                                             vma, vmf->address, false);
+                     folio = alloc_swap_folio(vmf);
                      page = &folio->page;

This is no longer correct. You need to set 'page' to the precise page that is being faulted rather than the first page of the folio. It was fine before because it always allocated a single-page folio, but now it must use folio_page() or folio_file_page() (whichever has the correct semantics for you).

Quoted 6 lines

Also you need to fix your test suite to notice this bug. I suggest doing that first so that you know whether you've got the calculation correct.

Quoted 5 lines

This is no problem now, we support large folios swapin as a whole, so the head page is used here instead of the page that is being faulted. You can also refer to the current code context, now support large folios swapin as a whole, and previously only support small page swapin is not the same.

You have completely failed to understand the problem. Let's try it this way:

We take a page fault at address 0x123456789000.
If part of a 16KiB folio, that's page 1 of the folio at 0x123456788000. If you now map page 0 of the folio at 0x123456789000, you've given the user the wrong page! That looks like data corruption.

Quoted 5 lines

The code in

        if (folio_test_large(folio) && folio_test_swapcache(folio)) {

as Barry pointed out will save you -- but what if those conditions fail? What if the mmap has been mremap()ed and the folio now crosses a PMD boundary? mk_pte() will now be called on the wrong page.

Chuanhua understood everything correctly. I think you might have missed that we have very strict checks both before allocating large folios and before mapping them for this new allocated mTHP swap-in case.

to allocate a large folio, we check all alignment requirements; PTEs have aligned swap offset and all physically contiguous, that is how mTHP is swapped out. if a mTHP has been mremap() to be unaligned, we won't swap them in as mTHP. two reasons: 1. we have no way to figure out what is the start address of a previous mTHP for non-swapcache case;
2. mremap() to unaligned addresses is rare.

to map a large folio, we check all PTEs are still there by double confirming can_swapin_thp() is true. if PTEs have changed, this is a "goto out_nomap" case.

        /* allocated large folios for SWP_SYNCHRONOUS_IO */
        if (folio_test_large(folio) && !folio_test_swapcache(folio)) {
                unsigned long nr = folio_nr_pages(folio);
                unsigned long folio_start = ALIGN_DOWN(vmf->address,
nr * PAGE_SIZE);
                unsigned long idx = (vmf->address - folio_start) / PAGE_SIZE;
                pte_t *folio_ptep = vmf->pte - idx;

                if (!can_swapin_thp(vmf, folio_ptep, nr))
                        goto out_nomap;

                page_idx = idx;
                address = folio_start;
                ptep = folio_ptep;
                goto check_folio;
        }

Thanks
Barry

#23 lore raw

Re: [PATCH v5 3/4] mm: support large folios swapin as a whole for zRAM-like swapfile

In reply to Matthew Wilcox
Quoted 28 lines · Matthew Wilcox <[email protected]> 于2024年7月29日周一 20:55写道:
Quoted 20 lines · Chuanhua Han
Quoted 13 lines · Matthew Wilcox <[email protected]> 于2024年7月29日周一 11:51写道:

mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4225

Quoted 4 lines · Barry Song
mm/memory.c +1 -2
-                     folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0,
-                                             vma, vmf->address, false);
+                     folio = alloc_swap_folio(vmf);
                      page = &folio->page;

This is no longer correct. You need to set 'page' to the precise page that is being faulted rather than the first page of the folio. It was fine before because it always allocated a single-page folio, but now it must use folio_page() or folio_file_page() (whichever has the correct semantics for you).

Quoted 6 lines

Also you need to fix your test suite to notice this bug. I suggest doing that first so that you know whether you've got the calculation correct.

Quoted 5 lines

This is no problem now, we support large folios swapin as a whole, so the head page is used here instead of the page that is being faulted. You can also refer to the current code context, now support large folios swapin as a whole, and previously only support small page swapin is not the same.

Quoted 7 lines

You have completely failed to understand the problem. Let's try it this way:

We take a page fault at address 0x123456789000.
If part of a 16KiB folio, that's page 1 of the folio at 0x123456788000. If you now map page 0 of the folio at 0x123456789000, you've given the user the wrong page! That looks like data corruption.

The user does not get the wrong data because we are mapping the whole, and for 16KiB folio, we map 16KiB through the page table.

Quoted 6 lines

The code in

        if (folio_test_large(folio) && folio_test_swapcache(folio)) {

as Barry pointed out will save you -- but what if those conditions fail? What if the mmap has been mremap()ed and the folio now crosses a PMD boundary? mk_pte() will now be called on the wrong page.

These special cases have been dealt with in our patch. For mthp's large folio, mk_pte uses head page to construct pte.

Signature · 2 lines
Thanks,
Chuanhua
#24 lore raw

Re: [PATCH v5 3/4] mm: support large folios swapin as a whole for zRAM-like swapfile

In reply to Barry Song

Hi Barry,

kernel test robot noticed the following build warnings:

url:    https://github.com/intel-lab-lkp/linux/commits/Barry-Song/mm-swap-introduce-swapcache_prepare_nr-and-swapcache_clear_nr-for-large-folios-swap-in/20240726-181412
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/20240726094618.401593-4-21cnbao%40gmail.com
patch subject: [PATCH v5 3/4] mm: support large folios swapin as a whole for zRAM-like swapfile
config: i386-randconfig-141-20240727 (https://download.01.org/0day-ci/archive/20240727/[email protected]/config)
compiler: clang version 18.1.5 (https://github.com/llvm/llvm-project 617a15a9eac96088ae5e9134248d8236e34b91b1)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags

| Reported-by: kernel test robot <[email protected]>
| Reported-by: Dan Carpenter <[email protected]>
| Closes: https://lore.kernel.org/r/[email protected]/

smatch warnings:
mm/memory.c:4467 do_swap_page() error: uninitialized symbol 'nr_pages'.

vim +/nr_pages +4467 mm/memory.c

2b7403035459c7 Souptick Joarder        2018-08-23  4143  vm_fault_t do_swap_page(struct vm_fault *vmf)
^1da177e4c3f41 Linus Torvalds          2005-04-16  4144  {
82b0f8c39a3869 Jan Kara                2016-12-14  4145  	struct vm_area_struct *vma = vmf->vma;
d4f9565ae598bd Matthew Wilcox (Oracle  2022-09-02  4146) 	struct folio *swapcache, *folio = NULL;
d4f9565ae598bd Matthew Wilcox (Oracle  2022-09-02  4147) 	struct page *page;
2799e77529c2a2 Miaohe Lin              2021-06-28  4148  	struct swap_info_struct *si = NULL;
14f9135d547060 David Hildenbrand       2022-05-09  4149  	rmap_t rmap_flags = RMAP_NONE;
13ddaf26be324a Kairui Song             2024-02-07  4150  	bool need_clear_cache = false;
1493a1913e34b0 David Hildenbrand       2022-05-09  4151  	bool exclusive = false;
65500d234e74fc Hugh Dickins            2005-10-29  4152  	swp_entry_t entry;
^1da177e4c3f41 Linus Torvalds          2005-04-16  4153  	pte_t pte;
2b7403035459c7 Souptick Joarder        2018-08-23  4154  	vm_fault_t ret = 0;
aae466b0052e18 Joonsoo Kim             2020-08-11  4155  	void *shadow = NULL;
508758960b8d89 Chuanhua Han            2024-05-29  4156  	int nr_pages;
508758960b8d89 Chuanhua Han            2024-05-29  4157  	unsigned long page_idx;
508758960b8d89 Chuanhua Han            2024-05-29  4158  	unsigned long address;
508758960b8d89 Chuanhua Han            2024-05-29  4159  	pte_t *ptep;
^1da177e4c3f41 Linus Torvalds          2005-04-16  4160  
2ca99358671ad3 Peter Xu                2021-11-05  4161  	if (!pte_unmap_same(vmf))
8f4e2101fd7df9 Hugh Dickins            2005-10-29  4162  		goto out;
65500d234e74fc Hugh Dickins            2005-10-29  4163  
2994302bc8a171 Jan Kara                2016-12-14  4164  	entry = pte_to_swp_entry(vmf->orig_pte);
d1737fdbec7f90 Andi Kleen              2009-09-16  4165  	if (unlikely(non_swap_entry(entry))) {
0697212a411c1d Christoph Lameter       2006-06-23  4166  		if (is_migration_entry(entry)) {
82b0f8c39a3869 Jan Kara                2016-12-14  4167  			migration_entry_wait(vma->vm_mm, vmf->pmd,
82b0f8c39a3869 Jan Kara                2016-12-14  4168  					     vmf->address);
b756a3b5e7ead8 Alistair Popple         2021-06-30  4169  		} else if (is_device_exclusive_entry(entry)) {
b756a3b5e7ead8 Alistair Popple         2021-06-30  4170  			vmf->page = pfn_swap_entry_to_page(entry);
b756a3b5e7ead8 Alistair Popple         2021-06-30  4171  			ret = remove_device_exclusive_entry(vmf);
5042db43cc26f5 Jérôme Glisse           2017-09-08  4172  		} else if (is_device_private_entry(entry)) {
1235ccd05b6dd6 Suren Baghdasaryan      2023-06-30  4173  			if (vmf->flags & FAULT_FLAG_VMA_LOCK) {
1235ccd05b6dd6 Suren Baghdasaryan      2023-06-30  4174  				/*
1235ccd05b6dd6 Suren Baghdasaryan      2023-06-30  4175  				 * migrate_to_ram is not yet ready to operate
1235ccd05b6dd6 Suren Baghdasaryan      2023-06-30  4176  				 * under VMA lock.
1235ccd05b6dd6 Suren Baghdasaryan      2023-06-30  4177  				 */
1235ccd05b6dd6 Suren Baghdasaryan      2023-06-30  4178  				vma_end_read(vma);
1235ccd05b6dd6 Suren Baghdasaryan      2023-06-30  4179  				ret = VM_FAULT_RETRY;
1235ccd05b6dd6 Suren Baghdasaryan      2023-06-30  4180  				goto out;
1235ccd05b6dd6 Suren Baghdasaryan      2023-06-30  4181  			}
1235ccd05b6dd6 Suren Baghdasaryan      2023-06-30  4182  
af5cdaf82238fb Alistair Popple         2021-06-30  4183  			vmf->page = pfn_swap_entry_to_page(entry);
16ce101db85db6 Alistair Popple         2022-09-28  4184  			vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
16ce101db85db6 Alistair Popple         2022-09-28  4185  					vmf->address, &vmf->ptl);
3db82b9374ca92 Hugh Dickins            2023-06-08  4186  			if (unlikely(!vmf->pte ||
c33c794828f212 Ryan Roberts            2023-06-12  4187  				     !pte_same(ptep_get(vmf->pte),
c33c794828f212 Ryan Roberts            2023-06-12  4188  							vmf->orig_pte)))
3b65f437d9e8dd Ryan Roberts            2023-06-02  4189  				goto unlock;
16ce101db85db6 Alistair Popple         2022-09-28  4190  
16ce101db85db6 Alistair Popple         2022-09-28  4191  			/*
16ce101db85db6 Alistair Popple         2022-09-28  4192  			 * Get a page reference while we know the page can't be
16ce101db85db6 Alistair Popple         2022-09-28  4193  			 * freed.
16ce101db85db6 Alistair Popple         2022-09-28  4194  			 */
16ce101db85db6 Alistair Popple         2022-09-28  4195  			get_page(vmf->page);
16ce101db85db6 Alistair Popple         2022-09-28  4196  			pte_unmap_unlock(vmf->pte, vmf->ptl);
4a955bed882e73 Alistair Popple         2022-11-14  4197  			ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);
16ce101db85db6 Alistair Popple         2022-09-28  4198  			put_page(vmf->page);
d1737fdbec7f90 Andi Kleen              2009-09-16  4199  		} else if (is_hwpoison_entry(entry)) {
d1737fdbec7f90 Andi Kleen              2009-09-16  4200  			ret = VM_FAULT_HWPOISON;
5c041f5d1f23d3 Peter Xu                2022-05-12  4201  		} else if (is_pte_marker_entry(entry)) {
5c041f5d1f23d3 Peter Xu                2022-05-12  4202  			ret = handle_pte_marker(vmf);
d1737fdbec7f90 Andi Kleen              2009-09-16  4203  		} else {
2994302bc8a171 Jan Kara                2016-12-14  4204  			print_bad_pte(vma, vmf->address, vmf->orig_pte, NULL);
d99be1a8ecf377 Hugh Dickins            2009-12-14  4205  			ret = VM_FAULT_SIGBUS;
d1737fdbec7f90 Andi Kleen              2009-09-16  4206  		}
0697212a411c1d Christoph Lameter       2006-06-23  4207  		goto out;
0697212a411c1d Christoph Lameter       2006-06-23  4208  	}
0bcac06f27d752 Minchan Kim             2017-11-15  4209  
2799e77529c2a2 Miaohe Lin              2021-06-28  4210  	/* Prevent swapoff from happening to us. */
2799e77529c2a2 Miaohe Lin              2021-06-28  4211  	si = get_swap_device(entry);
2799e77529c2a2 Miaohe Lin              2021-06-28  4212  	if (unlikely(!si))
2799e77529c2a2 Miaohe Lin              2021-06-28  4213  		goto out;
0bcac06f27d752 Minchan Kim             2017-11-15  4214  
5a423081b2465d Matthew Wilcox (Oracle  2022-09-02  4215) 	folio = swap_cache_get_folio(entry, vma, vmf->address);
5a423081b2465d Matthew Wilcox (Oracle  2022-09-02  4216) 	if (folio)
5a423081b2465d Matthew Wilcox (Oracle  2022-09-02  4217) 		page = folio_file_page(folio, swp_offset(entry));
d4f9565ae598bd Matthew Wilcox (Oracle  2022-09-02  4218) 	swapcache = folio;
f80207727aaca3 Minchan Kim             2018-01-18  4219  
d4f9565ae598bd Matthew Wilcox (Oracle  2022-09-02  4220) 	if (!folio) {
a449bf58e45abf Qian Cai                2020-08-14  4221  		if (data_race(si->flags & SWP_SYNCHRONOUS_IO) &&
eb085574a7526c Huang Ying              2019-07-11  4222  		    __swap_count(entry) == 1) {
684d098daf0b3a Chuanhua Han            2024-07-26  4223  			/* skip swapcache */
684d098daf0b3a Chuanhua Han            2024-07-26  4224  			folio = alloc_swap_folio(vmf);
684d098daf0b3a Chuanhua Han            2024-07-26  4225  			page = &folio->page;
684d098daf0b3a Chuanhua Han            2024-07-26  4226  			if (folio) {
684d098daf0b3a Chuanhua Han            2024-07-26  4227  				__folio_set_locked(folio);
684d098daf0b3a Chuanhua Han            2024-07-26  4228  				__folio_set_swapbacked(folio);
684d098daf0b3a Chuanhua Han            2024-07-26  4229  
684d098daf0b3a Chuanhua Han            2024-07-26  4230  				nr_pages = folio_nr_pages(folio);

nr_pages is initialized here

684d098daf0b3a Chuanhua Han            2024-07-26  4231  				if (folio_test_large(folio))
684d098daf0b3a Chuanhua Han            2024-07-26  4232  					entry.val = ALIGN_DOWN(entry.val, nr_pages);
13ddaf26be324a Kairui Song             2024-02-07  4233  				/*
13ddaf26be324a Kairui Song             2024-02-07  4234  				 * Prevent parallel swapin from proceeding with
13ddaf26be324a Kairui Song             2024-02-07  4235  				 * the cache flag. Otherwise, another thread may
13ddaf26be324a Kairui Song             2024-02-07  4236  				 * finish swapin first, free the entry, and swapout
13ddaf26be324a Kairui Song             2024-02-07  4237  				 * reusing the same entry. It's undetectable as
13ddaf26be324a Kairui Song             2024-02-07  4238  				 * pte_same() returns true due to entry reuse.
13ddaf26be324a Kairui Song             2024-02-07  4239  				 */
684d098daf0b3a Chuanhua Han            2024-07-26  4240  				if (swapcache_prepare_nr(entry, nr_pages)) {
13ddaf26be324a Kairui Song             2024-02-07  4241  					/* Relax a bit to prevent rapid repeated page faults */
13ddaf26be324a Kairui Song             2024-02-07  4242  					schedule_timeout_uninterruptible(1);
684d098daf0b3a Chuanhua Han            2024-07-26  4243  					goto out_page;
13ddaf26be324a Kairui Song             2024-02-07  4244  				}
13ddaf26be324a Kairui Song             2024-02-07  4245  				need_clear_cache = true;
13ddaf26be324a Kairui Song             2024-02-07  4246  
6599591816f522 Matthew Wilcox (Oracle  2022-09-02  4247) 				if (mem_cgroup_swapin_charge_folio(folio,
63ad4add382305 Matthew Wilcox (Oracle  2022-09-02  4248) 							vma->vm_mm, GFP_KERNEL,
63ad4add382305 Matthew Wilcox (Oracle  2022-09-02  4249) 							entry)) {
545b1b077ca6b3 Michal Hocko            2020-06-25  4250  					ret = VM_FAULT_OOM;
4c6355b25e8bb8 Johannes Weiner         2020-06-03  4251  					goto out_page;
545b1b077ca6b3 Michal Hocko            2020-06-25  4252  				}
684d098daf0b3a Chuanhua Han            2024-07-26  4253  				mem_cgroup_swapin_uncharge_swap_nr(entry, nr_pages);
4c6355b25e8bb8 Johannes Weiner         2020-06-03  4254  
aae466b0052e18 Joonsoo Kim             2020-08-11  4255  				shadow = get_shadow_from_swap_cache(entry);
aae466b0052e18 Joonsoo Kim             2020-08-11  4256  				if (shadow)
63ad4add382305 Matthew Wilcox (Oracle  2022-09-02  4257) 					workingset_refault(folio, shadow);
0076f029cb2906 Joonsoo Kim             2020-06-25  4258  
63ad4add382305 Matthew Wilcox (Oracle  2022-09-02  4259) 				folio_add_lru(folio);
0add0c77a9bd0c Shakeel Butt            2021-04-29  4260  
c9bdf768dd9319 Matthew Wilcox (Oracle  2023-12-13  4261) 				/* To provide entry to swap_read_folio() */
3d2c9087688777 David Hildenbrand       2023-08-21  4262  				folio->swap = entry;
b2d1f38b524121 Yosry Ahmed             2024-06-07  4263  				swap_read_folio(folio, NULL);
63ad4add382305 Matthew Wilcox (Oracle  2022-09-02  4264) 				folio->private = NULL;
0bcac06f27d752 Minchan Kim             2017-11-15  4265  			}
aa8d22a11da933 Minchan Kim             2017-11-15  4266  		} else {
e9e9b7ecee4a13 Minchan Kim             2018-04-05  4267  			page = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE,
e9e9b7ecee4a13 Minchan Kim             2018-04-05  4268  						vmf);
63ad4add382305 Matthew Wilcox (Oracle  2022-09-02  4269) 			if (page)
63ad4add382305 Matthew Wilcox (Oracle  2022-09-02  4270) 				folio = page_folio(page);
d4f9565ae598bd Matthew Wilcox (Oracle  2022-09-02  4271) 			swapcache = folio;
0bcac06f27d752 Minchan Kim             2017-11-15  4272  		}
0bcac06f27d752 Minchan Kim             2017-11-15  4273  
d4f9565ae598bd Matthew Wilcox (Oracle  2022-09-02  4274) 		if (!folio) {
^1da177e4c3f41 Linus Torvalds          2005-04-16  4275  			/*
8f4e2101fd7df9 Hugh Dickins            2005-10-29  4276  			 * Back out if somebody else faulted in this pte
8f4e2101fd7df9 Hugh Dickins            2005-10-29  4277  			 * while we released the pte lock.
^1da177e4c3f41 Linus Torvalds          2005-04-16  4278  			 */
82b0f8c39a3869 Jan Kara                2016-12-14  4279  			vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
82b0f8c39a3869 Jan Kara                2016-12-14  4280  					vmf->address, &vmf->ptl);
c33c794828f212 Ryan Roberts            2023-06-12  4281  			if (likely(vmf->pte &&
c33c794828f212 Ryan Roberts            2023-06-12  4282  				   pte_same(ptep_get(vmf->pte), vmf->orig_pte)))
^1da177e4c3f41 Linus Torvalds          2005-04-16  4283  				ret = VM_FAULT_OOM;
65500d234e74fc Hugh Dickins            2005-10-29  4284  			goto unlock;
^1da177e4c3f41 Linus Torvalds          2005-04-16  4285  		}
^1da177e4c3f41 Linus Torvalds          2005-04-16  4286  
^1da177e4c3f41 Linus Torvalds          2005-04-16  4287  		/* Had to read the page from swap area: Major fault */
^1da177e4c3f41 Linus Torvalds          2005-04-16  4288  		ret = VM_FAULT_MAJOR;
f8891e5e1f93a1 Christoph Lameter       2006-06-30  4289  		count_vm_event(PGMAJFAULT);
2262185c5b287f Roman Gushchin          2017-07-06  4290  		count_memcg_event_mm(vma->vm_mm, PGMAJFAULT);
d1737fdbec7f90 Andi Kleen              2009-09-16  4291  	} else if (PageHWPoison(page)) {
71f72525dfaaec Wu Fengguang            2009-12-16  4292  		/*
71f72525dfaaec Wu Fengguang            2009-12-16  4293  		 * hwpoisoned dirty swapcache pages are kept for killing
71f72525dfaaec Wu Fengguang            2009-12-16  4294  		 * owner processes (which may be unknown at hwpoison time)
71f72525dfaaec Wu Fengguang            2009-12-16  4295  		 */
d1737fdbec7f90 Andi Kleen              2009-09-16  4296  		ret = VM_FAULT_HWPOISON;
4779cb31c0ee3b Andi Kleen              2009-10-14  4297  		goto out_release;
^1da177e4c3f41 Linus Torvalds          2005-04-16  4298  	}
^1da177e4c3f41 Linus Torvalds          2005-04-16  4299  
fdc724d6aa44ef Suren Baghdasaryan      2023-06-30  4300  	ret |= folio_lock_or_retry(folio, vmf);
fdc724d6aa44ef Suren Baghdasaryan      2023-06-30  4301  	if (ret & VM_FAULT_RETRY)
d065bd810b6deb Michel Lespinasse       2010-10-26  4302  		goto out_release;
073e587ec2cc37 KAMEZAWA Hiroyuki       2008-10-18  4303  
84d60fdd3733fb David Hildenbrand       2022-03-24  4304  	if (swapcache) {
4969c1192d15af Andrea Arcangeli        2010-09-09  4305  		/*
3b344157c0c15b Matthew Wilcox (Oracle  2022-09-02  4306) 		 * Make sure folio_free_swap() or swapoff did not release the
84d60fdd3733fb David Hildenbrand       2022-03-24  4307  		 * swapcache from under us.  The page pin, and pte_same test
84d60fdd3733fb David Hildenbrand       2022-03-24  4308  		 * below, are not enough to exclude that.  Even if it is still
84d60fdd3733fb David Hildenbrand       2022-03-24  4309  		 * swapcache, we need to check that the page's swap has not
84d60fdd3733fb David Hildenbrand       2022-03-24  4310  		 * changed.
4969c1192d15af Andrea Arcangeli        2010-09-09  4311  		 */
63ad4add382305 Matthew Wilcox (Oracle  2022-09-02  4312) 		if (unlikely(!folio_test_swapcache(folio) ||
cfeed8ffe55b37 David Hildenbrand       2023-08-21  4313  			     page_swap_entry(page).val != entry.val))
4969c1192d15af Andrea Arcangeli        2010-09-09  4314  			goto out_page;
4969c1192d15af Andrea Arcangeli        2010-09-09  4315  
84d60fdd3733fb David Hildenbrand       2022-03-24  4316  		/*
84d60fdd3733fb David Hildenbrand       2022-03-24  4317  		 * KSM sometimes has to copy on read faults, for example, if
84d60fdd3733fb David Hildenbrand       2022-03-24  4318  		 * page->index of !PageKSM() pages would be nonlinear inside the
84d60fdd3733fb David Hildenbrand       2022-03-24  4319  		 * anon VMA -- PageKSM() is lost on actual swapout.
84d60fdd3733fb David Hildenbrand       2022-03-24  4320  		 */
96db66d9c8f3c1 Matthew Wilcox (Oracle  2023-12-11  4321) 		folio = ksm_might_need_to_copy(folio, vma, vmf->address);
96db66d9c8f3c1 Matthew Wilcox (Oracle  2023-12-11  4322) 		if (unlikely(!folio)) {
5ad6468801d28c Hugh Dickins            2009-12-14  4323  			ret = VM_FAULT_OOM;
96db66d9c8f3c1 Matthew Wilcox (Oracle  2023-12-11  4324) 			folio = swapcache;
4969c1192d15af Andrea Arcangeli        2010-09-09  4325  			goto out_page;
96db66d9c8f3c1 Matthew Wilcox (Oracle  2023-12-11  4326) 		} else if (unlikely(folio == ERR_PTR(-EHWPOISON))) {
6b970599e807ea Kefeng Wang             2022-12-09  4327  			ret = VM_FAULT_HWPOISON;
96db66d9c8f3c1 Matthew Wilcox (Oracle  2023-12-11  4328) 			folio = swapcache;
6b970599e807ea Kefeng Wang             2022-12-09  4329  			goto out_page;
4969c1192d15af Andrea Arcangeli        2010-09-09  4330  		}
96db66d9c8f3c1 Matthew Wilcox (Oracle  2023-12-11  4331) 		if (folio != swapcache)
96db66d9c8f3c1 Matthew Wilcox (Oracle  2023-12-11  4332) 			page = folio_page(folio, 0);
c145e0b47c77eb David Hildenbrand       2022-03-24  4333  
c145e0b47c77eb David Hildenbrand       2022-03-24  4334  		/*
c145e0b47c77eb David Hildenbrand       2022-03-24  4335  		 * If we want to map a page that's in the swapcache writable, we
c145e0b47c77eb David Hildenbrand       2022-03-24  4336  		 * have to detect via the refcount if we're really the exclusive
c145e0b47c77eb David Hildenbrand       2022-03-24  4337  		 * owner. Try removing the extra reference from the local LRU
1fec6890bf2247 Matthew Wilcox (Oracle  2023-06-21  4338) 		 * caches if required.
c145e0b47c77eb David Hildenbrand       2022-03-24  4339  		 */
d4f9565ae598bd Matthew Wilcox (Oracle  2022-09-02  4340) 		if ((vmf->flags & FAULT_FLAG_WRITE) && folio == swapcache &&
63ad4add382305 Matthew Wilcox (Oracle  2022-09-02  4341) 		    !folio_test_ksm(folio) && !folio_test_lru(folio))
c145e0b47c77eb David Hildenbrand       2022-03-24  4342  			lru_add_drain();
84d60fdd3733fb David Hildenbrand       2022-03-24  4343  	}
5ad6468801d28c Hugh Dickins            2009-12-14  4344  
4231f8425833b1 Kefeng Wang             2023-03-02  4345  	folio_throttle_swaprate(folio, GFP_KERNEL);
8a9f3ccd24741b Balbir Singh            2008-02-07  4346  
^1da177e4c3f41 Linus Torvalds          2005-04-16  4347  	/*
8f4e2101fd7df9 Hugh Dickins            2005-10-29  4348  	 * Back out if somebody else already faulted in this pte.
^1da177e4c3f41 Linus Torvalds          2005-04-16  4349  	 */
82b0f8c39a3869 Jan Kara                2016-12-14  4350  	vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
82b0f8c39a3869 Jan Kara                2016-12-14  4351  			&vmf->ptl);
c33c794828f212 Ryan Roberts            2023-06-12  4352  	if (unlikely(!vmf->pte || !pte_same(ptep_get(vmf->pte), vmf->orig_pte)))
b81074800b98ac Kirill Korotaev         2005-05-16  4353  		goto out_nomap;
b81074800b98ac Kirill Korotaev         2005-05-16  4354  
63ad4add382305 Matthew Wilcox (Oracle  2022-09-02  4355) 	if (unlikely(!folio_test_uptodate(folio))) {
b81074800b98ac Kirill Korotaev         2005-05-16  4356  		ret = VM_FAULT_SIGBUS;
b81074800b98ac Kirill Korotaev         2005-05-16  4357  		goto out_nomap;
^1da177e4c3f41 Linus Torvalds          2005-04-16  4358  	}
^1da177e4c3f41 Linus Torvalds          2005-04-16  4359  
684d098daf0b3a Chuanhua Han            2024-07-26  4360  	/* allocated large folios for SWP_SYNCHRONOUS_IO */
684d098daf0b3a Chuanhua Han            2024-07-26  4361  	if (folio_test_large(folio) && !folio_test_swapcache(folio)) {
684d098daf0b3a Chuanhua Han            2024-07-26  4362  		unsigned long nr = folio_nr_pages(folio);
684d098daf0b3a Chuanhua Han            2024-07-26  4363  		unsigned long folio_start = ALIGN_DOWN(vmf->address, nr * PAGE_SIZE);
684d098daf0b3a Chuanhua Han            2024-07-26  4364  		unsigned long idx = (vmf->address - folio_start) / PAGE_SIZE;
684d098daf0b3a Chuanhua Han            2024-07-26  4365  		pte_t *folio_ptep = vmf->pte - idx;
684d098daf0b3a Chuanhua Han            2024-07-26  4366  
684d098daf0b3a Chuanhua Han            2024-07-26  4367  		if (!can_swapin_thp(vmf, folio_ptep, nr))
684d098daf0b3a Chuanhua Han            2024-07-26  4368  			goto out_nomap;
684d098daf0b3a Chuanhua Han            2024-07-26  4369  
684d098daf0b3a Chuanhua Han            2024-07-26  4370  		page_idx = idx;
684d098daf0b3a Chuanhua Han            2024-07-26  4371  		address = folio_start;
684d098daf0b3a Chuanhua Han            2024-07-26  4372  		ptep = folio_ptep;
684d098daf0b3a Chuanhua Han            2024-07-26  4373  		goto check_folio;

Let's say we hit this goto

684d098daf0b3a Chuanhua Han            2024-07-26  4374  	}
684d098daf0b3a Chuanhua Han            2024-07-26  4375  
508758960b8d89 Chuanhua Han            2024-05-29  4376  	nr_pages = 1;
508758960b8d89 Chuanhua Han            2024-05-29  4377  	page_idx = 0;
508758960b8d89 Chuanhua Han            2024-05-29  4378  	address = vmf->address;
508758960b8d89 Chuanhua Han            2024-05-29  4379  	ptep = vmf->pte;
508758960b8d89 Chuanhua Han            2024-05-29  4380  	if (folio_test_large(folio) && folio_test_swapcache(folio)) {
508758960b8d89 Chuanhua Han            2024-05-29  4381  		int nr = folio_nr_pages(folio);
508758960b8d89 Chuanhua Han            2024-05-29  4382  		unsigned long idx = folio_page_idx(folio, page);
508758960b8d89 Chuanhua Han            2024-05-29  4383  		unsigned long folio_start = address - idx * PAGE_SIZE;
508758960b8d89 Chuanhua Han            2024-05-29  4384  		unsigned long folio_end = folio_start + nr * PAGE_SIZE;
508758960b8d89 Chuanhua Han            2024-05-29  4385  		pte_t *folio_ptep;
508758960b8d89 Chuanhua Han            2024-05-29  4386  		pte_t folio_pte;
508758960b8d89 Chuanhua Han            2024-05-29  4387  
508758960b8d89 Chuanhua Han            2024-05-29  4388  		if (unlikely(folio_start < max(address & PMD_MASK, vma->vm_start)))
508758960b8d89 Chuanhua Han            2024-05-29  4389  			goto check_folio;
508758960b8d89 Chuanhua Han            2024-05-29  4390  		if (unlikely(folio_end > pmd_addr_end(address, vma->vm_end)))
508758960b8d89 Chuanhua Han            2024-05-29  4391  			goto check_folio;
508758960b8d89 Chuanhua Han            2024-05-29  4392  
508758960b8d89 Chuanhua Han            2024-05-29  4393  		folio_ptep = vmf->pte - idx;
508758960b8d89 Chuanhua Han            2024-05-29  4394  		folio_pte = ptep_get(folio_ptep);
508758960b8d89 Chuanhua Han            2024-05-29  4395  		if (!pte_same(folio_pte, pte_move_swp_offset(vmf->orig_pte, -idx)) ||
508758960b8d89 Chuanhua Han            2024-05-29  4396  		    swap_pte_batch(folio_ptep, nr, folio_pte) != nr)
508758960b8d89 Chuanhua Han            2024-05-29  4397  			goto check_folio;
508758960b8d89 Chuanhua Han            2024-05-29  4398  
508758960b8d89 Chuanhua Han            2024-05-29  4399  		page_idx = idx;
508758960b8d89 Chuanhua Han            2024-05-29  4400  		address = folio_start;
508758960b8d89 Chuanhua Han            2024-05-29  4401  		ptep = folio_ptep;
508758960b8d89 Chuanhua Han            2024-05-29  4402  		nr_pages = nr;
508758960b8d89 Chuanhua Han            2024-05-29  4403  		entry = folio->swap;
508758960b8d89 Chuanhua Han            2024-05-29  4404  		page = &folio->page;
508758960b8d89 Chuanhua Han            2024-05-29  4405  	}
508758960b8d89 Chuanhua Han            2024-05-29  4406  
508758960b8d89 Chuanhua Han            2024-05-29  4407  check_folio:
78fbe906cc900b David Hildenbrand       2022-05-09  4408  	/*
78fbe906cc900b David Hildenbrand       2022-05-09  4409  	 * PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte
78fbe906cc900b David Hildenbrand       2022-05-09  4410  	 * must never point at an anonymous page in the swapcache that is
78fbe906cc900b David Hildenbrand       2022-05-09  4411  	 * PG_anon_exclusive. Sanity check that this holds and especially, that
78fbe906cc900b David Hildenbrand       2022-05-09  4412  	 * no filesystem set PG_mappedtodisk on a page in the swapcache. Sanity
78fbe906cc900b David Hildenbrand       2022-05-09  4413  	 * check after taking the PT lock and making sure that nobody
78fbe906cc900b David Hildenbrand       2022-05-09  4414  	 * concurrently faulted in this page and set PG_anon_exclusive.
78fbe906cc900b David Hildenbrand       2022-05-09  4415  	 */
63ad4add382305 Matthew Wilcox (Oracle  2022-09-02  4416) 	BUG_ON(!folio_test_anon(folio) && folio_test_mappedtodisk(folio));
63ad4add382305 Matthew Wilcox (Oracle  2022-09-02  4417) 	BUG_ON(folio_test_anon(folio) && PageAnonExclusive(page));
78fbe906cc900b David Hildenbrand       2022-05-09  4418  
1493a1913e34b0 David Hildenbrand       2022-05-09  4419  	/*
1493a1913e34b0 David Hildenbrand       2022-05-09  4420  	 * Check under PT lock (to protect against concurrent fork() sharing
1493a1913e34b0 David Hildenbrand       2022-05-09  4421  	 * the swap entry concurrently) for certainly exclusive pages.
1493a1913e34b0 David Hildenbrand       2022-05-09  4422  	 */
63ad4add382305 Matthew Wilcox (Oracle  2022-09-02  4423) 	if (!folio_test_ksm(folio)) {
1493a1913e34b0 David Hildenbrand       2022-05-09  4424  		exclusive = pte_swp_exclusive(vmf->orig_pte);
d4f9565ae598bd Matthew Wilcox (Oracle  2022-09-02  4425) 		if (folio != swapcache) {
1493a1913e34b0 David Hildenbrand       2022-05-09  4426  			/*
1493a1913e34b0 David Hildenbrand       2022-05-09  4427  			 * We have a fresh page that is not exposed to the
1493a1913e34b0 David Hildenbrand       2022-05-09  4428  			 * swapcache -> certainly exclusive.
1493a1913e34b0 David Hildenbrand       2022-05-09  4429  			 */
1493a1913e34b0 David Hildenbrand       2022-05-09  4430  			exclusive = true;
63ad4add382305 Matthew Wilcox (Oracle  2022-09-02  4431) 		} else if (exclusive && folio_test_writeback(folio) &&
eacde32757c756 Miaohe Lin              2022-05-19  4432  			  data_race(si->flags & SWP_STABLE_WRITES)) {
1493a1913e34b0 David Hildenbrand       2022-05-09  4433  			/*
1493a1913e34b0 David Hildenbrand       2022-05-09  4434  			 * This is tricky: not all swap backends support
1493a1913e34b0 David Hildenbrand       2022-05-09  4435  			 * concurrent page modifications while under writeback.
1493a1913e34b0 David Hildenbrand       2022-05-09  4436  			 *
1493a1913e34b0 David Hildenbrand       2022-05-09  4437  			 * So if we stumble over such a page in the swapcache
1493a1913e34b0 David Hildenbrand       2022-05-09  4438  			 * we must not set the page exclusive, otherwise we can
1493a1913e34b0 David Hildenbrand       2022-05-09  4439  			 * map it writable without further checks and modify it
1493a1913e34b0 David Hildenbrand       2022-05-09  4440  			 * while still under writeback.
1493a1913e34b0 David Hildenbrand       2022-05-09  4441  			 *
1493a1913e34b0 David Hildenbrand       2022-05-09  4442  			 * For these problematic swap backends, simply drop the
1493a1913e34b0 David Hildenbrand       2022-05-09  4443  			 * exclusive marker: this is perfectly fine as we start
1493a1913e34b0 David Hildenbrand       2022-05-09  4444  			 * writeback only if we fully unmapped the page and
1493a1913e34b0 David Hildenbrand       2022-05-09  4445  			 * there are no unexpected references on the page after
1493a1913e34b0 David Hildenbrand       2022-05-09  4446  			 * unmapping succeeded. After fully unmapped, no
1493a1913e34b0 David Hildenbrand       2022-05-09  4447  			 * further GUP references (FOLL_GET and FOLL_PIN) can
1493a1913e34b0 David Hildenbrand       2022-05-09  4448  			 * appear, so dropping the exclusive marker and mapping
1493a1913e34b0 David Hildenbrand       2022-05-09  4449  			 * it only R/O is fine.
1493a1913e34b0 David Hildenbrand       2022-05-09  4450  			 */
1493a1913e34b0 David Hildenbrand       2022-05-09  4451  			exclusive = false;
1493a1913e34b0 David Hildenbrand       2022-05-09  4452  		}
1493a1913e34b0 David Hildenbrand       2022-05-09  4453  	}
1493a1913e34b0 David Hildenbrand       2022-05-09  4454  
6dca4ac6fc91fd Peter Collingbourne     2023-05-22  4455  	/*
6dca4ac6fc91fd Peter Collingbourne     2023-05-22  4456  	 * Some architectures may have to restore extra metadata to the page
6dca4ac6fc91fd Peter Collingbourne     2023-05-22  4457  	 * when reading from swap. This metadata may be indexed by swap entry
6dca4ac6fc91fd Peter Collingbourne     2023-05-22  4458  	 * so this must be called before swap_free().
6dca4ac6fc91fd Peter Collingbourne     2023-05-22  4459  	 */
f238b8c33c6738 Barry Song              2024-03-23  4460  	arch_swap_restore(folio_swap(entry, folio), folio);
6dca4ac6fc91fd Peter Collingbourne     2023-05-22  4461  
8c7c6e34a1256a KAMEZAWA Hiroyuki       2009-01-07  4462  	/*
c145e0b47c77eb David Hildenbrand       2022-03-24  4463  	 * Remove the swap entry and conditionally try to free up the swapcache.
c145e0b47c77eb David Hildenbrand       2022-03-24  4464  	 * We're already holding a reference on the page but haven't mapped it
c145e0b47c77eb David Hildenbrand       2022-03-24  4465  	 * yet.
8c7c6e34a1256a KAMEZAWA Hiroyuki       2009-01-07  4466  	 */
508758960b8d89 Chuanhua Han            2024-05-29 @4467  	swap_free_nr(entry, nr_pages);
                                                                                    ^^^^^^^^

Smatch warning. The code is a bit complicated so it could be a false positive.

a160e5377b55bc Matthew Wilcox (Oracle  2022-09-02  4468) 	if (should_try_to_free_swap(folio, vma, vmf->flags))
a160e5377b55bc Matthew Wilcox (Oracle  2022-09-02  4469) 		folio_free_swap(folio);
^1da177e4c3f41 Linus Torvalds          2005-04-16  4470  
508758960b8d89 Chuanhua Han            2024-05-29  4471  	add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages);
508758960b8d89 Chuanhua Han            2024-05-29  4472  	add_mm_counter(vma->vm_mm, MM_SWAPENTS, -nr_pages);
^1da177e4c3f41 Linus Torvalds          2005-04-16  4473  	pte = mk_pte(page, vma->vm_page_prot);
c18160dba5ff63 Barry Song              2024-06-02  4474  	if (pte_swp_soft_dirty(vmf->orig_pte))
c18160dba5ff63 Barry Song              2024-06-02  4475  		pte = pte_mksoft_dirty(pte);
c18160dba5ff63 Barry Song              2024-06-02  4476  	if (pte_swp_uffd_wp(vmf->orig_pte))
c18160dba5ff63 Barry Song              2024-06-02  4477  		pte = pte_mkuffd_wp(pte);
c145e0b47c77eb David Hildenbrand       2022-03-24  4478  
c145e0b47c77eb David Hildenbrand       2022-03-24  4479  	/*
1493a1913e34b0 David Hildenbrand       2022-05-09  4480  	 * Same logic as in do_wp_page(); however, optimize for pages that are
1493a1913e34b0 David Hildenbrand       2022-05-09  4481  	 * certainly not shared either because we just allocated them without
1493a1913e34b0 David Hildenbrand       2022-05-09  4482  	 * exposing them to the swapcache or because the swap entry indicates
1493a1913e34b0 David Hildenbrand       2022-05-09  4483  	 * exclusivity.
c145e0b47c77eb David Hildenbrand       2022-03-24  4484  	 */
63ad4add382305 Matthew Wilcox (Oracle  2022-09-02  4485) 	if (!folio_test_ksm(folio) &&
63ad4add382305 Matthew Wilcox (Oracle  2022-09-02  4486) 	    (exclusive || folio_ref_count(folio) == 1)) {
c18160dba5ff63 Barry Song              2024-06-02  4487  		if ((vma->vm_flags & VM_WRITE) && !userfaultfd_pte_wp(vma, pte) &&
20dfa5b7adc5a1 Barry Song              2024-06-08  4488  		    !pte_needs_soft_dirty_wp(vma, pte)) {
c18160dba5ff63 Barry Song              2024-06-02  4489  			pte = pte_mkwrite(pte, vma);
6c287605fd5646 David Hildenbrand       2022-05-09  4490  			if (vmf->flags & FAULT_FLAG_WRITE) {
c18160dba5ff63 Barry Song              2024-06-02  4491  				pte = pte_mkdirty(pte);
82b0f8c39a3869 Jan Kara                2016-12-14  4492  				vmf->flags &= ~FAULT_FLAG_WRITE;
6c287605fd5646 David Hildenbrand       2022-05-09  4493  			}
c18160dba5ff63 Barry Song              2024-06-02  4494  		}
14f9135d547060 David Hildenbrand       2022-05-09  4495  		rmap_flags |= RMAP_EXCLUSIVE;
^1da177e4c3f41 Linus Torvalds          2005-04-16  4496  	}
508758960b8d89 Chuanhua Han            2024-05-29  4497  	folio_ref_add(folio, nr_pages - 1);
508758960b8d89 Chuanhua Han            2024-05-29  4498  	flush_icache_pages(vma, page, nr_pages);
508758960b8d89 Chuanhua Han            2024-05-29  4499  	vmf->orig_pte = pte_advance_pfn(pte, page_idx);
0bcac06f27d752 Minchan Kim             2017-11-15  4500  
0bcac06f27d752 Minchan Kim             2017-11-15  4501  	/* ksm created a completely new copy */
d4f9565ae598bd Matthew Wilcox (Oracle  2022-09-02  4502) 	if (unlikely(folio != swapcache && swapcache)) {
15bde4abab734c Barry Song              2024-06-18  4503  		folio_add_new_anon_rmap(folio, vma, address, RMAP_EXCLUSIVE);
63ad4add382305 Matthew Wilcox (Oracle  2022-09-02  4504) 		folio_add_lru_vma(folio, vma);
9ae2feacedde16 Barry Song              2024-06-18  4505  	} else if (!folio_test_anon(folio)) {
9ae2feacedde16 Barry Song              2024-06-18  4506  		/*
684d098daf0b3a Chuanhua Han            2024-07-26  4507  		 * We currently only expect small !anon folios which are either
684d098daf0b3a Chuanhua Han            2024-07-26  4508  		 * fully exclusive or fully shared, or new allocated large folios
684d098daf0b3a Chuanhua Han            2024-07-26  4509  		 * which are fully exclusive. If we ever get large folios within
684d098daf0b3a Chuanhua Han            2024-07-26  4510  		 * swapcache here, we have to be careful.
9ae2feacedde16 Barry Song              2024-06-18  4511  		 */
684d098daf0b3a Chuanhua Han            2024-07-26  4512  		VM_WARN_ON_ONCE(folio_test_large(folio) && folio_test_swapcache(folio));
9ae2feacedde16 Barry Song              2024-06-18  4513  		VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
9ae2feacedde16 Barry Song              2024-06-18  4514  		folio_add_new_anon_rmap(folio, vma, address, rmap_flags);
0bcac06f27d752 Minchan Kim             2017-11-15  4515  	} else {
508758960b8d89 Chuanhua Han            2024-05-29  4516  		folio_add_anon_rmap_ptes(folio, page, nr_pages, vma, address,
b832a354d787bf David Hildenbrand       2023-12-20  4517  					rmap_flags);
00501b531c4723 Johannes Weiner         2014-08-08  4518  	}
^1da177e4c3f41 Linus Torvalds          2005-04-16  4519  
63ad4add382305 Matthew Wilcox (Oracle  2022-09-02  4520) 	VM_BUG_ON(!folio_test_anon(folio) ||
63ad4add382305 Matthew Wilcox (Oracle  2022-09-02  4521) 			(pte_write(pte) && !PageAnonExclusive(page)));
508758960b8d89 Chuanhua Han            2024-05-29  4522  	set_ptes(vma->vm_mm, address, ptep, pte, nr_pages);
508758960b8d89 Chuanhua Han            2024-05-29  4523  	arch_do_swap_page_nr(vma->vm_mm, vma, address,
508758960b8d89 Chuanhua Han            2024-05-29  4524  			pte, pte, nr_pages);
1eba86c096e35e Pasha Tatashin          2022-01-14  4525  
63ad4add382305 Matthew Wilcox (Oracle  2022-09-02  4526) 	folio_unlock(folio);
d4f9565ae598bd Matthew Wilcox (Oracle  2022-09-02  4527) 	if (folio != swapcache && swapcache) {
4969c1192d15af Andrea Arcangeli        2010-09-09  4528  		/*
4969c1192d15af Andrea Arcangeli        2010-09-09  4529  		 * Hold the lock to avoid the swap entry to be reused
4969c1192d15af Andrea Arcangeli        2010-09-09  4530  		 * until we take the PT lock for the pte_same() check
4969c1192d15af Andrea Arcangeli        2010-09-09  4531  		 * (to avoid false positives from pte_same). For
4969c1192d15af Andrea Arcangeli        2010-09-09  4532  		 * further safety release the lock after the swap_free
4969c1192d15af Andrea Arcangeli        2010-09-09  4533  		 * so that the swap count won't change under a
4969c1192d15af Andrea Arcangeli        2010-09-09  4534  		 * parallel locked swapcache.
4969c1192d15af Andrea Arcangeli        2010-09-09  4535  		 */
d4f9565ae598bd Matthew Wilcox (Oracle  2022-09-02  4536) 		folio_unlock(swapcache);
d4f9565ae598bd Matthew Wilcox (Oracle  2022-09-02  4537) 		folio_put(swapcache);
4969c1192d15af Andrea Arcangeli        2010-09-09  4538  	}
c475a8ab625d56 Hugh Dickins            2005-06-21  4539  
82b0f8c39a3869 Jan Kara                2016-12-14  4540  	if (vmf->flags & FAULT_FLAG_WRITE) {
2994302bc8a171 Jan Kara                2016-12-14  4541  		ret |= do_wp_page(vmf);
61469f1d51777f Hugh Dickins            2008-03-04  4542  		if (ret & VM_FAULT_ERROR)
61469f1d51777f Hugh Dickins            2008-03-04  4543  			ret &= VM_FAULT_ERROR;
^1da177e4c3f41 Linus Torvalds          2005-04-16  4544  		goto out;
^1da177e4c3f41 Linus Torvalds          2005-04-16  4545  	}
^1da177e4c3f41 Linus Torvalds          2005-04-16  4546  
^1da177e4c3f41 Linus Torvalds          2005-04-16  4547  	/* No need to invalidate - it was non-present before */
508758960b8d89 Chuanhua Han            2024-05-29  4548  	update_mmu_cache_range(vmf, vma, address, ptep, nr_pages);
65500d234e74fc Hugh Dickins            2005-10-29  4549  unlock:
3db82b9374ca92 Hugh Dickins            2023-06-08  4550  	if (vmf->pte)
82b0f8c39a3869 Jan Kara                2016-12-14  4551  		pte_unmap_unlock(vmf->pte, vmf->ptl);
^1da177e4c3f41 Linus Torvalds          2005-04-16  4552  out:
13ddaf26be324a Kairui Song             2024-02-07  4553  	/* Clear the swap cache pin for direct swapin after PTL unlock */
13ddaf26be324a Kairui Song             2024-02-07  4554  	if (need_clear_cache)
684d098daf0b3a Chuanhua Han            2024-07-26  4555  		swapcache_clear_nr(si, entry, nr_pages);
2799e77529c2a2 Miaohe Lin              2021-06-28  4556  	if (si)
2799e77529c2a2 Miaohe Lin              2021-06-28  4557  		put_swap_device(si);
^1da177e4c3f41 Linus Torvalds          2005-04-16  4558  	return ret;
b81074800b98ac Kirill Korotaev         2005-05-16  4559  out_nomap:
3db82b9374ca92 Hugh Dickins            2023-06-08  4560  	if (vmf->pte)
82b0f8c39a3869 Jan Kara                2016-12-14  4561  		pte_unmap_unlock(vmf->pte, vmf->ptl);
bc43f75cd98158 Johannes Weiner         2009-04-30  4562  out_page:
63ad4add382305 Matthew Wilcox (Oracle  2022-09-02  4563) 	folio_unlock(folio);
4779cb31c0ee3b Andi Kleen              2009-10-14  4564  out_release:
63ad4add382305 Matthew Wilcox (Oracle  2022-09-02  4565) 	folio_put(folio);
d4f9565ae598bd Matthew Wilcox (Oracle  2022-09-02  4566) 	if (folio != swapcache && swapcache) {
d4f9565ae598bd Matthew Wilcox (Oracle  2022-09-02  4567) 		folio_unlock(swapcache);
d4f9565ae598bd Matthew Wilcox (Oracle  2022-09-02  4568) 		folio_put(swapcache);
4969c1192d15af Andrea Arcangeli        2010-09-09  4569  	}
13ddaf26be324a Kairui Song             2024-02-07  4570  	if (need_clear_cache)
684d098daf0b3a Chuanhua Han            2024-07-26  4571  		swapcache_clear_nr(si, entry, nr_pages);
2799e77529c2a2 Miaohe Lin              2021-06-28  4572  	if (si)
2799e77529c2a2 Miaohe Lin              2021-06-28  4573  		put_swap_device(si);
65500d234e74fc Hugh Dickins            2005-10-29  4574  	return ret;
^1da177e4c3f41 Linus Torvalds          2005-04-16  4575  }
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
author
#25 lore raw

PATCH v5 4/4 mm: Introduce per-thpsize swapin control policy

In reply to Barry Song

From: Barry Song <[email protected]>

Quote Ying's comment:
A user space interface can be implemented to select different swap-in order policies, similar to the mTHP allocation order policy. We need a distinct policy because the performance characteristics of memory allocation differ significantly from those of swap-in. For example, SSD read speeds can be much slower than memory allocation. With policy selection, I believe we can implement mTHP swap-in for non-SWAP_SYNCHRONOUS scenarios as well. However, users need to understand the implications of their choices. I think that it's better to start with at least always never. I believe that we will add auto in the future to tune automatically, which can be used as default finally.

---
 Documentation/admin-guide/mm/transhuge.rst |  6 +++
 include/linux/huge_mm.h                    |  1 +
 mm/huge_memory.c                           | 44 ++++++++++++++++++++++
 mm/memory.c                                |  3 +-
 4 files changed, 53 insertions(+), 1 deletion(-)
Documentation/admin-guide/mm/transhuge.rst +6 -0
index 058485daf186..2e94e956ee12 100644
@@ -144,6 +144,12 @@ hugepage sizes have enabled="never". If enabling multiple hugepage
 sizes, the kernel will select the most appropriate enabled size for a
 given allocation.
 
+Transparent Hugepage Swap-in for anonymous memory can be disabled or enabled
+by per-supported-THP-size with one of::
+
+	echo always >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/swapin_enabled
+	echo never >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/swapin_enabled
+
 It's also possible to limit defrag efforts in the VM to generate
 anonymous hugepages in case they're not immediately free to madvise
 regions or to never try to defrag memory and simply fallback to regular
include/linux/huge_mm.h +1 -0
index e25d9ebfdf89..25174305b17f 100644
@@ -92,6 +92,7 @@ extern struct kobj_attribute thpsize_shmem_enabled_attr;
 #define TVA_SMAPS		(1 << 0)	/* Will be used for procfs */
 #define TVA_IN_PF		(1 << 1)	/* Page fault handler */
 #define TVA_ENFORCE_SYSFS	(1 << 2)	/* Obey sysfs configuration */
+#define TVA_IN_SWAPIN          (1 << 3)        /* Do swap-in */
 
 #define thp_vma_allowable_order(vma, vm_flags, tva_flags, order) \
 	(!!thp_vma_allowable_orders(vma, vm_flags, tva_flags, BIT(order)))
mm/huge_memory.c +44 -0

70 lines not on this page · raw message

mm/memory.c +2 -1
index 14048e9285d4..27c77f739a2c 100644
@@ -4091,7 +4091,8 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)
 	 * and suitable for swapping THP.
 	 */
 	orders = thp_vma_allowable_orders(vma, vma->vm_flags,
-			TVA_IN_PF | TVA_ENFORCE_SYSFS, BIT(PMD_ORDER) - 1);
+			TVA_IN_PF | TVA_IN_SWAPIN | TVA_ENFORCE_SYSFS,
+			BIT(PMD_ORDER) - 1);
 	orders = thp_vma_suitable_orders(vma, vmf->address, orders);
 	orders = thp_swap_suitable_orders(swp_offset(entry), vmf->address, orders);
 
Signature · 1 line
2.34.1
#26 lore raw

Re: [PATCH v5 4/4] mm: Introduce per-thpsize swapin control policy

kernel test robot noticed the following build warnings:

Hi Barry,

kernel test robot noticed the following build warnings:

[auto build test WARNING on akpm-mm/mm-everything]

url:    https://github.com/intel-lab-lkp/linux/commits/Barry-Song/mm-swap-introduce-swapcache_prepare_nr-and-swapcache_clear_nr-for-large-folios-swap-in/20240726-181412
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/20240726094618.401593-5-21cnbao%40gmail.com
patch subject: [PATCH v5 4/4] mm: Introduce per-thpsize swapin control policy
config: x86_64-randconfig-121-20240727 (https://download.01.org/0day-ci/archive/20240727/[email protected]/config)
compiler: gcc-11 (Ubuntu 11.4.0-4ubuntu1) 11.4.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240727/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

sparse warnings: (new ones prefixed by >>)

>> mm/huge_memory.c:83:15: sparse: sparse: symbol 'huge_anon_orders_swapin_always' was not declared. Should it be static?

mm/huge_memory.c: note: in included file (through include/linux/smp.h, include/linux/lockdep.h, include/linux/spinlock.h, ...):
include/linux/list.h:83:21: sparse: sparse: self-comparison always evaluates to true
mm/huge_memory.c: note: in included file (through include/linux/mmzone.h, include/linux/gfp.h, include/linux/mm.h):
include/linux/page-flags.h:235:46: sparse: sparse: self-comparison always evaluates to false include/linux/page-flags.h:235:46: sparse: sparse: self-comparison always evaluates to false mm/huge_memory.c:1867:20: sparse: sparse: context imbalance in 'madvise_free_huge_pmd' - unexpected unlock include/linux/page-flags.h:235:46: sparse: sparse: self-comparison always evaluates to false mm/huge_memory.c:1905:28: sparse: sparse: context imbalance in 'zap_huge_pmd' - unexpected unlock mm/huge_memory.c:2016:28: sparse: sparse: context imbalance in 'move_huge_pmd' - unexpected unlock mm/huge_memory.c:2156:20: sparse: sparse: context imbalance in 'change_huge_pmd' - unexpected unlock mm/huge_memory.c:2306:12: sparse: sparse: context imbalance in '__pmd_trans_huge_lock' - wrong count at exit mm/huge_memory.c:2323:12: sparse: sparse: context imbalance in '__pud_trans_huge_lock' - wrong count at exit mm/huge_memory.c:2347:28: sparse: sparse: context imbalance in 'zap_huge_pud' - unexpected unlock mm/huge_memory.c:2426:18: sparse: sparse: context imbalance in '__split_huge_zero_page_pmd' - unexpected unlock mm/huge_memory.c:2640:18: sparse: sparse: context imbalance in '__split_huge_pmd_locked' - unexpected unlock include/linux/page-flags.h:235:46: sparse: sparse: self-comparison always evaluates to false
mm/huge_memory.c: note: in included file (through include/linux/smp.h, include/linux/lockdep.h, include/linux/spinlock.h, ...):
include/linux/list.h:83:21: sparse: sparse: self-comparison always evaluates to true
include/linux/list.h:83:21: sparse: sparse: self-comparison always evaluates to true
mm/huge_memory.c: note: in included file (through include/linux/mmzone.h, include/linux/gfp.h, include/linux/mm.h):
include/linux/page-flags.h:235:46: sparse: sparse: self-comparison always evaluates to false mm/huge_memory.c:3031:30: sparse: sparse: context imbalance in '__split_huge_page' - unexpected unlock mm/huge_memory.c:3306:17: sparse: sparse: context imbalance in 'split_huge_page_to_list_to_order' - different lock contexts for basic block mm/huge_memory.c: note: in included file (through include/linux/smp.h, include/linux/lockdep.h, include/linux/spinlock.h, ...):
include/linux/list.h:83:21: sparse: sparse: self-comparison always evaluates to true
mm/huge_memory.c: note: in included file (through include/linux/mmzone.h, include/linux/gfp.h, include/linux/mm.h):
include/linux/page-flags.h:235:46: sparse: sparse: self-comparison always evaluates to false
mm/huge_memory.c: note: in included file (through include/linux/smp.h, include/linux/lockdep.h, include/linux/spinlock.h, ...):
include/linux/list.h:83:21: sparse: sparse: self-comparison always evaluates to true
mm/huge_memory.c: note: in included file (through include/linux/mmzone.h, include/linux/gfp.h, include/linux/mm.h):
include/linux/page-flags.h:235:46: sparse: sparse: self-comparison always evaluates to false include/linux/page-flags.h:235:46: sparse: sparse: self-comparison always evaluates to false

vim +/huge_anon_orders_swapin_always +83 mm/huge_memory.c

    51	
    52	/*
    53	 * By default, transparent hugepage support is disabled in order to avoid
    54	 * risking an increased memory footprint for applications that are not
    55	 * guaranteed to benefit from it. When transparent hugepage support is
    56	 * enabled, it is for all mappings, and khugepaged scans all mappings.
    57	 * Defrag is invoked by khugepaged hugepage allocations and by page faults
    58	 * for all hugepage allocations.
    59	 */
    60	unsigned long transparent_hugepage_flags __read_mostly =
    61	#ifdef CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS
    62		(1<<TRANSPARENT_HUGEPAGE_FLAG)|
    63	#endif
    64	#ifdef CONFIG_TRANSPARENT_HUGEPAGE_MADVISE
    65		(1<<TRANSPARENT_HUGEPAGE_REQ_MADV_FLAG)|
    66	#endif
    67		(1<<TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG)|
    68		(1<<TRANSPARENT_HUGEPAGE_DEFRAG_KHUGEPAGED_FLAG)|
    69		(1<<TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG);
    70	
    71	static struct shrinker *deferred_split_shrinker;
    72	static unsigned long deferred_split_count(struct shrinker *shrink,
    73						  struct shrink_control *sc);
    74	static unsigned long deferred_split_scan(struct shrinker *shrink,
    75						 struct shrink_control *sc);
    76	
    77	static atomic_t huge_zero_refcount;
    78	struct folio *huge_zero_folio __read_mostly;
    79	unsigned long huge_zero_pfn __read_mostly = ~0UL;
    80	unsigned long huge_anon_orders_always __read_mostly;
    81	unsigned long huge_anon_orders_madvise __read_mostly;
    82	unsigned long huge_anon_orders_inherit __read_mostly;
  > 83	unsigned long huge_anon_orders_swapin_always __read_mostly;
    84	
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
author
#27 lore raw

Re: [PATCH v5 4/4] mm: Introduce per-thpsize swapin control policy

Quoted 6 lines · kernel test robot

Hi Barry,

kernel test robot noticed the following build warnings:

[auto build test WARNING on akpm-mm/mm-everything]

Hi Thanks!
Would you check if the below patch fixes the problem?

mm/huge_memory.c +1 -1
index 41460847988c..06984a325af7 100644
@@ -80,7 +80,7 @@ unsigned long huge_zero_pfn __read_mostly = ~0UL;
 unsigned long huge_anon_orders_always __read_mostly;
 unsigned long huge_anon_orders_madvise __read_mostly;
 unsigned long huge_anon_orders_inherit __read_mostly;
-unsigned long huge_anon_orders_swapin_always __read_mostly;
+static unsigned long huge_anon_orders_swapin_always __read_mostly;
 
 unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,
 					 unsigned long vm_flags,
Quoted 86 lines

url:    https://github.com/intel-lab-lkp/linux/commits/Barry-Song/mm-swap-introduce-swapcache_prepare_nr-and-swapcache_clear_nr-for-large-folios-swap-in/20240726-181412
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/20240726094618.401593-5-21cnbao%40gmail.com
patch subject: [PATCH v5 4/4] mm: Introduce per-thpsize swapin control policy
config: x86_64-randconfig-121-20240727 (https://download.01.org/0day-ci/archive/20240727/[email protected]/config)
compiler: gcc-11 (Ubuntu 11.4.0-4ubuntu1) 11.4.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240727/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

sparse warnings: (new ones prefixed by >>)

Quoted 1 line
Quoted 1 line

mm/huge_memory.c:83:15: sparse: sparse: symbol 'huge_anon_orders_swapin_always' was not declared. Should it be static?

   mm/huge_memory.c: note: in included file (through include/linux/smp.h, include/linux/lockdep.h, include/linux/spinlock.h, ...):
   include/linux/list.h:83:21: sparse: sparse: self-comparison always evaluates to true
   mm/huge_memory.c: note: in included file (through include/linux/mmzone.h, include/linux/gfp.h, include/linux/mm.h):
   include/linux/page-flags.h:235:46: sparse: sparse: self-comparison always evaluates to false include/linux/page-flags.h:235:46: sparse: sparse: self-comparison always evaluates to false mm/huge_memory.c:1867:20: sparse: sparse: context imbalance in 'madvise_free_huge_pmd' - unexpected unlock include/linux/page-flags.h:235:46: sparse: sparse: self-comparison always evaluates to false mm/huge_memory.c:1905:28: sparse: sparse: context imbalance in 'zap_huge_pmd' - unexpected unlock mm/huge_memory.c:2016:28: sparse: sparse: context imbalance in 'move_huge_pmd' - unexpected unlock mm/huge_memory.c:2156:20: sparse: sparse: context imbalance in 'change_huge_pmd' - unexpected unlock mm/huge_memory.c:2306:12: sparse: sparse: context imbalance in '__pmd_trans_huge_lock' - wrong count at exit mm/huge_memory.c:2323:12: sparse: sparse: context imbalance in '__pud_trans_huge_lock' - wrong count at exit mm/huge_memory.c:2347:28: sparse: sparse: context imbalance in 'zap_huge_pud' - unexpected unlock mm/huge_memory.c:2426:18: sparse: sparse: context imbalance in '__split_huge_zero_page_pmd' - unexpected unlock mm/huge_memory.c:2640:18: sparse: sparse: context imbalance in '__split_huge_pmd_locked' - unexpected unlock include/linux/page-flags.h:235:46: sparse: sparse: self-comparison always evaluates to false
   mm/huge_memory.c: note: in included file (through include/linux/smp.h, include/linux/lockdep.h, include/linux/spinlock.h, ...):
   include/linux/list.h:83:21: sparse: sparse: self-comparison always evaluates to true
   include/linux/list.h:83:21: sparse: sparse: self-comparison always evaluates to true
   mm/huge_memory.c: note: in included file (through include/linux/mmzone.h, include/linux/gfp.h, include/linux/mm.h):
   include/linux/page-flags.h:235:46: sparse: sparse: self-comparison always evaluates to false mm/huge_memory.c:3031:30: sparse: sparse: context imbalance in '__split_huge_page' - unexpected unlock mm/huge_memory.c:3306:17: sparse: sparse: context imbalance in 'split_huge_page_to_list_to_order' - different lock contexts for basic block mm/huge_memory.c: note: in included file (through include/linux/smp.h, include/linux/lockdep.h, include/linux/spinlock.h, ...):
   include/linux/list.h:83:21: sparse: sparse: self-comparison always evaluates to true
   mm/huge_memory.c: note: in included file (through include/linux/mmzone.h, include/linux/gfp.h, include/linux/mm.h):
   include/linux/page-flags.h:235:46: sparse: sparse: self-comparison always evaluates to false
   mm/huge_memory.c: note: in included file (through include/linux/smp.h, include/linux/lockdep.h, include/linux/spinlock.h, ...):
   include/linux/list.h:83:21: sparse: sparse: self-comparison always evaluates to true
   mm/huge_memory.c: note: in included file (through include/linux/mmzone.h, include/linux/gfp.h, include/linux/mm.h):
   include/linux/page-flags.h:235:46: sparse: sparse: self-comparison always evaluates to false include/linux/page-flags.h:235:46: sparse: sparse: self-comparison always evaluates to false

vim +/huge_anon_orders_swapin_always +83 mm/huge_memory.c

    51
    52  /*
    53   * By default, transparent hugepage support is disabled in order to avoid 54   * risking an increased memory footprint for applications that are not 55   * guaranteed to benefit from it. When transparent hugepage support is 56   * enabled, it is for all mappings, and khugepaged scans all mappings. 57   * Defrag is invoked by khugepaged hugepage allocations and by page faults 58   * for all hugepage allocations.
    59   */
    60  unsigned long transparent_hugepage_flags __read_mostly =
    61  #ifdef CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS
    62          (1<<TRANSPARENT_HUGEPAGE_FLAG)|
    63  #endif
    64  #ifdef CONFIG_TRANSPARENT_HUGEPAGE_MADVISE
    65          (1<<TRANSPARENT_HUGEPAGE_REQ_MADV_FLAG)|
    66  #endif
    67          (1<<TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG)|
    68          (1<<TRANSPARENT_HUGEPAGE_DEFRAG_KHUGEPAGED_FLAG)|
    69          (1<<TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG);
    70
    71  static struct shrinker *deferred_split_shrinker;
    72  static unsigned long deferred_split_count(struct shrinker *shrink, 73                                            struct shrink_control *sc); 74  static unsigned long deferred_split_scan(struct shrinker *shrink, 75                                           struct shrink_control *sc); 76
    77  static atomic_t huge_zero_refcount;
    78  struct folio *huge_zero_folio __read_mostly;
    79  unsigned long huge_zero_pfn __read_mostly = ~0UL;
    80  unsigned long huge_anon_orders_always __read_mostly;
    81  unsigned long huge_anon_orders_madvise __read_mostly;
    82  unsigned long huge_anon_orders_inherit __read_mostly;
  > 83  unsigned long huge_anon_orders_swapin_always __read_mostly; 84

Quoted 3 lines

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Thanks
Barry

#28 lore raw

Re: [PATCH v5 4/4] mm: Introduce per-thpsize swapin control policy

In reply to Barry Song
Quoted 10 lines · Barry Song

A user space interface can be implemented to select different swap-in order policies, similar to the mTHP allocation order policy. We need a distinct policy because the performance characteristics of memory allocation differ significantly from those of swap-in. For example, SSD read speeds can be much slower than memory allocation. With policy selection, I believe we can implement mTHP swap-in for non-SWAP_SYNCHRONOUS scenarios as well. However, users need to understand the implications of their choices. I think that it's better to start with at least always never. I believe that we will add auto in the future to tune automatically, which can be used as default finally.

I strongly disagree. Use the same sysctl as the other anonymous memory allocations.

author
#29 lore raw

Re: [PATCH v5 4/4] mm: Introduce per-thpsize swapin control policy

Quoted 13 lines · Matthew Wilcox
Quoted 10 lines · Barry Song

A user space interface can be implemented to select different swap-in order policies, similar to the mTHP allocation order policy. We need a distinct policy because the performance characteristics of memory allocation differ significantly from those of swap-in. For example, SSD read speeds can be much slower than memory allocation. With policy selection, I believe we can implement mTHP swap-in for non-SWAP_SYNCHRONOUS scenarios as well. However, users need to understand the implications of their choices. I think that it's better to start with at least always never. I believe that we will add auto in the future to tune automatically, which can be used as default finally.

Quoted 2 lines

I strongly disagree. Use the same sysctl as the other anonymous memory allocations.

In versions v1-v4, we used the same controls as anonymous memory allocations. Ying expressed concerns that this approach isn't always ideal, especially for non-zRAM devices, as SSD read speeds can be much slower than memory allocation. I think his concern is reasonable to some extent.

However, this patchset only addresses scenarios involving zRAM-like devices and will not impact SSDs. I would like to get Ying's feedback on whether it's acceptable to drop this one in v6.

Thanks
Barry

#30 lore raw

Re: [PATCH v5 4/4] mm: Introduce per-thpsize swapin control policy

In reply to Matthew Wilcox
Quoted 2 lines · Matthew Wilcox

I strongly disagree. Use the same sysctl as the other anonymous memory allocations.

I agree with Matthew here.

We also really need to stop optimizing for this weird zram case and move people to zswap instead after fixing the various issues. A special block device that isn't really a block device and needs various special hooks isn't the right abstraction for different zwap strategies.

author
#31 lore raw

Re: [PATCH v5 4/4] mm: Introduce per-thpsize swapin control policy

Quoted 6 lines · Christoph Hellwig
Quoted 2 lines · Matthew Wilcox

I strongly disagree. Use the same sysctl as the other anonymous memory allocations.

I agree with Matthew here.

The whole anonymous memory allocation control is still used here. this is just an addition: anonymous memory allocation control & swapin policy, primarily for addressing SSD concern not for zRAM in the original v4's comment.

Quoted 5 lines

We also really need to stop optimizing for this weird zram case and move people to zswap instead after fixing the various issues. A special block device that isn't really a block device and needs various special hooks isn't the right abstraction for different zwap strategies.

My understanding is zRAM is much more popularly used in embedded systems than zswap. I seldomly(or never) hear who is using zswap in Android. it seems pointless to force people to move to zswap, in embedded systems we don't have a backend real block disk device after zswap.

Quoted 1 line
#32 lore raw

Re: [PATCH v5 4/4] mm: Introduce per-thpsize swapin control policy

Quoted 10 lines · Barry Song
Quoted 4 lines

We also really need to stop optimizing for this weird zram case and move people to zswap instead after fixing the various issues. A special block device that isn't really a block device and needs various special hooks isn't the right abstraction for different zwap strategies.

My understanding is zRAM is much more popularly used in embedded systems than zswap. I seldomly(or never) hear who is using zswap in Android. it seems pointless to force people to move to zswap, in embedded systems we don't have a backend real block disk device after zswap.

Well, that is the point. zram is a horrible hack that abuses a block device to implement a feature missing the VM layer. Right now people have a reason for it because zswap requires a "real" backing device and that's fine for them and for now. But instead of building VM infrastructure around these kinds of hacks we need to fix the VM infrastructure. Chris Li has been talking about and working towards a proper swap abstraction and that needs to happen.

reviewer
#33 lore raw

Re: [PATCH v5 4/4] mm: Introduce per-thpsize swapin control policy

Quoted 6 lines · Christoph Hellwig

Well, that is the point. zram is a horrible hack that abuses a block device to implement a feature missing the VM layer. Right now people have a reason for it because zswap requires a "real" backing device and that's fine for them and for now. But instead of building VM

I completely agree with this assessment.

Quoted 3 lines

infrastructure around these kinds of hacks we need to fix the VM infrastructure. Chris Li has been talking about and working towards a proper swap abstraction and that needs to happen.

I'm also working towards something along this line. My design would add a "virtual" swap ID that will be stored in the page table, and can refer to either a real, storage-backed swap entry, or a zswap entry. zswap can then exist without any backing swap device.

There are several additional benefits of this approach:

1. We can optimize swapoff as well - the page table can still refer to the swap ID, but the ID now points to a physical page frame. swapoff code just needs to sever the link from the swap ID to the physical swap entry (which either just requires a swap ID mapping walk, or even faster if we have a reverse mapping mechanism), and update the link to the page frame instead.

2. We can take this opportunity to clean up the swap count code.

I'd be happy to collaborate/compare notes :)

author
#34 lore raw

Re: [PATCH v5 4/4] mm: Introduce per-thpsize swapin control policy

Quoted 29 lines · Nhat Pham
Quoted 6 lines · Christoph Hellwig

Well, that is the point. zram is a horrible hack that abuses a block device to implement a feature missing the VM layer. Right now people have a reason for it because zswap requires a "real" backing device and that's fine for them and for now. But instead of building VM

I completely agree with this assessment.

Quoted 3 lines

infrastructure around these kinds of hacks we need to fix the VM infrastructure. Chris Li has been talking about and working towards a proper swap abstraction and that needs to happen.

I'm also working towards something along this line. My design would add a "virtual" swap ID that will be stored in the page table, and can refer to either a real, storage-backed swap entry, or a zswap entry. zswap can then exist without any backing swap device.

There are several additional benefits of this approach:

1. We can optimize swapoff as well - the page table can still refer to the swap ID, but the ID now points to a physical page frame. swapoff code just needs to sever the link from the swap ID to the physical swap entry (which either just requires a swap ID mapping walk, or even faster if we have a reverse mapping mechanism), and update the link to the page frame instead.

Quoted 3 lines

2. We can take this opportunity to clean up the swap count code.

I'd be happy to collaborate/compare notes :)

I appreciate that you have a good plan, and I welcome the improvements in zswap. However, we need to face reality. Having a good plan doesn't mean we should wait for you to proceed.

In my experience, I've never heard of anyone using zswap in an embedded system, especially among the billions of Android devices.(Correct me if you know one.) How soon do you expect embedded systems and Android to adopt zswap? In one year, two years, five years, or ten years? Have you asked if Google plans to use zswap in Android?

Currently, zswap does not support large folios, which is why Yosry has introduced
an API like zswap_never_enabled() to allow others to explore parallel options like
mTHP swap. Meanwhile, If zswap encounters large folios, it will trigger a SIGBUS error. I believe you were involved in those discussions:

mm: zswap: add zswap_never_enabled()
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=2d4d2b1cfb85cc07f6
mm: zswap: handle incorrect attempts to load large folios
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c63f210d4891f5b1

Should everyone around the world hold off on working on mTHP swap until zswap has addressed the issue to support large folios? Not to mention whether people are ready and happy to switch to zswap.

I don't see any reason why we should wait and not start implementing something that could benefit billions of devices worldwide. Parallel exploration leads to human progress in different fields. That's why I believe Yosry's patch, which allows others to move forward, is a more considerate approach.

Thanks
Barry

reviewer
#35 lore raw

Re: [PATCH v5 4/4] mm: Introduce per-thpsize swapin control policy

Quoted 7 lines · Barry Song
Quoted 2 lines

I'd be happy to collaborate/compare notes :)

I appreciate that you have a good plan, and I welcome the improvements in zswap. However, we need to face reality. Having a good plan doesn't mean we should wait for you to proceed.

Quoted 5 lines

In my experience, I've never heard of anyone using zswap in an embedded system, especially among the billions of Android devices.(Correct me if you know one.) How soon do you expect embedded systems and Android to adopt zswap? In one year, two years, five years, or ten years? Have you asked if Google plans to use zswap in Android?

Well, no one uses zswap in an embedded environment precisely because of the aforementioned issues, which we are working to resolve :)

Quoted 8 lines

Currently, zswap does not support large folios, which is why Yosry has introduced
an API like zswap_never_enabled() to allow others to explore parallel options like
mTHP swap. Meanwhile, If zswap encounters large folios, it will trigger a SIGBUS error. I believe you were involved in those discussions:

Quoted 5 lines

mm: zswap: add zswap_never_enabled()
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=2d4d2b1cfb85cc07f6
mm: zswap: handle incorrect attempts to load large folios
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c63f210d4891f5b1

I am, and for the record I reviewed and/or ack-ed all of these patches, and provided my inputs on how to move forward with zswap's support for large folios. I do not want zswap to prevent the development of the rest of the swap ecosystem.

Quoted 4 lines

Should everyone around the world hold off on working on mTHP swap until zswap has addressed the issue to support large folios? Not to mention whether people are ready and happy to switch to zswap.

I think you misunderstood my intention. For the record, I'm not trying to stop you from improving zram, and I'm not proposing that we kill zram right away. Well, at least not until zswap reaches feature parity with zram, which, as you point out, will take awhile. Both support for large folios and swap/zswap decoupling are on our agenda, and you're welcome to participate in the discussion - for what it's worth, your attempt with zram (+zstd) is the basis/proof-of-concept for our future efforts :)

That said, I believe that there is a fundamental redundancy here, which we (zram and zswap developers) should resolve at some point by unifying the two memory compression systems. The sooner we can unify these two, the less effort we will have to spend on developing and maintaining two separate mechanisms for the same (or very similar) purpose. For instance, large folio support has to be done twice. Same goes with writeback/offloading to backend storage, etc. And I (admittedly with a bias), agree with Christoph that zswap is the way to go moving forwards.

I will not address the rest - seems like there isn't something to disagree or discuss down there :)

#36 lore raw

Re: [PATCH v5 4/4] mm: Introduce per-thpsize swapin control policy

Quoted 3 lines · Nhat Pham

I'm not proposing that we kill zram right away.

Just for the record, zram is a generic block device and has use-cases outside of swap. Just mkfs on /dev/zram0, mount it and do whatever. The "kill zram" thing is not going to fly.

maintainer
#37 lore raw

Re: [PATCH v5 4/4] mm: Introduce per-thpsize swapin control policy

In reply to Christoph Hellwig
Quoted 13 lines · Christoph Hellwig
Quoted 10 lines · Barry Song
Quoted 4 lines

We also really need to stop optimizing for this weird zram case and move people to zswap instead after fixing the various issues. A special block device that isn't really a block device and needs various special hooks isn't the right abstraction for different zwap strategies.

My understanding is zRAM is much more popularly used in embedded systems than zswap. I seldomly(or never) hear who is using zswap in Android. it seems pointless to force people to move to zswap, in embedded systems we don't have a backend real block disk device after zswap.

Quoted 7 lines

Well, that is the point. zram is a horrible hack that abuses a block device to implement a feature missing the VM layer. Right now people have a reason for it because zswap requires a "real" backing device and that's fine for them and for now. But instead of building VM infrastructure around these kinds of hacks we need to fix the VM infrastructure. Chris Li has been talking about and working towards a proper swap abstraction and that needs to happen.

Yes, I have been working on the swap allocator for the mTHP usage case. Haven't got to the zswap vs zram yet.
Currently there is a feature gap between zswap and zram, so zswap doesn't do all the stuff zram does. For the zswap "real" backend issue, Google has been using the ghost swapfile for many years. That can be one way to get around that. The patch is much smaller than overhauling the swap back end abstraction.

Currently Android uses zram and it needs to be the Android team's decision to move from zram to something else. I don't see that happening any time soon. There are practical limitations. Personally I have been using zram as some way to provide a block like device as my goto route for testing the swap stack. I still do an SSD drive swap test, but at the same time I want to reduce the SSD swap usage to avoid the wear on my SSD drive. I already destroyed two of my old HDD drives during the swap testing. The swap random seek is very unfriendly to HDD, not sure who is still using HDD for swap any more.

Anyway, removing zram is never a goal of the swap abstraction because I am still using it. We can start with reducing the feature gap between zswap and ZRAM. The end of the day, it is the Android team's call using zram or not.

Chris

#38 lore raw

Re: [PATCH v5 4/4] mm: Introduce per-thpsize swapin control policy

Quoted 3 lines · Chris Li

Currently Android uses zram and it needs to be the Android team's decision to move from zram to something else. I don't see that happening any time soon. There are practical limitations.

No one can tell anyone to stop using things. But we can stop adding new hacks for this, and especially user facing controls.

author
#39 lore raw

Re: [PATCH v5 4/4] mm: Introduce per-thpsize swapin control policy

Quoted 8 lines · Christoph Hellwig
Quoted 3 lines · Chris Li

Currently Android uses zram and it needs to be the Android team's decision to move from zram to something else. I don't see that happening any time soon. There are practical limitations.

No one can tell anyone to stop using things. But we can stop adding new hacks for this, and especially user facing controls.

Well, this user-facing control has absolutely nothing to do with zram-related hacks. It's meant to address a general issue, mainly concerning slow-speed swap devices like SSDs, as suggested in Ying's comment on v4.

Thanks
Barry

#40 lore raw

Re: [PATCH v5 4/4] mm: Introduce per-thpsize swapin control policy

In reply to Christoph Hellwig
Quoted 11 lines · Christoph Hellwig <[email protected]> 于2024年7月30日周二 00:11写道:
Quoted 2 lines · Matthew Wilcox

I strongly disagree. Use the same sysctl as the other anonymous memory allocations.

I agree with Matthew here.

We also really need to stop optimizing for this weird zram case and move people to zswap instead after fixing the various issues. A special block device that isn't really a block device and needs various special hooks isn't the right abstraction for different zwap strategies.

I disagree, zram is most popular in embedded systems (like Android).

Quoted 2 lines
Signature · 2 lines
Thanks,
Chuanhua
reviewer
#41 lore raw

Re: [PATCH v5 4/4] mm: Introduce per-thpsize swapin control policy

In reply to Matthew Wilcox
Quoted 12 lines · Matthew Wilcox
Quoted 10 lines · Barry Song

A user space interface can be implemented to select different swap-in order policies, similar to the mTHP allocation order policy. We need a distinct policy because the performance characteristics of memory allocation differ significantly from those of swap-in. For example, SSD read speeds can be much slower than memory allocation. With policy selection, I believe we can implement mTHP swap-in for non-SWAP_SYNCHRONOUS scenarios as well. However, users need to understand the implications of their choices. I think that it's better to start with at least always never. I believe that we will add auto in the future to tune automatically, which can be used as default finally.

Quoted 2 lines

I strongly disagree. Use the same sysctl as the other anonymous memory allocations.

I vaguely recall arguing in the past that just because the user has requested 2M THP that doesn't mean its the right thing to do for performance to swap-in the whole 2M in one go. That's potentially a pretty huge latency, depending on where the backend is, and it could be a waste of IO if the application never touches most of the 2M. Although the fact that the application hinted for a 2M THP in the first place hopefully means that they are storing objects that need to be accessed at similar times. Today it will be swapped in page-by-page then eventually collapsed by khugepaged.

But I think those arguments become weaker as the THP size gets smaller. 16K/64K swap-in will likely yield significant performance improvements, and I think Barry has numbers for this?

So I guess we have a few options:

- Just use the same sysfs interface as for anon allocation, And see if anyone
reports performance regressions. Investigate one of the options below if an issue is raised. That's the simplest and cleanest approach, I think.

- New sysfs interface as Barry has implemented; nobody really wants more
controls if it can be helped.

- Hardcode a size limit (e.g. 64K); I've tried this in a few different contexts
and never got any traction.

- Secret option 4: Can we allocate a full-size folio but only choose to swap-in
to it bit-by-bit? You would need a way to mark which pages of the folio are valid (e.g. per-page flag) but guess that's a non-starter given the strategy to remove per-page flags?

Thanks,
Ryan

#42 lore raw

Re: [PATCH v5 4/4] mm: Introduce per-thpsize swapin control policy

Quoted 38 lines · Ryan Roberts
Quoted 12 lines · Matthew Wilcox
Quoted 10 lines · Barry Song

A user space interface can be implemented to select different swap-in order policies, similar to the mTHP allocation order policy. We need a distinct policy because the performance characteristics of memory allocation differ significantly from those of swap-in. For example, SSD read speeds can be much slower than memory allocation. With policy selection, I believe we can implement mTHP swap-in for non-SWAP_SYNCHRONOUS scenarios as well. However, users need to understand the implications of their choices. I think that it's better to start with at least always never. I believe that we will add auto in the future to tune automatically, which can be used as default finally.

Quoted 2 lines

I strongly disagree. Use the same sysctl as the other anonymous memory allocations.

I vaguely recall arguing in the past that just because the user has requested 2M THP that doesn't mean its the right thing to do for performance to swap-in the whole 2M in one go. That's potentially a pretty huge latency, depending on where the backend is, and it could be a waste of IO if the application never touches most of the 2M. Although the fact that the application hinted for a 2M THP in the first place hopefully means that they are storing objects that need to be accessed at similar times. Today it will be swapped in page-by-page then eventually collapsed by khugepaged.

But I think those arguments become weaker as the THP size gets smaller. 16K/64K swap-in will likely yield significant performance improvements, and I think Barry has numbers for this?

So I guess we have a few options:

- Just use the same sysfs interface as for anon allocation, And see if anyone
reports performance regressions. Investigate one of the options below if an issue is raised. That's the simplest and cleanest approach, I think.

- New sysfs interface as Barry has implemented; nobody really wants more
controls if it can be helped.

Quoted 7 lines

- Hardcode a size limit (e.g. 64K); I've tried this in a few different contexts
and never got any traction.

- Secret option 4: Can we allocate a full-size folio but only choose to swap-in
to it bit-by-bit? You would need a way to mark which pages of the folio are valid (e.g. per-page flag) but guess that's a non-starter given the strategy to remove per-page flags?

Maybe we could allocate for folios in the swapcache a bitmap to store that information (folio->private).

But I am not convinced that is the right thing to do.

If we know some basic properties of the backend, can't we automatically make a pretty good decision regarding the folio size to use? E.g., slow disk, avoid 2M ...

Avoiding sysctls if possible here would really be preferable...

Signature · 3 lines
Cheers,

David / dhildenb
#43 lore raw

Re: [PATCH v5 4/4] mm: Introduce per-thpsize swapin control policy

In reply to Matthew Wilcox
Quoted 12 lines · Matthew Wilcox
Quoted 10 lines · Barry Song

A user space interface can be implemented to select different swap-in order policies, similar to the mTHP allocation order policy. We need a distinct policy because the performance characteristics of memory allocation differ significantly from those of swap-in. For example, SSD read speeds can be much slower than memory allocation. With policy selection, I believe we can implement mTHP swap-in for non-SWAP_SYNCHRONOUS scenarios as well. However, users need to understand the implications of their choices. I think that it's better to start with at least always never. I believe that we will add auto in the future to tune automatically, which can be used as default finally.

Quoted 2 lines

I strongly disagree. Use the same sysctl as the other anonymous memory allocations.

I still believe we have some reasons for this tunable.

1. As Ryan pointed out in [1], swap-in with large mTHP orders may cause long latency, which some users might want to avoid.

[1] https://lore.kernel.org/lkml/[email protected]/

2. We have readahead information available for swap-in, which is unavailable for anonymous page allocation. This enables us to build an automatic swap-in order policy similar to that for page cache order based on readahead.

3. Swap-out/swap-in cycles present an opportunity to identify hot pages. In many use cases, we can utilize mTHP for hot pages and order-0 page for cold pages, especially under memory pressure. When an mTHP has been swapped out, it indicates that it could be a cold page. Converting it to order-0 pages might be a beneficial policy.

--
Best Regards,
Huang, Ying

author
#44 lore raw

v6 cover mm: Ignite large folios swap-in support

In reply to Barry Song

From: Barry Song <[email protected]>

Currently, we support mTHP swapout but not swapin. This means that once mTHP is swapped out, it will come back as small folios when swapped in. This is particularly detrimental for devices like Android, where more than half of the memory is in swap.

The lack of mTHP swapin functionality makes mTHP a showstopper in scenarios that heavily rely on swap. This patchset introduces mTHP swap-in support. It starts with synchronous devices similar to zRAM, aiming to benefit as many users as possible with minimal changes.

-v6:
* remove the swapin control added in v5, per Willy, Christoph; The original reason for adding the swpin_enabled control was primarily to address concerns for slower devices. Currently, since we only support fast sync devices, swap-in size is less of a concern.
We’ll gain a clearer understanding of the next steps while more devices begin to support mTHP swap-in.
* add nr argument in mem_cgroup_swapin_uncharge_swap() instead of adding new API, Willy;
* swapcache_prepare() and swapcache_clear() large folios support is also removed as it has been separated per Baolin's request, right now has been in mm-unstable.
* provide more data in changelog.

-v5:
https://lore.kernel.org/linux-mm/[email protected]/

* Add swap-in control policy according to Ying's proposal. Right now only "always" and "never" are supported, later we can extend to "auto";
* Fix the comment regarding zswap_never_enabled() according to Yosry;
* Filter out unaligned swp entries earlier;
* add mem_cgroup_swapin_uncharge_swap_nr() helper

-v4:
https://lore.kernel.org/linux-mm/[email protected]/

Many parts of v3 have been merged into the mm tree with the help on reviewing from Ryan, David, Ying and Chris etc. Thank you very much!
This is the final part to allocate large folios and map them.

* Use Yosry's zswap_never_enabled(), notice there is a bug. I put the bug fix in this v4 RFC though it should be fixed in Yosry's patch
* lots of code improvement (drop large stack, hold ptl etc) according to Yosry's and Ryan's feedback
* rebased on top of the latest mm-unstable and utilized some new helpers introduced recently.

-v3:
https://lore.kernel.org/linux-mm/[email protected]/
* avoid over-writing err in __swap_duplicate_nr, pointed out by Yosry, thanks!
* fix the issue folio is charged twice for do_swap_page, separating alloc_anon_folio and alloc_swap_folio as they have many differences now on
* memcg charing
* clearing allocated folio or not

-v2:
https://lore.kernel.org/linux-mm/[email protected]/
* lots of code cleanup according to Chris's comments, thanks!
* collect Chris's ack tags, thanks!
* address David's comment on moving to use folio_add_new_anon_rmap for !folio_test_anon in do_swap_page, thanks!
* remove the MADV_PAGEOUT patch from this series as Ryan will intergrate it into swap-out series
* Apply Kairui's work of "mm/swap: fix race when skipping swapcache" on large folios swap-in as well
* fixed corrupted data(zero-filled data) in two races: zswap and a part of entries are in swapcache while some others are not in by checking SWAP_HAS_CACHE while swapping in a large folio

-v1:
https://lore.kernel.org/all/[email protected]/#t

Barry Song (1):
mm: add nr argument in mem_cgroup_swapin_uncharge_swap() helper to
support large folios

Chuanhua Han (1):
mm: support large folios swap-in for zRAM-like devices

 include/linux/memcontrol.h |   5 +-
 mm/memcontrol.c            |   7 +-
 mm/memory.c                | 211 +++++++++++++++++++++++++++++++++----
 mm/swap_state.c            |   2 +-
 4 files changed, 196 insertions(+), 29 deletions(-)
Signature · 1 line
2.34.1
author
#45 lore raw

PATCH v6 1/2 mm: add nr argument in mem_cgroup_swapin_uncharge_swap() helper to support large folios

From: Barry Song <[email protected]>

With large folios swap-in, we might need to uncharge multiple entries all together, add nr argument in mem_cgroup_swapin_uncharge_swap().

For the existing two users, just pass nr=1.

---
 include/linux/memcontrol.h | 5 +++--
 mm/memcontrol.c            | 7 ++++---
 mm/memory.c                | 2 +-
 mm/swap_state.c            | 2 +-
 4 files changed, 9 insertions(+), 7 deletions(-)
include/linux/memcontrol.h +3 -2
index 1b79760af685..44f7fb7dc0c8 100644
@@ -682,7 +682,8 @@ int mem_cgroup_hugetlb_try_charge(struct mem_cgroup *memcg, gfp_t gfp,
 
 int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm,
 				  gfp_t gfp, swp_entry_t entry);
-void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry);
+
+void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry, unsigned int nr_pages);
 
 void __mem_cgroup_uncharge(struct folio *folio);
 
@@ -1181,7 +1182,7 @@ static inline int mem_cgroup_swapin_charge_folio(struct folio *folio,
 	return 0;
 }
 
-static inline void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry)
+static inline void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry, unsigned int nr)
 {
 }
 
mm/memcontrol.c +4 -3
index b889a7fbf382..5d763c234c44 100644
@@ -4572,14 +4572,15 @@ int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm,
 
 /*
  * mem_cgroup_swapin_uncharge_swap - uncharge swap slot
- * @entry: swap entry for which the page is charged
+ * @entry: the first swap entry for which the pages are charged
+ * @nr_pages: number of pages which will be uncharged
  *
  * Call this function after successfully adding the charged page to swapcache.
  *
  * Note: This function assumes the page for which swap slot is being uncharged
  * is order 0 page.
  */
-void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry)
+void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry, unsigned int nr_pages)
 {
 	/*
 	 * Cgroup1's unified memory+swap counter has been charged with the
@@ -4599,7 +4600,7 @@ void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry)
 		 * let's not wait for it.  The page already received a
 		 * memory+swap charge, drop the swap entry duplicate.
 		 */
-		mem_cgroup_uncharge_swap(entry, 1);
+		mem_cgroup_uncharge_swap(entry, nr_pages);
 	}
 }
 
mm/memory.c +1 -1
index 4c8716cb306c..4cf4902db1ec 100644
@@ -4102,7 +4102,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 					ret = VM_FAULT_OOM;
 					goto out_page;
 				}
-				mem_cgroup_swapin_uncharge_swap(entry);
+				mem_cgroup_swapin_uncharge_swap(entry, 1);
 
 				shadow = get_shadow_from_swap_cache(entry);
 				if (shadow)
mm/swap_state.c +1 -1
index 293ff1afdca4..1159e3225754 100644
@@ -522,7 +522,7 @@ struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,
 	if (add_to_swap_cache(new_folio, entry, gfp_mask & GFP_RECLAIM_MASK, &shadow))
 		goto fail_unlock;
 
-	mem_cgroup_swapin_uncharge_swap(entry);
+	mem_cgroup_swapin_uncharge_swap(entry, 1);
 
 	if (shadow)
 		workingset_refault(new_folio, shadow);
Signature · 1 line
2.34.1
maintainer Acked-by on v6 1/2
#46 lore raw

Re: [PATCH v6 1/2] mm: add nr argument in mem_cgroup_swapin_uncharge_swap() helper to support large folios

Chris

Quoted 99 lines · Barry Song

From: Barry Song <[email protected]>

With large folios swap-in, we might need to uncharge multiple entries all together, add nr argument in mem_cgroup_swapin_uncharge_swap().

For the existing two users, just pass nr=1.

---
 include/linux/memcontrol.h | 5 +++--
 mm/memcontrol.c            | 7 ++++---
 mm/memory.c                | 2 +-
 mm/swap_state.c            | 2 +-
 4 files changed, 9 insertions(+), 7 deletions(-)
include/linux/memcontrol.h +3 -2
index 1b79760af685..44f7fb7dc0c8 100644
@@ -682,7 +682,8 @@ int mem_cgroup_hugetlb_try_charge(struct mem_cgroup *memcg, gfp_t gfp,
 
 int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm,
                                  gfp_t gfp, swp_entry_t entry);
-void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry);
+
+void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry, unsigned int nr_pages);
 
 void __mem_cgroup_uncharge(struct folio *folio);
 
@@ -1181,7 +1182,7 @@ static inline int mem_cgroup_swapin_charge_folio(struct folio *folio,
        return 0;
 }
 
-static inline void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry)
+static inline void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry, unsigned int nr)
 {
 }
 
mm/memcontrol.c +4 -3
index b889a7fbf382..5d763c234c44 100644
@@ -4572,14 +4572,15 @@ int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm,
 
 /*
  * mem_cgroup_swapin_uncharge_swap - uncharge swap slot
- * @entry: swap entry for which the page is charged
+ * @entry: the first swap entry for which the pages are charged
+ * @nr_pages: number of pages which will be uncharged
  *
  * Call this function after successfully adding the charged page to swapcache.
  *
  * Note: This function assumes the page for which swap slot is being uncharged
  * is order 0 page.
  */
-void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry)
+void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry, unsigned int nr_pages)
 {
        /*
         * Cgroup1's unified memory+swap counter has been charged with the
@@ -4599,7 +4600,7 @@ void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry)
                 * let's not wait for it.  The page already received a
                 * memory+swap charge, drop the swap entry duplicate.
                 */
-               mem_cgroup_uncharge_swap(entry, 1);
+               mem_cgroup_uncharge_swap(entry, nr_pages);
        }
 }
 
mm/memory.c +1 -1
index 4c8716cb306c..4cf4902db1ec 100644
@@ -4102,7 +4102,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
                                        ret = VM_FAULT_OOM;
                                        goto out_page;
                                }
-                               mem_cgroup_swapin_uncharge_swap(entry);
+                               mem_cgroup_swapin_uncharge_swap(entry, 1);
 
                                shadow = get_shadow_from_swap_cache(entry);
                                if (shadow)
mm/swap_state.c +1 -2
index 293ff1afdca4..1159e3225754 100644
@@ -522,7 +522,7 @@ struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,
        if (add_to_swap_cache(new_folio, entry, gfp_mask & GFP_RECLAIM_MASK, &shadow))
                goto fail_unlock;
 
-       mem_cgroup_swapin_uncharge_swap(entry);
+       mem_cgroup_swapin_uncharge_swap(entry, 1);
 
        if (shadow)
                workingset_refault(new_folio, shadow);
--

2.34.1

author
#47 lore raw

PATCH v6 2/2 mm: support large folios swap-in for zRAM-like devices

In reply to Barry Song

From: Chuanhua Han <[email protected]>

Currently, we have mTHP features, but unfortunately, without support for large folio swap-ins, once these large folios are swapped out, they are lost because mTHP swap is a one-way process. The lack of mTHP swap-in functionality prevents mTHP from being used on devices like Android that heavily rely on swap.

This patch introduces mTHP swap-in support. It starts from sync devices such as zRAM. This is probably the simplest and most common use case, benefiting billions of Android phones and similar devices with minimal implementation cost. In this straightforward scenario, large folios are always exclusive, eliminating the need to handle complex rmap and swapcache issues.

It offers several benefits:
1. Enables bidirectional mTHP swapping, allowing retrieval of mTHP after swap-out and swap-in. Large folios in the buddy system are also preserved as much as possible, rather than being fragmented due to swap-in.

2. Eliminates fragmentation in swap slots and supports successful THP_SWPOUT.

w/o this patch (Refer to the data from Chris's and Kairui's latest swap allocator optimization while running ./thp_swap_allocator_test w/o "-a" option [1]):

./thp_swap_allocator_test
Iteration 1: swpout inc: 233, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 2: swpout inc: 131, swpout fallback inc: 101, Fallback percentage: 43.53% Iteration 3: swpout inc: 71, swpout fallback inc: 155, Fallback percentage: 68.58% Iteration 4: swpout inc: 55, swpout fallback inc: 168, Fallback percentage: 75.34% Iteration 5: swpout inc: 35, swpout fallback inc: 191, Fallback percentage: 84.51% Iteration 6: swpout inc: 25, swpout fallback inc: 199, Fallback percentage: 88.84% Iteration 7: swpout inc: 23, swpout fallback inc: 205, Fallback percentage: 89.91% Iteration 8: swpout inc: 9, swpout fallback inc: 219, Fallback percentage: 96.05% Iteration 9: swpout inc: 13, swpout fallback inc: 213, Fallback percentage: 94.25% Iteration 10: swpout inc: 12, swpout fallback inc: 216, Fallback percentage: 94.74% Iteration 11: swpout inc: 16, swpout fallback inc: 213, Fallback percentage: 93.01% Iteration 12: swpout inc: 10, swpout fallback inc: 210, Fallback percentage: 95.45% Iteration 13: swpout inc: 16, swpout fallback inc: 212, Fallback percentage: 92.98% Iteration 14: swpout inc: 12, swpout fallback inc: 212, Fallback percentage: 94.64% Iteration 15: swpout inc: 15, swpout fallback inc: 211, Fallback percentage: 93.36% Iteration 16: swpout inc: 15, swpout fallback inc: 200, Fallback percentage: 93.02% Iteration 17: swpout inc: 9, swpout fallback inc: 220, Fallback percentage: 96.07%

w/ this patch (always 0%):
Iteration 1: swpout inc: 948, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 2: swpout inc: 953, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 3: swpout inc: 950, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 4: swpout inc: 952, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 5: swpout inc: 950, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 6: swpout inc: 950, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 7: swpout inc: 947, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 8: swpout inc: 950, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 9: swpout inc: 950, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 10: swpout inc: 945, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 11: swpout inc: 947, swpout fallback inc: 0, Fallback percentage: 0.00%
...

3. With both mTHP swap-out and swap-in supported, we offer the option to enable zsmalloc compression/decompression with larger granularity[2]. The upcoming optimization in zsmalloc will significantly increase swap speed and improve compression efficiency. Tested by running 100 iterations of swapping 100MiB of anon memory, the swap speed improved dramatically:
time consumption of swapin(ms) time consumption of swapout(ms)
lz4 4k 45274 90540
lz4 64k 22942 55667
zstdn 4k 85035 186585
zstdn 64k 46558 118533

The compression ratio also improved, as evaluated with 1 GiB of data:
granularity orig_data_size compr_data_size
4KiB-zstd 1048576000 246876055
64KiB-zstd 1048576000 199763892

Without mTHP swap-in, the potential optimizations in zsmalloc cannot be realized.

4. Even mTHP swap-in itself can reduce swap-in page faults by a factor of nr_pages. Swapping in content filled with the same data 0x11, w/o and w/ the patch for five rounds (Since the content is the same, decompression will be very fast. This primarily assesses the impact of reduced page faults):

swp in bandwidth(bytes/ms) w/o w/

   round1                     624152          1127501
   round2                     631672          1127501
   round3                     620459          1139756
   round4                     606113          1139756
   round5                     624152          1152281
   avg                        621310          1137359      +83%

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/

---
 mm/memory.c | 211 ++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 188 insertions(+), 23 deletions(-)
mm/memory.c +188 -23
index 4cf4902db1ec..07029532469a 100644
@@ -3986,6 +3986,152 @@ static vm_fault_t handle_pte_marker(struct vm_fault *vmf)
 	return VM_FAULT_SIGBUS;
 }
 
+/*
+ * check a range of PTEs are completely swap entries with
+ * contiguous swap offsets and the same SWAP_HAS_CACHE.
+ * ptep must be first one in the range
+ */
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+static bool can_swapin_thp(struct vm_fault *vmf, pte_t *ptep, int nr_pages)
+{
+	struct swap_info_struct *si;
+	unsigned long addr;
+	swp_entry_t entry;
+	pgoff_t offset;
+	char has_cache;
+	int idx, i;
+	pte_t pte;
+
+	addr = ALIGN_DOWN(vmf->address, nr_pages * PAGE_SIZE);
+	idx = (vmf->address - addr) / PAGE_SIZE;
+	pte = ptep_get(ptep);
+
+	if (!pte_same(pte, pte_move_swp_offset(vmf->orig_pte, -idx)))
+		return false;
+	entry = pte_to_swp_entry(pte);
+	offset = swp_offset(entry);
+	if (swap_pte_batch(ptep, nr_pages, pte) != nr_pages)
+		return false;
+
+	si = swp_swap_info(entry);
+	has_cache = si->swap_map[offset] & SWAP_HAS_CACHE;
+	for (i = 1; i < nr_pages; i++) {
+		/*
+		 * while allocating a large folio and doing swap_read_folio for the
+		 * SWP_SYNCHRONOUS_IO path, which is the case the being faulted pte
+		 * doesn't have swapcache. We need to ensure all PTEs have no cache
+		 * as well, otherwise, we might go to swap devices while the content
+		 * is in swapcache
+		 */
+		if ((si->swap_map[offset + i] & SWAP_HAS_CACHE) != has_cache)
+			return false;
+	}
+
+	return true;
+}
+
+static inline unsigned long thp_swap_suitable_orders(pgoff_t swp_offset,
+		unsigned long addr, unsigned long orders)
+{
+	int order, nr;
+
+	order = highest_order(orders);
+
+	/*
+	 * To swap-in a THP with nr pages, we require its first swap_offset
+	 * is aligned with nr. This can filter out most invalid entries.
+	 */
+	while (orders) {
+		nr = 1 << order;
+		if ((addr >> PAGE_SHIFT) % nr == swp_offset % nr)
+			break;
+		order = next_order(&orders, order);
+	}
+
+	return orders;
+}
+#else
+static inline bool can_swapin_thp(struct vm_fault *vmf, pte_t *ptep, int nr_pages)
+{
+	return false;
+}
+#endif
+
+static struct folio *alloc_swap_folio(struct vm_fault *vmf)
+{
+	struct vm_area_struct *vma = vmf->vma;
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+	unsigned long orders;
+	struct folio *folio;
+	unsigned long addr;
+	swp_entry_t entry;
+	spinlock_t *ptl;
+	pte_t *pte;
+	gfp_t gfp;
+	int order;
+
+	/*
+	 * If uffd is active for the vma we need per-page fault fidelity to
+	 * maintain the uffd semantics.
+	 */
+	if (unlikely(userfaultfd_armed(vma)))
+		goto fallback;
+
+	/*
+	 * A large swapped out folio could be partially or fully in zswap. We
+	 * lack handling for such cases, so fallback to swapping in order-0
+	 * folio.
+	 */
+	if (!zswap_never_enabled())
+		goto fallback;
+
+	entry = pte_to_swp_entry(vmf->orig_pte);
+	/*
+	 * Get a list of all the (large) orders below PMD_ORDER that are enabled
+	 * and suitable for swapping THP.
+	 */
+	orders = thp_vma_allowable_orders(vma, vma->vm_flags,
+			TVA_IN_PF | TVA_ENFORCE_SYSFS, BIT(PMD_ORDER) - 1);
+	orders = thp_vma_suitable_orders(vma, vmf->address, orders);
+	orders = thp_swap_suitable_orders(swp_offset(entry), vmf->address, orders);
+
+	if (!orders)
+		goto fallback;
+
+	pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address & PMD_MASK, &ptl);
+	if (unlikely(!pte))
+		goto fallback;
+
+	/*
+	 * For do_swap_page, find the highest order where the aligned range is
+	 * completely swap entries with contiguous swap offsets.
+	 */
+	order = highest_order(orders);
+	while (orders) {
+		addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
+		if (can_swapin_thp(vmf, pte + pte_index(addr), 1 << order))
+			break;
+		order = next_order(&orders, order);
+	}
+
+	pte_unmap_unlock(pte, ptl);
+
+	/* Try allocating the highest of the remaining orders. */
+	gfp = vma_thp_gfp_mask(vma);
+	while (orders) {
+		addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
+		folio = vma_alloc_folio(gfp, order, vma, addr, true);
+		if (folio)
+			return folio;
+		order = next_order(&orders, order);
+	}
+
+fallback:
+#endif
+	return vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, vmf->address, false);
+}
+
+
 /*
  * We enter with non-exclusive mmap_lock (to exclude vma changes,
  * but allow concurrent faults), and pte mapped but not yet locked.
@@ -4074,35 +4220,37 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 	if (!folio) {
 		if (data_race(si->flags & SWP_SYNCHRONOUS_IO) &&
 		    __swap_count(entry) == 1) {
-			/*
-			 * Prevent parallel swapin from proceeding with
-			 * the cache flag. Otherwise, another thread may
-			 * finish swapin first, free the entry, and swapout
-			 * reusing the same entry. It's undetectable as
-			 * pte_same() returns true due to entry reuse.
-			 */
-			if (swapcache_prepare(entry, 1)) {
-				/* Relax a bit to prevent rapid repeated page faults */
-				schedule_timeout_uninterruptible(1);
-				goto out;
-			}
-			need_clear_cache = true;
-
 			/* skip swapcache */
-			folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0,
-						vma, vmf->address, false);
+			folio = alloc_swap_folio(vmf);
 			page = &folio->page;
 			if (folio) {
 				__folio_set_locked(folio);
 				__folio_set_swapbacked(folio);
 
+				nr_pages = folio_nr_pages(folio);
+				if (folio_test_large(folio))
+					entry.val = ALIGN_DOWN(entry.val, nr_pages);
+				/*
+				 * Prevent parallel swapin from proceeding with
+				 * the cache flag. Otherwise, another thread may
+				 * finish swapin first, free the entry, and swapout
+				 * reusing the same entry. It's undetectable as
+				 * pte_same() returns true due to entry reuse.
+				 */
+				if (swapcache_prepare(entry, nr_pages)) {
+					/* Relax a bit to prevent rapid repeated page faults */
+					schedule_timeout_uninterruptible(1);
+					goto out_page;
+				}
+				need_clear_cache = true;
+
 				if (mem_cgroup_swapin_charge_folio(folio,
 							vma->vm_mm, GFP_KERNEL,
 							entry)) {
 					ret = VM_FAULT_OOM;
 					goto out_page;
 				}
-				mem_cgroup_swapin_uncharge_swap(entry, 1);
+				mem_cgroup_swapin_uncharge_swap(entry, nr_pages);
 
 				shadow = get_shadow_from_swap_cache(entry);
 				if (shadow)
@@ -4209,6 +4357,22 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 		goto out_nomap;
 	}
 
+	/* allocated large folios for SWP_SYNCHRONOUS_IO */
+	if (folio_test_large(folio) && !folio_test_swapcache(folio)) {
+		unsigned long nr = folio_nr_pages(folio);
+		unsigned long folio_start = ALIGN_DOWN(vmf->address, nr * PAGE_SIZE);
+		unsigned long idx = (vmf->address - folio_start) / PAGE_SIZE;
+		pte_t *folio_ptep = vmf->pte - idx;
+
+		if (!can_swapin_thp(vmf, folio_ptep, nr))
+			goto out_nomap;
+
+		page_idx = idx;
+		address = folio_start;
+		ptep = folio_ptep;
+		goto check_folio;
+	}
+
 	nr_pages = 1;
 	page_idx = 0;
 	address = vmf->address;
@@ -4340,11 +4504,12 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 		folio_add_lru_vma(folio, vma);
 	} else if (!folio_test_anon(folio)) {
 		/*
-		 * We currently only expect small !anon folios, which are either
-		 * fully exclusive or fully shared. If we ever get large folios
-		 * here, we have to be careful.
+		 * We currently only expect small !anon folios which are either
+		 * fully exclusive or fully shared, or new allocated large folios
+		 * which are fully exclusive. If we ever get large folios within
+		 * swapcache here, we have to be careful.
 		 */
-		VM_WARN_ON_ONCE(folio_test_large(folio));
+		VM_WARN_ON_ONCE(folio_test_large(folio) && folio_test_swapcache(folio));
 		VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
 		folio_add_new_anon_rmap(folio, vma, address, rmap_flags);
 	} else {
@@ -4387,7 +4552,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 out:
 	/* Clear the swap cache pin for direct swapin after PTL unlock */
 	if (need_clear_cache)
-		swapcache_clear(si, entry, 1);
+		swapcache_clear(si, entry, nr_pages);
 	if (si)
 		put_swap_device(si);
 	return ret;
@@ -4403,7 +4568,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 		folio_put(swapcache);
 	}
 	if (need_clear_cache)
-		swapcache_clear(si, entry, 1);
+		swapcache_clear(si, entry, nr_pages);
 	if (si)
 		put_swap_device(si);
 	return ret;
Signature · 1 line
2.34.1
maintainer
#48 lore raw

Re: [PATCH v6 2/2] mm: support large folios swap-in for zRAM-like devices

Quoted 28 lines · On Sat, 3 Aug 2024 00:20:31 +1200 Barry Song <[email protected]> wrote:

From: Chuanhua Han <[email protected]>

Currently, we have mTHP features, but unfortunately, without support for large folio swap-ins, once these large folios are swapped out, they are lost because mTHP swap is a one-way process. The lack of mTHP swap-in functionality prevents mTHP from being used on devices like Android that heavily rely on swap.

This patch introduces mTHP swap-in support. It starts from sync devices such as zRAM. This is probably the simplest and most common use case, benefiting billions of Android phones and similar devices with minimal implementation cost. In this straightforward scenario, large folios are always exclusive, eliminating the need to handle complex rmap and swapcache issues.

It offers several benefits:
1. Enables bidirectional mTHP swapping, allowing retrieval of mTHP after swap-out and swap-in. Large folios in the buddy system are also preserved as much as possible, rather than being fragmented due to swap-in.

2. Eliminates fragmentation in swap slots and supports successful THP_SWPOUT.

w/o this patch (Refer to the data from Chris's and Kairui's latest swap allocator optimization while running ./thp_swap_allocator_test w/o "-a" option [1]):

...

Review

Quoted 10 lines
(quoted diff) +7 -0
+static struct folio *alloc_swap_folio(struct vm_fault *vmf)
+{
+	struct vm_area_struct *vma = vmf->vma;
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
 
...
 
+#endif
+	return vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, vmf->address, false);
+}

Generates an unused-variable warning with allnoconfig. Because vma_alloc_folio_noprof() was implemented as a macro instead of an inlined C function. Why do we keep doing this.

Please check:

From: Andrew Morton <[email protected]>
Subject: mm-support-large-folios-swap-in-for-zram-like-devices-fix
Date: Sat Aug 3 11:59:00 AM PDT 2024

fix unused var warning

mm/memory.c: In function 'alloc_swap_folio':
mm/memory.c:4062:32: warning: unused variable 'vma' [-Wunused-variable]
4062 | struct vm_area_struct *vma = vmf->vma;
| ^~~

Cc Baolin Wang <[email protected]>
Cc Barry Song <[email protected]>
Cc Chris Li <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Chuanhua Han <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Gao Xiang <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Johannes Weiner <[email protected]>
Cc Kairui Song <[email protected]>
Cc Kalesh Singh <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Nhat Pham <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc Sergey Senozhatsky <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Yang Shi <[email protected]>
Cc Yosry Ahmed <[email protected]>

---

 mm/memory.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
mm/memory.c renamed from mm/memory.c~mm-support-large-folios-swap-in-for-zram-like-devices-fix +3 -2
@@ -4059,8 +4059,8 @@ static inline bool can_swapin_thp(struct
 
 static struct folio *alloc_swap_folio(struct vm_fault *vmf)
 {
-	struct vm_area_struct *vma = vmf->vma;
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
+	struct vm_area_struct *vma = vmf->vma;
 	unsigned long orders;
 	struct folio *folio;
 	unsigned long addr;
@@ -4128,7 +4128,8 @@ static struct folio *alloc_swap_folio(st
 
 fallback:
 #endif
-	return vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, vmf->address, false);
+	return vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vmf->vma,
+			       vmf->address, false);
 }
 
 

_

#49 lore raw

Re: [PATCH v6 2/2] mm: support large folios swap-in for zRAM-like devices

In reply to Barry Song

The subject feels wrong. Nothing particular about zram, it is all about SWP_SYNCHRONOUS_IO, so the Subject and commit log should state that.

Quoted 5 lines · Barry Song

From: Chuanhua Han <[email protected]>

Currently, we have mTHP features, but unfortunately, without support for large folio swap-ins, once these large folios are swapped out, they are lost because mTHP swap is a one-way process. The lack of mTHP swap-in functionality prevents

Please wrap your commit logs after 73 characters to make them readable.

Quoted 5 lines

+/*
+ * check a range of PTEs are completely swap entries with
+ * contiguous swap offsets and the same SWAP_HAS_CACHE.
+ * ptep must be first one in the range
+ */

Please capitalize the first character of block comments, make them full sentences and use up all 80 characters.

mm/memory.c static vm_fault_t handle_pte_marker(struct vm_fault *vmf) · line 4020

Quoted 3 lines
mm/memory.c +3 -0
+	for (i = 1; i < nr_pages; i++) {
+		/*
+		 * while allocating a large folio and doing swap_read_folio for the

And also do not go over 80 characters for them, which renders them really hard to read.

mm/memory.c static vm_fault_t handle_pte_marker(struct vm_fault *vmf) · line 4063

Quoted 4 lines
mm/memory.c +4 -0
+static struct folio *alloc_swap_folio(struct vm_fault *vmf)
+{
+	struct vm_area_struct *vma = vmf->vma;
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE

Please stub out the entire function.

author
#50 lore raw

Re: [PATCH v6 2/2] mm: support large folios swap-in for zRAM-like devices

Quoted 3 lines · Christoph Hellwig

The subject feels wrong. Nothing particular about zram, it is all about SWP_SYNCHRONOUS_IO, so the Subject and commit log should state that.

right.

This is absolutely for sync io, zram is the most typical one which is widely used in Android and embedded systems. Others could be nvdimm, brd.

Quoted 9 lines · Barry Song
Quoted 5 lines

From: Chuanhua Han <[email protected]>

Currently, we have mTHP features, but unfortunately, without support for large folio swap-ins, once these large folios are swapped out, they are lost because mTHP swap is a one-way process. The lack of mTHP swap-in functionality prevents

Please wrap your commit logs after 73 characters to make them readable.

ack.

Quoted 9 lines
Quoted 6 lines

+/*
+ * check a range of PTEs are completely swap entries with
+ * contiguous swap offsets and the same SWAP_HAS_CACHE.
+ * ptep must be first one in the range
+ */

Please capitalize the first character of block comments, make them full sentences and use up all 80 characters.

ack.

Quoted 8 lines

mm/memory.c static vm_fault_t handle_pte_marker(struct vm_fault *vmf) · line 4020

Quoted 4 lines
mm/memory.c +3 -0
+     for (i = 1; i < nr_pages; i++) {
+             /*
+              * while allocating a large folio and doing swap_read_folio for the

And also do not go over 80 characters for them, which renders them really hard to read.

Quoted 6 lines

mm/memory.c static vm_fault_t handle_pte_marker(struct vm_fault *vmf) · line 4063

Quoted 4 lines
mm/memory.c +4 -0
+static struct folio *alloc_swap_folio(struct vm_fault *vmf)
+{
+     struct vm_area_struct *vma = vmf->vma;
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE

Please stub out the entire function.

I assume you mean the below?

#ifdef CONFIG_TRANSPARENT_HUGEPAGE
static struct folio *alloc_swap_folio(struct vm_fault *vmf) {
}
#else
static struct folio *alloc_swap_folio(struct vm_fault *vmf) {
}
#endif

If so, this is fine to me. the only reason I am using the current pattern is that i am trying to follow the same pattern with

static struct folio *alloc_anon_folio(struct vm_fault *vmf)

{
        struct vm_area_struct *vma = vmf->vma;
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
#endif
        ...
}

Likely we also want to change that one?

Thanks
Barry

#51 lore raw

Re: [PATCH v6 2/2] mm: support large folios swap-in for zRAM-like devices

Quoted 12 lines · Barry Song

On Mon, Aug 12, 2024 at 8:27 PM Christoph Hellwig <[email protected]> wrote:
I assume you mean the below?

#ifdef CONFIG_TRANSPARENT_HUGEPAGE
static struct folio *alloc_swap_folio(struct vm_fault *vmf) {
}
#else
static struct folio *alloc_swap_folio(struct vm_fault *vmf) {
}
#endif

Yes.

Quoted 4 lines

If so, this is fine to me. the only reason I am using the current pattern is that i am trying to follow the same pattern with

static struct folio *alloc_anon_folio(struct vm_fault *vmf)

Quoted 8 lines
{
        struct vm_area_struct *vma = vmf->vma;
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
#endif
        ...
}

Likely we also want to change that one?

It would be nice to fix that a well, probably noy in this series, though.

#52 lore raw

Re: [PATCH v6 2/2] mm: support large folios swap-in for zRAM-like devices

In reply to Barry Song
Quoted 2 lines · Barry Song

From: Chuanhua Han <[email protected]>

Hi Chuanhua,

Quoted 298 lines

Currently, we have mTHP features, but unfortunately, without support for large folio swap-ins, once these large folios are swapped out, they are lost because mTHP swap is a one-way process. The lack of mTHP swap-in functionality prevents mTHP from being used on devices like Android that heavily rely on swap.

This patch introduces mTHP swap-in support. It starts from sync devices such as zRAM. This is probably the simplest and most common use case, benefiting billions of Android phones and similar devices with minimal implementation cost. In this straightforward scenario, large folios are always exclusive, eliminating the need to handle complex rmap and swapcache issues.

It offers several benefits:
1. Enables bidirectional mTHP swapping, allowing retrieval of mTHP after swap-out and swap-in. Large folios in the buddy system are also preserved as much as possible, rather than being fragmented due to swap-in.

2. Eliminates fragmentation in swap slots and supports successful THP_SWPOUT.

w/o this patch (Refer to the data from Chris's and Kairui's latest swap allocator optimization while running ./thp_swap_allocator_test w/o "-a" option [1]):

./thp_swap_allocator_test
Iteration 1: swpout inc: 233, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 2: swpout inc: 131, swpout fallback inc: 101, Fallback percentage: 43.53% Iteration 3: swpout inc: 71, swpout fallback inc: 155, Fallback percentage: 68.58% Iteration 4: swpout inc: 55, swpout fallback inc: 168, Fallback percentage: 75.34% Iteration 5: swpout inc: 35, swpout fallback inc: 191, Fallback percentage: 84.51% Iteration 6: swpout inc: 25, swpout fallback inc: 199, Fallback percentage: 88.84% Iteration 7: swpout inc: 23, swpout fallback inc: 205, Fallback percentage: 89.91% Iteration 8: swpout inc: 9, swpout fallback inc: 219, Fallback percentage: 96.05% Iteration 9: swpout inc: 13, swpout fallback inc: 213, Fallback percentage: 94.25% Iteration 10: swpout inc: 12, swpout fallback inc: 216, Fallback percentage: 94.74% Iteration 11: swpout inc: 16, swpout fallback inc: 213, Fallback percentage: 93.01% Iteration 12: swpout inc: 10, swpout fallback inc: 210, Fallback percentage: 95.45% Iteration 13: swpout inc: 16, swpout fallback inc: 212, Fallback percentage: 92.98% Iteration 14: swpout inc: 12, swpout fallback inc: 212, Fallback percentage: 94.64% Iteration 15: swpout inc: 15, swpout fallback inc: 211, Fallback percentage: 93.36% Iteration 16: swpout inc: 15, swpout fallback inc: 200, Fallback percentage: 93.02% Iteration 17: swpout inc: 9, swpout fallback inc: 220, Fallback percentage: 96.07%

w/ this patch (always 0%):
Iteration 1: swpout inc: 948, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 2: swpout inc: 953, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 3: swpout inc: 950, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 4: swpout inc: 952, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 5: swpout inc: 950, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 6: swpout inc: 950, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 7: swpout inc: 947, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 8: swpout inc: 950, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 9: swpout inc: 950, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 10: swpout inc: 945, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 11: swpout inc: 947, swpout fallback inc: 0, Fallback percentage: 0.00%
...

3. With both mTHP swap-out and swap-in supported, we offer the option to enable zsmalloc compression/decompression with larger granularity[2]. The upcoming optimization in zsmalloc will significantly increase swap speed and improve compression efficiency. Tested by running 100 iterations of swapping 100MiB of anon memory, the swap speed improved dramatically:
time consumption of swapin(ms) time consumption of swapout(ms)
lz4 4k 45274 90540
lz4 64k 22942 55667
zstdn 4k 85035 186585
zstdn 64k 46558 118533

The compression ratio also improved, as evaluated with 1 GiB of data:
granularity orig_data_size compr_data_size
4KiB-zstd 1048576000 246876055
64KiB-zstd 1048576000 199763892

Without mTHP swap-in, the potential optimizations in zsmalloc cannot be realized.

4. Even mTHP swap-in itself can reduce swap-in page faults by a factor of nr_pages. Swapping in content filled with the same data 0x11, w/o and w/ the patch for five rounds (Since the content is the same, decompression will be very fast. This primarily assesses the impact of reduced page faults):

swp in bandwidth(bytes/ms) w/o w/

   round1                     624152          1127501
   round2                     631672          1127501
   round3                     620459          1139756
   round4                     606113          1139756
   round5                     624152          1152281
   avg                        621310          1137359      +83%

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/

---
 mm/memory.c | 211 ++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 188 insertions(+), 23 deletions(-)
mm/memory.c +160 -16
index 4cf4902db1ec..07029532469a 100644
@@ -3986,6 +3986,152 @@ static vm_fault_t handle_pte_marker(struct vm_fault *vmf)
        return VM_FAULT_SIGBUS;
 }
 
+/*
+ * check a range of PTEs are completely swap entries with
+ * contiguous swap offsets and the same SWAP_HAS_CACHE.
+ * ptep must be first one in the range
+ */
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+static bool can_swapin_thp(struct vm_fault *vmf, pte_t *ptep, int nr_pages)
+{
+       struct swap_info_struct *si;
+       unsigned long addr;
+       swp_entry_t entry;
+       pgoff_t offset;
+       char has_cache;
+       int idx, i;
+       pte_t pte;
+
+       addr = ALIGN_DOWN(vmf->address, nr_pages * PAGE_SIZE);
+       idx = (vmf->address - addr) / PAGE_SIZE;
+       pte = ptep_get(ptep);
+
+       if (!pte_same(pte, pte_move_swp_offset(vmf->orig_pte, -idx)))
+               return false;
+       entry = pte_to_swp_entry(pte);
+       offset = swp_offset(entry);
+       if (swap_pte_batch(ptep, nr_pages, pte) != nr_pages)
+               return false;
+
+       si = swp_swap_info(entry);
+       has_cache = si->swap_map[offset] & SWAP_HAS_CACHE;
+       for (i = 1; i < nr_pages; i++) {
+               /*
+                * while allocating a large folio and doing swap_read_folio for the
+                * SWP_SYNCHRONOUS_IO path, which is the case the being faulted pte
+                * doesn't have swapcache. We need to ensure all PTEs have no cache
+                * as well, otherwise, we might go to swap devices while the content
+                * is in swapcache
+                */
+               if ((si->swap_map[offset + i] & SWAP_HAS_CACHE) != has_cache)
+                       return false;
+       }
+
+       return true;
+}
+
+static inline unsigned long thp_swap_suitable_orders(pgoff_t swp_offset,
+               unsigned long addr, unsigned long orders)
+{
+       int order, nr;
+
+       order = highest_order(orders);
+
+       /*
+        * To swap-in a THP with nr pages, we require its first swap_offset
+        * is aligned with nr. This can filter out most invalid entries.
+        */
+       while (orders) {
+               nr = 1 << order;
+               if ((addr >> PAGE_SHIFT) % nr == swp_offset % nr)
+                       break;
+               order = next_order(&orders, order);
+       }
+
+       return orders;
+}
+#else
+static inline bool can_swapin_thp(struct vm_fault *vmf, pte_t *ptep, int nr_pages)
+{
+       return false;
+}
+#endif
+
+static struct folio *alloc_swap_folio(struct vm_fault *vmf)
+{
+       struct vm_area_struct *vma = vmf->vma;
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+       unsigned long orders;
+       struct folio *folio;
+       unsigned long addr;
+       swp_entry_t entry;
+       spinlock_t *ptl;
+       pte_t *pte;
+       gfp_t gfp;
+       int order;
+
+       /*
+        * If uffd is active for the vma we need per-page fault fidelity to
+        * maintain the uffd semantics.
+        */
+       if (unlikely(userfaultfd_armed(vma)))
+               goto fallback;
+
+       /*
+        * A large swapped out folio could be partially or fully in zswap. We
+        * lack handling for such cases, so fallback to swapping in order-0
+        * folio.
+        */
+       if (!zswap_never_enabled())
+               goto fallback;
+
+       entry = pte_to_swp_entry(vmf->orig_pte);
+       /*
+        * Get a list of all the (large) orders below PMD_ORDER that are enabled
+        * and suitable for swapping THP.
+        */
+       orders = thp_vma_allowable_orders(vma, vma->vm_flags,
+                       TVA_IN_PF | TVA_ENFORCE_SYSFS, BIT(PMD_ORDER) - 1);
+       orders = thp_vma_suitable_orders(vma, vmf->address, orders);
+       orders = thp_swap_suitable_orders(swp_offset(entry), vmf->address, orders);
+
+       if (!orders)
+               goto fallback;
+
+       pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address & PMD_MASK, &ptl);
+       if (unlikely(!pte))
+               goto fallback;
+
+       /*
+        * For do_swap_page, find the highest order where the aligned range is
+        * completely swap entries with contiguous swap offsets.
+        */
+       order = highest_order(orders);
+       while (orders) {
+               addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
+               if (can_swapin_thp(vmf, pte + pte_index(addr), 1 << order))
+                       break;
+               order = next_order(&orders, order);
+       }
+
+       pte_unmap_unlock(pte, ptl);
+
+       /* Try allocating the highest of the remaining orders. */
+       gfp = vma_thp_gfp_mask(vma);
+       while (orders) {
+               addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
+               folio = vma_alloc_folio(gfp, order, vma, addr, true);
+               if (folio)
+                       return folio;
+               order = next_order(&orders, order);
+       }
+
+fallback:
+#endif
+       return vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, vmf->address, false);
+}
+
+
 /*
  * We enter with non-exclusive mmap_lock (to exclude vma changes,
  * but allow concurrent faults), and pte mapped but not yet locked.
@@ -4074,35 +4220,37 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
        if (!folio) {
                if (data_race(si->flags & SWP_SYNCHRONOUS_IO) &&
                    __swap_count(entry) == 1) {
-                       /*
-                        * Prevent parallel swapin from proceeding with
-                        * the cache flag. Otherwise, another thread may
-                        * finish swapin first, free the entry, and swapout
-                        * reusing the same entry. It's undetectable as
-                        * pte_same() returns true due to entry reuse.
-                        */
-                       if (swapcache_prepare(entry, 1)) {
-                               /* Relax a bit to prevent rapid repeated page faults */
-                               schedule_timeout_uninterruptible(1);
-                               goto out;
-                       }
-                       need_clear_cache = true;
-
                        /* skip swapcache */
-                       folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0,
-                                               vma, vmf->address, false);
+                       folio = alloc_swap_folio(vmf);
                        page = &folio->page;
                        if (folio) {
                                __folio_set_locked(folio);
                                __folio_set_swapbacked(folio);
 
+                               nr_pages = folio_nr_pages(folio);
+                               if (folio_test_large(folio))
+                                       entry.val = ALIGN_DOWN(entry.val, nr_pages);
+                               /*
+                                * Prevent parallel swapin from proceeding with
+                                * the cache flag. Otherwise, another thread may
+                                * finish swapin first, free the entry, and swapout
+                                * reusing the same entry. It's undetectable as
+                                * pte_same() returns true due to entry reuse.
+                                */
+                               if (swapcache_prepare(entry, nr_pages)) {
+                                       /* Relax a bit to prevent rapid repeated page faults */
+                                       schedule_timeout_uninterruptible(1);

mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4252

Quoted 10 lines
mm/memory.c +4 -0
+                                       goto out_page;
+                               }
+                               need_clear_cache = true;
+
                                if (mem_cgroup_swapin_charge_folio(folio,
                                                        vma->vm_mm, GFP_KERNEL,
                                                        entry)) {
                                        ret = VM_FAULT_OOM;
                                        goto out_page;
                                }

After your patch, with build kernel test, I'm seeing kernel log spamming like this:

[  101.048594] pagefault_out_of_memory: 95 callbacks suppressed
[  101.048599] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[  101.059416] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[  101.118575] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[  101.125585] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[  101.182501] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[  101.215351] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[  101.272822] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[  101.403195] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF

............

And heavy performance loss with workloads limited by memcg, mTHP enabled.

After some debugging, the problematic part is the mem_cgroup_swapin_charge_folio call above.
When under pressure, cgroup charge fails easily for mTHP. One 64k swapin will require a much more aggressive reclaim to success.

If I change MAX_RECLAIM_RETRIES from 16 to 512, the spamming log is gone and mTHP swapin should have a much higher swapin success rate. But this might not be the right way.

For this particular issue, maybe you can change the charge order, try charging first, if successful, use mTHP. if failed, fallback to 4k?

Quoted 67 lines
mm/memory.c +24 -8
-                               mem_cgroup_swapin_uncharge_swap(entry, 1);
+                               mem_cgroup_swapin_uncharge_swap(entry, nr_pages);
 
                                shadow = get_shadow_from_swap_cache(entry);
                                if (shadow)
@@ -4209,6 +4357,22 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
                goto out_nomap;
        }
 
+       /* allocated large folios for SWP_SYNCHRONOUS_IO */
+       if (folio_test_large(folio) && !folio_test_swapcache(folio)) {
+               unsigned long nr = folio_nr_pages(folio);
+               unsigned long folio_start = ALIGN_DOWN(vmf->address, nr * PAGE_SIZE);
+               unsigned long idx = (vmf->address - folio_start) / PAGE_SIZE;
+               pte_t *folio_ptep = vmf->pte - idx;
+
+               if (!can_swapin_thp(vmf, folio_ptep, nr))
+                       goto out_nomap;
+
+               page_idx = idx;
+               address = folio_start;
+               ptep = folio_ptep;
+               goto check_folio;
+       }
+
        nr_pages = 1;
        page_idx = 0;
        address = vmf->address;
@@ -4340,11 +4504,12 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
                folio_add_lru_vma(folio, vma);
        } else if (!folio_test_anon(folio)) {
                /*
-                * We currently only expect small !anon folios, which are either
-                * fully exclusive or fully shared. If we ever get large folios
-                * here, we have to be careful.
+                * We currently only expect small !anon folios which are either
+                * fully exclusive or fully shared, or new allocated large folios
+                * which are fully exclusive. If we ever get large folios within
+                * swapcache here, we have to be careful.
                 */
-               VM_WARN_ON_ONCE(folio_test_large(folio));
+               VM_WARN_ON_ONCE(folio_test_large(folio) && folio_test_swapcache(folio));
                VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
                folio_add_new_anon_rmap(folio, vma, address, rmap_flags);
        } else {
@@ -4387,7 +4552,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 out:
        /* Clear the swap cache pin for direct swapin after PTL unlock */
        if (need_clear_cache)
-               swapcache_clear(si, entry, 1);
+               swapcache_clear(si, entry, nr_pages);
        if (si)
                put_swap_device(si);
        return ret;
@@ -4403,7 +4568,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
                folio_put(swapcache);
        }
        if (need_clear_cache)
-               swapcache_clear(si, entry, 1);
+               swapcache_clear(si, entry, nr_pages);
        if (si)
                put_swap_device(si);
        return ret;
--

2.34.1

#53 lore raw

Re: [PATCH v6 2/2] mm: support large folios swap-in for zRAM-like devices

Quoted 7 lines · Kairui Song
Quoted 2 lines · Barry Song

From: Chuanhua Han <[email protected]>

Hi Chuanhua,

Quoted 1 line

...

Quoted 150 lines
Quoted 119 lines
mm/memory.c +90 -16
+
+static struct folio *alloc_swap_folio(struct vm_fault *vmf)
+{
+       struct vm_area_struct *vma = vmf->vma;
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+       unsigned long orders;
+       struct folio *folio;
+       unsigned long addr;
+       swp_entry_t entry;
+       spinlock_t *ptl;
+       pte_t *pte;
+       gfp_t gfp;
+       int order;
+
+       /*
+        * If uffd is active for the vma we need per-page fault fidelity to
+        * maintain the uffd semantics.
+        */
+       if (unlikely(userfaultfd_armed(vma)))
+               goto fallback;
+
+       /*
+        * A large swapped out folio could be partially or fully in zswap. We
+        * lack handling for such cases, so fallback to swapping in order-0
+        * folio.
+        */
+       if (!zswap_never_enabled())
+               goto fallback;
+
+       entry = pte_to_swp_entry(vmf->orig_pte);
+       /*
+        * Get a list of all the (large) orders below PMD_ORDER that are enabled
+        * and suitable for swapping THP.
+        */
+       orders = thp_vma_allowable_orders(vma, vma->vm_flags,
+                       TVA_IN_PF | TVA_ENFORCE_SYSFS, BIT(PMD_ORDER) - 1);
+       orders = thp_vma_suitable_orders(vma, vmf->address, orders);
+       orders = thp_swap_suitable_orders(swp_offset(entry), vmf->address, orders);
+
+       if (!orders)
+               goto fallback;
+
+       pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address & PMD_MASK, &ptl);
+       if (unlikely(!pte))
+               goto fallback;
+
+       /*
+        * For do_swap_page, find the highest order where the aligned range is
+        * completely swap entries with contiguous swap offsets.
+        */
+       order = highest_order(orders);
+       while (orders) {
+               addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
+               if (can_swapin_thp(vmf, pte + pte_index(addr), 1 << order))
+                       break;
+               order = next_order(&orders, order);
+       }
+
+       pte_unmap_unlock(pte, ptl);
+
+       /* Try allocating the highest of the remaining orders. */
+       gfp = vma_thp_gfp_mask(vma);
+       while (orders) {
+               addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
+               folio = vma_alloc_folio(gfp, order, vma, addr, true);
+               if (folio)
+                       return folio;
+               order = next_order(&orders, order);
+       }
+
+fallback:
+#endif
+       return vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, vmf->address, false);
+}
+
+
  /*
   * We enter with non-exclusive mmap_lock (to exclude vma changes,
   * but allow concurrent faults), and pte mapped but not yet locked.
@@ -4074,35 +4220,37 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
         if (!folio) {
                 if (data_race(si->flags & SWP_SYNCHRONOUS_IO) &&
                     __swap_count(entry) == 1) {
-                       /*
-                        * Prevent parallel swapin from proceeding with
-                        * the cache flag. Otherwise, another thread may
-                        * finish swapin first, free the entry, and swapout
-                        * reusing the same entry. It's undetectable as
-                        * pte_same() returns true due to entry reuse.
-                        */
-                       if (swapcache_prepare(entry, 1)) {
-                               /* Relax a bit to prevent rapid repeated page faults */
-                               schedule_timeout_uninterruptible(1);
-                               goto out;
-                       }
-                       need_clear_cache = true;
-
                         /* skip swapcache */
-                       folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0,
-                                               vma, vmf->address, false);
+                       folio = alloc_swap_folio(vmf);
                         page = &folio->page;
                         if (folio) {
                                 __folio_set_locked(folio);
                                 __folio_set_swapbacked(folio);
 
+                               nr_pages = folio_nr_pages(folio);
+                               if (folio_test_large(folio))
+                                       entry.val = ALIGN_DOWN(entry.val, nr_pages);
+                               /*
+                                * Prevent parallel swapin from proceeding with
+                                * the cache flag. Otherwise, another thread may
+                                * finish swapin first, free the entry, and swapout
+                                * reusing the same entry. It's undetectable as
+                                * pte_same() returns true due to entry reuse.
+                                */
+                               if (swapcache_prepare(entry, nr_pages)) {
+                                       /* Relax a bit to prevent rapid repeated page faults */
+                                       schedule_timeout_uninterruptible(1);

mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4252

Quoted 10 lines
mm/memory.c +4 -0
+                                       goto out_page;
+                               }
+                               need_clear_cache = true;
+
                                 if (mem_cgroup_swapin_charge_folio(folio,
                                                         vma->vm_mm, GFP_KERNEL,
                                                         entry)) {
                                         ret = VM_FAULT_OOM;
                                         goto out_page;
                                 }

After your patch, with build kernel test, I'm seeing kernel log spamming like this:

[  101.048594] pagefault_out_of_memory: 95 callbacks suppressed
[  101.048599] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[  101.059416] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[  101.118575] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[  101.125585] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[  101.182501] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[  101.215351] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[  101.272822] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[  101.403195] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF

............

And heavy performance loss with workloads limited by memcg, mTHP enabled.

After some debugging, the problematic part is the mem_cgroup_swapin_charge_folio call above.
When under pressure, cgroup charge fails easily for mTHP. One 64k swapin will require a much more aggressive reclaim to success.

Quoted 6 lines

If I change MAX_RECLAIM_RETRIES from 16 to 512, the spamming log is gone and mTHP swapin should have a much higher swapin success rate. But this might not be the right way.

For this particular issue, maybe you can change the charge order, try charging first, if successful, use mTHP. if failed, fallback to 4k?

This is what we did in alloc_anon_folio(), see 085ff35e7636 ("mm: memory: move mem_cgroup_charge() into alloc_anon_folio()"),
1) fallback earlier
2) using same GFP flags for allocation and charge

but it seems that there is a little complicated for swapin charge

Quoted 69 lines
Quoted 68 lines
mm/memory.c +24 -8
-                               mem_cgroup_swapin_uncharge_swap(entry, 1);
+                               mem_cgroup_swapin_uncharge_swap(entry, nr_pages);
 
                                 shadow = get_shadow_from_swap_cache(entry);
                                 if (shadow)
@@ -4209,6 +4357,22 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
                 goto out_nomap;
         }
 
+       /* allocated large folios for SWP_SYNCHRONOUS_IO */
+       if (folio_test_large(folio) && !folio_test_swapcache(folio)) {
+               unsigned long nr = folio_nr_pages(folio);
+               unsigned long folio_start = ALIGN_DOWN(vmf->address, nr * PAGE_SIZE);
+               unsigned long idx = (vmf->address - folio_start) / PAGE_SIZE;
+               pte_t *folio_ptep = vmf->pte - idx;
+
+               if (!can_swapin_thp(vmf, folio_ptep, nr))
+                       goto out_nomap;
+
+               page_idx = idx;
+               address = folio_start;
+               ptep = folio_ptep;
+               goto check_folio;
+       }
+
         nr_pages = 1;
         page_idx = 0;
         address = vmf->address;
@@ -4340,11 +4504,12 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
                 folio_add_lru_vma(folio, vma);
         } else if (!folio_test_anon(folio)) {
                 /*
-                * We currently only expect small !anon folios, which are either
-                * fully exclusive or fully shared. If we ever get large folios
-                * here, we have to be careful.
+                * We currently only expect small !anon folios which are either
+                * fully exclusive or fully shared, or new allocated large folios
+                * which are fully exclusive. If we ever get large folios within
+                * swapcache here, we have to be careful.
                  */
-               VM_WARN_ON_ONCE(folio_test_large(folio));
+               VM_WARN_ON_ONCE(folio_test_large(folio) && folio_test_swapcache(folio));
                 VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
                 folio_add_new_anon_rmap(folio, vma, address, rmap_flags);
         } else {
@@ -4387,7 +4552,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
  out:
         /* Clear the swap cache pin for direct swapin after PTL unlock */
         if (need_clear_cache)
-               swapcache_clear(si, entry, 1);
+               swapcache_clear(si, entry, nr_pages);
         if (si)
                 put_swap_device(si);
         return ret;
@@ -4403,7 +4568,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
                 folio_put(swapcache);
         }
         if (need_clear_cache)
-               swapcache_clear(si, entry, 1);
+               swapcache_clear(si, entry, nr_pages);
         if (si)
                 put_swap_device(si);
         return ret;
--

2.34.1

author
#54 lore raw

Re: [PATCH v6 2/2] mm: support large folios swap-in for zRAM-like devices

Quoted 170 lines · Kefeng Wang
Quoted 7 lines · Kairui Song
Quoted 2 lines · Barry Song

From: Chuanhua Han <[email protected]>

Hi Chuanhua,

Quoted 1 line

...

Quoted 150 lines
Quoted 121 lines
(quoted diff) +76 -0
+
+static struct folio *alloc_swap_folio(struct vm_fault *vmf)
+{
+       struct vm_area_struct *vma = vmf->vma;
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+       unsigned long orders;
+       struct folio *folio;
+       unsigned long addr;
+       swp_entry_t entry;
+       spinlock_t *ptl;
+       pte_t *pte;
+       gfp_t gfp;
+       int order;
+
+       /*
+        * If uffd is active for the vma we need per-page fault fidelity to
+        * maintain the uffd semantics.
+        */
+       if (unlikely(userfaultfd_armed(vma)))
+               goto fallback;
+
+       /*
+        * A large swapped out folio could be partially or fully in zswap. We
+        * lack handling for such cases, so fallback to swapping in order-0
+        * folio.
+        */
+       if (!zswap_never_enabled())
+               goto fallback;
+
+       entry = pte_to_swp_entry(vmf->orig_pte);
+       /*
+        * Get a list of all the (large) orders below PMD_ORDER that are enabled
+        * and suitable for swapping THP.
+        */
+       orders = thp_vma_allowable_orders(vma, vma->vm_flags,
+                       TVA_IN_PF | TVA_ENFORCE_SYSFS, BIT(PMD_ORDER) - 1);
+       orders = thp_vma_suitable_orders(vma, vmf->address, orders);
+       orders = thp_swap_suitable_orders(swp_offset(entry), vmf->address, orders);
+
+       if (!orders)
+               goto fallback;
+
+       pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address & PMD_MASK, &ptl);
+       if (unlikely(!pte))
+               goto fallback;
+
+       /*
+        * For do_swap_page, find the highest order where the aligned range is
+        * completely swap entries with contiguous swap offsets.
+        */
+       order = highest_order(orders);
+       while (orders) {
+               addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
+               if (can_swapin_thp(vmf, pte + pte_index(addr), 1 << order))
+                       break;
+               order = next_order(&orders, order);
+       }
+
+       pte_unmap_unlock(pte, ptl);
+
+       /* Try allocating the highest of the remaining orders. */
+       gfp = vma_thp_gfp_mask(vma);
+       while (orders) {
+               addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
+               folio = vma_alloc_folio(gfp, order, vma, addr, true);
+               if (folio)
+                       return folio;
+               order = next_order(&orders, order);
+       }
+
+fallback:
+#endif
+       return vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, vmf->address, false);
+}
+
+

  /*
   * We enter with non-exclusive mmap_lock (to exclude vma changes,
   * but allow concurrent faults), and pte mapped but not yet locked.

(quoted diff) +0 -0
@@ -4074,35 +4220,37 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
          if (!folio) {

                 if (data_race(si->flags & SWP_SYNCHRONOUS_IO) && __swap_count(entry) == 1) {
-                       /*
-                        * Prevent parallel swapin from proceeding with
-                        * the cache flag. Otherwise, another thread may
-                        * finish swapin first, free the entry, and swapout
-                        * reusing the same entry. It's undetectable as
-                        * pte_same() returns true due to entry reuse.
-                        */
-                       if (swapcache_prepare(entry, 1)) {
-                               /* Relax a bit to prevent rapid repeated page faults */
-                               schedule_timeout_uninterruptible(1);
-                               goto out;
-                       }
-                       need_clear_cache = true;
-
                         /* skip swapcache */
-                       folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0,
-                                               vma, vmf->address, false);
+                       folio = alloc_swap_folio(vmf);
                         page = &folio->page;
                         if (folio) {
                                 __folio_set_locked(folio); __folio_set_swapbacked(folio);

+                               nr_pages = folio_nr_pages(folio);
+                               if (folio_test_large(folio))
+                                       entry.val = ALIGN_DOWN(entry.val, nr_pages);
+                               /*
+                                * Prevent parallel swapin from proceeding with
+                                * the cache flag. Otherwise, another thread may
+                                * finish swapin first, free the entry, and swapout
+                                * reusing the same entry. It's undetectable as
+                                * pte_same() returns true due to entry reuse.
+                                */
+                               if (swapcache_prepare(entry, nr_pages)) {
+                                       /* Relax a bit to prevent rapid repeated page faults */
+                                       schedule_timeout_uninterruptible(1);
+                                       goto out_page;
+                               }

Quoted 8 lines

+                               need_clear_cache = true;
+
                                 if (mem_cgroup_swapin_charge_folio(folio, vma->vm_mm, GFP_KERNEL, entry)) { ret = VM_FAULT_OOM;
                                         goto out_page;
                                 }

After your patch, with build kernel test, I'm seeing kernel log spamming like this:
[  101.048594] pagefault_out_of_memory: 95 callbacks suppressed [  101.048599] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF [  101.059416] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF [  101.118575] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF [  101.125585] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF [  101.182501] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF [  101.215351] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF [  101.272822] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF [  101.403195] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
............

And heavy performance loss with workloads limited by memcg, mTHP enabled.

After some debugging, the problematic part is the mem_cgroup_swapin_charge_folio call above.
When under pressure, cgroup charge fails easily for mTHP. One 64k swapin will require a much more aggressive reclaim to success.

Quoted 6 lines

If I change MAX_RECLAIM_RETRIES from 16 to 512, the spamming log is gone and mTHP swapin should have a much higher swapin success rate. But this might not be the right way.

For this particular issue, maybe you can change the charge order, try charging first, if successful, use mTHP. if failed, fallback to 4k?

Quoted 6 lines

This is what we did in alloc_anon_folio(), see 085ff35e7636 ("mm: memory: move mem_cgroup_charge() into alloc_anon_folio()"),
1) fallback earlier
2) using same GFP flags for allocation and charge

but it seems that there is a little complicated for swapin charge

Kefeng, thanks! I guess we can continue using the same approach and it's not too complicated.

Kairui, sorry for the trouble and thanks for the report! could you check if the solution below resolves the issue? On phones, we don't encounter the scenarios you’re facing.

From 2daaf91077705a8fa26a3a428117f158f05375b0 Mon Sep 17 00:00:00 2001
From: Barry Song <[email protected]>
Date: Fri, 16 Aug 2024 10:51:48 +1200
Subject: [PATCH] mm: fallback to next_order if charing mTHP fails

When memcg approaches its limit, charging mTHP becomes difficult. At this point, when the charge fails, we fallback to the next order to avoid repeatedly retrying larger orders.

---
 mm/memory.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
mm/memory.c +7 -3
index 0ed3603aaf31..6cba28ef91e7 100644
@@ -4121,8 +4121,12 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)
 	while (orders) {
 		addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
 		folio = vma_alloc_folio(gfp, order, vma, addr, true);
-		if (folio)
-			return folio;
+		if (folio) {
+			if (!mem_cgroup_swapin_charge_folio(folio,
+					vma->vm_mm, gfp, entry))
+				return folio;
+			folio_put(folio);
+		}
 		order = next_order(&orders, order);
 	}
 
@@ -4244,7 +4248,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 				}
 				need_clear_cache = true;
 
-				if (mem_cgroup_swapin_charge_folio(folio,
+				if (nr_pages == 1 && mem_cgroup_swapin_charge_folio(folio,
 							vma->vm_mm, GFP_KERNEL,
 							entry)) {
 					ret = VM_FAULT_OOM;
Signature · 5 lines
2.34.1


Thanks
Barry
#55 lore raw

Re: [PATCH v6 2/2] mm: support large folios swap-in for zRAM-like devices

Quoted 230 lines · Barry Song
Quoted 170 lines · Kefeng Wang
Quoted 7 lines · Kairui Song
Quoted 2 lines · Barry Song

From: Chuanhua Han <[email protected]>

Hi Chuanhua,

Quoted 1 line

...

Quoted 150 lines
Quoted 119 lines
(quoted diff) +90 -16
+
+static struct folio *alloc_swap_folio(struct vm_fault *vmf)
+{
+       struct vm_area_struct *vma = vmf->vma;
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+       unsigned long orders;
+       struct folio *folio;
+       unsigned long addr;
+       swp_entry_t entry;
+       spinlock_t *ptl;
+       pte_t *pte;
+       gfp_t gfp;
+       int order;
+
+       /*
+        * If uffd is active for the vma we need per-page fault fidelity to
+        * maintain the uffd semantics.
+        */
+       if (unlikely(userfaultfd_armed(vma)))
+               goto fallback;
+
+       /*
+        * A large swapped out folio could be partially or fully in zswap. We
+        * lack handling for such cases, so fallback to swapping in order-0
+        * folio.
+        */
+       if (!zswap_never_enabled())
+               goto fallback;
+
+       entry = pte_to_swp_entry(vmf->orig_pte);
+       /*
+        * Get a list of all the (large) orders below PMD_ORDER that are enabled
+        * and suitable for swapping THP.
+        */
+       orders = thp_vma_allowable_orders(vma, vma->vm_flags,
+                       TVA_IN_PF | TVA_ENFORCE_SYSFS, BIT(PMD_ORDER) - 1);
+       orders = thp_vma_suitable_orders(vma, vmf->address, orders);
+       orders = thp_swap_suitable_orders(swp_offset(entry), vmf->address, orders);
+
+       if (!orders)
+               goto fallback;
+
+       pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address & PMD_MASK, &ptl);
+       if (unlikely(!pte))
+               goto fallback;
+
+       /*
+        * For do_swap_page, find the highest order where the aligned range is
+        * completely swap entries with contiguous swap offsets.
+        */
+       order = highest_order(orders);
+       while (orders) {
+               addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
+               if (can_swapin_thp(vmf, pte + pte_index(addr), 1 << order))
+                       break;
+               order = next_order(&orders, order);
+       }
+
+       pte_unmap_unlock(pte, ptl);
+
+       /* Try allocating the highest of the remaining orders. */
+       gfp = vma_thp_gfp_mask(vma);
+       while (orders) {
+               addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
+               folio = vma_alloc_folio(gfp, order, vma, addr, true);
+               if (folio)
+                       return folio;
+               order = next_order(&orders, order);
+       }
+
+fallback:
+#endif
+       return vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, vmf->address, false);
+}
+
+
  /*
   * We enter with non-exclusive mmap_lock (to exclude vma changes,
   * but allow concurrent faults), and pte mapped but not yet locked.
@@ -4074,35 +4220,37 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
         if (!folio) {
                 if (data_race(si->flags & SWP_SYNCHRONOUS_IO) &&
                     __swap_count(entry) == 1) {
-                       /*
-                        * Prevent parallel swapin from proceeding with
-                        * the cache flag. Otherwise, another thread may
-                        * finish swapin first, free the entry, and swapout
-                        * reusing the same entry. It's undetectable as
-                        * pte_same() returns true due to entry reuse.
-                        */
-                       if (swapcache_prepare(entry, 1)) {
-                               /* Relax a bit to prevent rapid repeated page faults */
-                               schedule_timeout_uninterruptible(1);
-                               goto out;
-                       }
-                       need_clear_cache = true;
-
                         /* skip swapcache */
-                       folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0,
-                                               vma, vmf->address, false);
+                       folio = alloc_swap_folio(vmf);
                         page = &folio->page;
                         if (folio) {
                                 __folio_set_locked(folio);
                                 __folio_set_swapbacked(folio);
 
+                               nr_pages = folio_nr_pages(folio);
+                               if (folio_test_large(folio))
+                                       entry.val = ALIGN_DOWN(entry.val, nr_pages);
+                               /*
+                                * Prevent parallel swapin from proceeding with
+                                * the cache flag. Otherwise, another thread may
+                                * finish swapin first, free the entry, and swapout
+                                * reusing the same entry. It's undetectable as
+                                * pte_same() returns true due to entry reuse.
+                                */
+                               if (swapcache_prepare(entry, nr_pages)) {
+                                       /* Relax a bit to prevent rapid repeated page faults */
+                                       schedule_timeout_uninterruptible(1);

Review vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4252

Quoted 10 lines
(quoted diff) +4 -0
+                                       goto out_page;
+                               }
+                               need_clear_cache = true;
+
                                 if (mem_cgroup_swapin_charge_folio(folio,
                                                         vma->vm_mm, GFP_KERNEL,
                                                         entry)) {
                                         ret = VM_FAULT_OOM;
                                         goto out_page;
                                 }

After your patch, with build kernel test, I'm seeing kernel log spamming like this:

[  101.048594] pagefault_out_of_memory: 95 callbacks suppressed
[  101.048599] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[  101.059416] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[  101.118575] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[  101.125585] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[  101.182501] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[  101.215351] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[  101.272822] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[  101.403195] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF

............

And heavy performance loss with workloads limited by memcg, mTHP enabled.

After some debugging, the problematic part is the mem_cgroup_swapin_charge_folio call above.
When under pressure, cgroup charge fails easily for mTHP. One 64k swapin will require a much more aggressive reclaim to success.

Quoted 6 lines

If I change MAX_RECLAIM_RETRIES from 16 to 512, the spamming log is gone and mTHP swapin should have a much higher swapin success rate. But this might not be the right way.

For this particular issue, maybe you can change the charge order, try charging first, if successful, use mTHP. if failed, fallback to 4k?

Quoted 6 lines

This is what we did in alloc_anon_folio(), see 085ff35e7636 ("mm: memory: move mem_cgroup_charge() into alloc_anon_folio()"),
1) fallback earlier
2) using same GFP flags for allocation and charge

but it seems that there is a little complicated for swapin charge

Kefeng, thanks! I guess we can continue using the same approach and it's not too complicated.

Kairui, sorry for the trouble and thanks for the report! could you check if the solution below resolves the issue? On phones, we don't encounter the scenarios you’re facing.

From 2daaf91077705a8fa26a3a428117f158f05375b0 Mon Sep 17 00:00:00 2001
From: Barry Song <[email protected]>
Date: Fri, 16 Aug 2024 10:51:48 +1200
Subject: [PATCH] mm: fallback to next_order if charing mTHP fails

When memcg approaches its limit, charging mTHP becomes difficult. At this point, when the charge fails, we fallback to the next order to avoid repeatedly retrying larger orders.

---
 mm/memory.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
mm/memory.c +7 -4
index 0ed3603aaf31..6cba28ef91e7 100644
@@ -4121,8 +4121,12 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)
        while (orders) {
                addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
                folio = vma_alloc_folio(gfp, order, vma, addr, true);
-               if (folio)
-                       return folio;
+               if (folio) {
+                       if (!mem_cgroup_swapin_charge_folio(folio,
+                                       vma->vm_mm, gfp, entry))
+                               return folio;
+                       folio_put(folio);
+               }
                order = next_order(&orders, order);
        }
 
@@ -4244,7 +4248,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
                                }
                                need_clear_cache = true;
 
-                               if (mem_cgroup_swapin_charge_folio(folio,
+                               if (nr_pages == 1 && mem_cgroup_swapin_charge_folio(folio,
                                                        vma->vm_mm, GFP_KERNEL,
                                                        entry)) {
                                        ret = VM_FAULT_OOM;
--
Quoted 2 lines

2.34.1

Hi Barry

After the fix the spamming log is gone, thanks for the fix.

Quoted 4 lines

Thanks
Barry

maintainer
#56 lore raw

Re: [PATCH v6 2/2] mm: support large folios swap-in for zRAM-like devices

Quoted 8 lines · On Sat, 17 Aug 2024 00:50:00 +0800 Kairui Song <[email protected]> wrote:
Quoted 3 lines

--
2.34.1

Hi Barry

After the fix the spamming log is gone, thanks for the fix.

Thanks, I'll drop the v6 series.

#57 lore raw

Re: [PATCH v6 2/2] mm: support large folios swap-in for zRAM-like devices

Quoted 7 lines · Andrew Morton <[email protected]> 于2024年8月17日周六 04:35写道:
Quoted 4 lines · On Sat, 17 Aug 2024 00:50:00 +0800 Kairui Song <[email protected]> wrote:
Quoted 3 lines

--
2.34.1

Quoted 6 lines
Quoted 4 lines

Hi Barry

After the fix the spamming log is gone, thanks for the fix.

Thanks, I'll drop the v6 series.

Hi, Andrew

Can you please queue v7 for testing:
https://lore.kernel.org/linux-mm/[email protected]/

V7 has addressed all comments regarding the changelog, the subject and order-0 charge from Christoph, Kairui and Willy.

Quoted 1 line
Signature · 2 lines
Thanks,
Chuanhua
#58 lore raw

Re: [PATCH v6 2/2] mm: support large folios swap-in for zRAM-like devices

In reply to Barry Song
Quoted 3 lines · Barry Song

When memcg approaches its limit, charging mTHP becomes difficult. At this point, when the charge fails, we fallback to the next order to avoid repeatedly retrying larger orders.

Why do you always find the ugliest possible solution to a problem?

mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4254

Quoted 9 lines
mm/memory.c +1 -1
@@ -4244,7 +4248,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 				}
 				need_clear_cache = true;
 
-				if (mem_cgroup_swapin_charge_folio(folio,
+				if (nr_pages == 1 && mem_cgroup_swapin_charge_folio(folio,
 							vma->vm_mm, GFP_KERNEL,
 							entry)) {
 					ret = VM_FAULT_OOM;

Just make alloc_swap_folio() always charge the folio, even for order-0.

And you'll have to uncharge it in the swapcache_prepare() failure case.

author
#59 lore raw

Re: [PATCH v6 2/2] mm: support large folios swap-in for zRAM-like devices

Quoted 8 lines · Matthew Wilcox
Quoted 3 lines · Barry Song

When memcg approaches its limit, charging mTHP becomes difficult. At this point, when the charge fails, we fallback to the next order to avoid repeatedly retrying larger orders.

Why do you always find the ugliest possible solution to a problem?

had definitely thought about charging order-0 as well in alloc_swap_folio() when sending this
quick fix mainly for quick verification it can fix the problem. v7 will definitely charge order-0
in alloc_swap_folio().

Quoted 12 lines

mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4254

Quoted 9 lines
mm/memory.c +1 -1
@@ -4244,7 +4248,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
                              }
                              need_clear_cache = true;
 
-                             if (mem_cgroup_swapin_charge_folio(folio,
+                             if (nr_pages == 1 && mem_cgroup_swapin_charge_folio(folio,
                                                      vma->vm_mm, GFP_KERNEL,
                                                      entry)) {
                                      ret = VM_FAULT_OOM;

Just make alloc_swap_folio() always charge the folio, even for order-0.

Quoted 1 line

And you'll have to uncharge it in the swapcache_prepare() failure case.

I suppose this is done by folio_put() automatically.

Thanks
Barry