Skip to contentSkip to conversation

mm/vmscan: avoid split PMD-mapped THP during shrink_folio_list()

PATCH Superseded Lance Yang 3 participants 8 messages Apr 17, 2024 – Apr 21, 2024

Subsystems (5)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Lance Yang

author
#0 lore raw

PATCH 1/1 mm/vmscan: avoid split PMD-mapped THP during shrink_folio_list()

When the user no longer requires the pages, they would use madvise(madv_free) to mark the pages as lazy free. IMO, they would not typically rewrite to the given range.

At present, a PMD-mapped THP marked as lazyfree during shrink_folio_list() is unconditionally split, which may be unnecessary. If the THP is exclusively mapped and clean, and the PMD associated with it is also clean, then we can attempt to remove the PMD mapping from it. This change will improve the efficiency of memory reclamation in this case.

On an Intel i5 CPU, reclaiming 1GiB of PMD-mapped THPs using mem_cgroup_force_empty() results in the following runtimes in seconds (shorter is better):

--------------------------------------------
| Old | New | Change |
--------------------------------------------
| 0.683426 | 0.049197 | -92.80% |
--------------------------------------------

---
 include/linux/huge_mm.h |  1 +
 include/linux/rmap.h    |  1 +
 mm/huge_memory.c        |  2 +-
 mm/rmap.c               | 81 +++++++++++++++++++++++++++++++++++++++++
 mm/vmscan.c             |  7 ++++
 5 files changed, 91 insertions(+), 1 deletion(-)
include/linux/huge_mm.h +1 -0
index 7cd07b83a3d0..02a71c05f68a 100644
@@ -36,6 +36,7 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,
 int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 		    pmd_t *pmd, unsigned long addr, pgprot_t newprot,
 		    unsigned long cp_flags);
+inline void zap_deposited_table(struct mm_struct *mm, pmd_t *pmd);
 
 vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, pfn_t pfn, bool write);
 vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, pfn_t pfn, bool write);
include/linux/rmap.h +1 -0
index 0f906dc6d280..8c2f45713351 100644
@@ -100,6 +100,7 @@ enum ttu_flags {
 					 * do a final flush if necessary */
 	TTU_RMAP_LOCKED		= 0x80,	/* do not grab rmap lock:
 					 * caller holds it */
+	TTU_LAZYFREE_THP	= 0x100, /* avoid split PMD-mapped THP */
 };
 
 #ifdef CONFIG_MMU
mm/huge_memory.c +1 -1
index 58f2c4745d80..309fba9624c2 100644
@@ -1801,7 +1801,7 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 	return ret;
 }
 
-static inline void zap_deposited_table(struct mm_struct *mm, pmd_t *pmd)
+inline void zap_deposited_table(struct mm_struct *mm, pmd_t *pmd)
 {
 	pgtable_t pgtable;
 
mm/rmap.c +81 -0

99 lines not on this page · raw message

mm/vmscan.c +7 -0
index 49bd94423961..2358b1cff8bf 100644
@@ -1277,6 +1277,13 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 
 			if (folio_test_pmd_mappable(folio))
 				flags |= TTU_SPLIT_HUGE_PMD;
+
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+			if (folio_test_anon(folio) && !was_swapbacked &&
+			    flags & TTU_SPLIT_HUGE_PMD)
+				flags |= TTU_LAZYFREE_THP;
+#endif
+
 			/*
 			 * Without TTU_SYNC, try_to_unmap will only begin to
 			 * hold PTL from the first present PTE within a large
Signature · 1 line
2.33.1
#1 lore raw
Quoted 98 lines · Lance Yang

When the user no longer requires the pages, they would use madvise(madv_free) to mark the pages as lazy free. IMO, they would not typically rewrite to the given range.

At present, a PMD-mapped THP marked as lazyfree during shrink_folio_list() is unconditionally split, which may be unnecessary. If the THP is exclusively mapped and clean, and the PMD associated with it is also clean, then we can attempt to remove the PMD mapping from it. This change will improve the efficiency of memory reclamation in this case.

On an Intel i5 CPU, reclaiming 1GiB of PMD-mapped THPs using mem_cgroup_force_empty() results in the following runtimes in seconds (shorter is better):

--------------------------------------------
| Old | New | Change |
--------------------------------------------
| 0.683426 | 0.049197 | -92.80% |
--------------------------------------------

---
  include/linux/huge_mm.h |  1 +
  include/linux/rmap.h    |  1 +
  mm/huge_memory.c        |  2 +-
  mm/rmap.c               | 81 +++++++++++++++++++++++++++++++++++++++++
  mm/vmscan.c             |  7 ++++
  5 files changed, 91 insertions(+), 1 deletion(-)
include/linux/huge_mm.h +1 -0
index 7cd07b83a3d0..02a71c05f68a 100644
@@ -36,6 +36,7 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,
  int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
  		    pmd_t *pmd, unsigned long addr, pgprot_t newprot,
  		    unsigned long cp_flags);
+inline void zap_deposited_table(struct mm_struct *mm, pmd_t *pmd);
 
  vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, pfn_t pfn, bool write);
  vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, pfn_t pfn, bool write);
include/linux/rmap.h +1 -0
index 0f906dc6d280..8c2f45713351 100644
@@ -100,6 +100,7 @@ enum ttu_flags {
  					 * do a final flush if necessary */
  	TTU_RMAP_LOCKED		= 0x80,	/* do not grab rmap lock:
  					 * caller holds it */
+	TTU_LAZYFREE_THP	= 0x100, /* avoid split PMD-mapped THP */
  };
 
  #ifdef CONFIG_MMU
mm/huge_memory.c +1 -1
index 58f2c4745d80..309fba9624c2 100644
@@ -1801,7 +1801,7 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
  	return ret;
  }
 
-static inline void zap_deposited_table(struct mm_struct *mm, pmd_t *pmd)
+inline void zap_deposited_table(struct mm_struct *mm, pmd_t *pmd)
  {
  	pgtable_t pgtable;
 
mm/rmap.c +17 -0
index 2608c40dffad..4994f9e402d4 100644
@@ -77,6 +77,7 @@
  #include <linux/mm_inline.h>
 
  #include <asm/tlbflush.h>
+#include <asm/tlb.h>
 
  #define CREATE_TRACE_POINTS
  #include <trace/events/tlb.h>
@@ -1606,6 +1607,80 @@ void folio_remove_rmap_pmd(struct folio *folio, struct page *page,
  #endif
  }
 
+static bool __try_to_unmap_lazyfree_thp(struct vm_area_struct *vma,
+					     unsigned long address,
+					     struct folio *folio)
+{
+	spinlock_t *ptl;
+	pmd_t *pmdp, orig_pmd;
+	struct mmu_notifier_range range;
+	struct mmu_gather tlb;
+	struct mm_struct *mm = vma->vm_mm;
+	struct page *page;
+	bool ret = false;
+
+	VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio);
+	VM_WARN_ON_FOLIO(folio_test_swapbacked(folio), folio);
+	VM_WARN_ON_FOLIO(!folio_test_pmd_mappable(folio), folio);
+	VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);

mm/rmap.c void folio_remove_rmap_pmd(struct folio *folio, struct page *page, · line 1635

Quoted 10 lines
mm/rmap.c +10 -0
+
+	/*
+	 * If we encounter a PMD-mapped THP that marked as lazyfree, we
+	 * will try to unmap it without splitting.
+	 *
+	 * The folio exclusively mapped should only have two refs:
+	 * one from the isolation and one from the rmap.
+	 */
+	if (folio_entire_mapcount(folio) != 1 || folio_test_dirty(folio) ||
+	    folio_ref_count(folio) != 2)

folio_mapcount() == 1 is a bit nicer. Bit I assume you can drop that completely and only check the refcount?

Quoted 15 lines
mm/rmap.c +15 -0
+		return false;
+
+	pmdp = mm_find_pmd(mm, address);
+	if (unlikely(!pmdp))
+		return false;
+	if (pmd_dirty(*pmdp))
+		return false;
+
+	tlb_gather_mmu(&tlb, mm);
+	mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm,
+				address & HPAGE_PMD_MASK,
+				(address & HPAGE_PMD_MASK) + HPAGE_PMD_SIZE);
+	mmu_notifier_invalidate_range_start(&range);
+
+	ptl = pmd_lock(mm, pmdp);

mm/rmap.c void folio_remove_rmap_pmd(struct folio *folio, struct page *page, · line 1660

Quoted 10 lines
mm/rmap.c +10 -0
+	orig_pmd = *pmdp;
+	if (unlikely(!pmd_present(orig_pmd) || !pmd_trans_huge(orig_pmd)))
+		goto out;
+
+	page = pmd_page(orig_pmd);
+	if (unlikely(page_folio(page) != folio))
+		goto out;
+
+	orig_pmd = pmdp_huge_get_and_clear(mm, address, pmdp);
+	tlb_remove_pmd_tlb_entry(&tlb, pmdp, address);

Until this point, the page could have been pinned (including GUP-fast) and we might be in trouble if we drop it.

Signature · 3 lines
Cheers,

David / dhildenb
author
#2 lore raw

Hey David,

Thanks for taking time to review.

Quoted 105 lines · David Hildenbrand
Quoted 103 lines · Lance Yang

When the user no longer requires the pages, they would use madvise(madv_free) to mark the pages as lazy free. IMO, they would not typically rewrite to the given range.

At present, a PMD-mapped THP marked as lazyfree during shrink_folio_list() is unconditionally split, which may be unnecessary. If the THP is exclusively mapped and clean, and the PMD associated with it is also clean, then we can attempt to remove the PMD mapping from it. This change will improve the efficiency of memory reclamation in this case.

On an Intel i5 CPU, reclaiming 1GiB of PMD-mapped THPs using mem_cgroup_force_empty() results in the following runtimes in seconds (shorter is better):

--------------------------------------------
| Old | New | Change |
--------------------------------------------
| 0.683426 | 0.049197 | -92.80% |
--------------------------------------------

---
  include/linux/huge_mm.h |  1 +
  include/linux/rmap.h    |  1 +
  mm/huge_memory.c        |  2 +-
  mm/rmap.c               | 81 +++++++++++++++++++++++++++++++++++++++++
  mm/vmscan.c             |  7 ++++
  5 files changed, 91 insertions(+), 1 deletion(-)
include/linux/huge_mm.h +1 -0
index 7cd07b83a3d0..02a71c05f68a 100644
@@ -36,6 +36,7 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,
  int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
                  pmd_t *pmd, unsigned long addr, pgprot_t newprot,
                  unsigned long cp_flags);
+inline void zap_deposited_table(struct mm_struct *mm, pmd_t *pmd);
 
  vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, pfn_t pfn, bool write);
  vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, pfn_t pfn, bool write);
include/linux/rmap.h +1 -0
index 0f906dc6d280..8c2f45713351 100644
@@ -100,6 +100,7 @@ enum ttu_flags {
                                       * do a final flush if necessary */
      TTU_RMAP_LOCKED         = 0x80, /* do not grab rmap lock:
                                       * caller holds it */
+     TTU_LAZYFREE_THP        = 0x100, /* avoid split PMD-mapped THP */
  };
 
  #ifdef CONFIG_MMU
mm/huge_memory.c +1 -1
index 58f2c4745d80..309fba9624c2 100644
@@ -1801,7 +1801,7 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
      return ret;
  }
 
-static inline void zap_deposited_table(struct mm_struct *mm, pmd_t *pmd)
+inline void zap_deposited_table(struct mm_struct *mm, pmd_t *pmd)
  {
      pgtable_t pgtable;
 
mm/rmap.c +22 -0
index 2608c40dffad..4994f9e402d4 100644
@@ -77,6 +77,7 @@
  #include <linux/mm_inline.h>
 
  #include <asm/tlbflush.h>
+#include <asm/tlb.h>
 
  #define CREATE_TRACE_POINTS
  #include <trace/events/tlb.h>
@@ -1606,6 +1607,80 @@ void folio_remove_rmap_pmd(struct folio *folio, struct page *page,
  #endif
  }
 
+static bool __try_to_unmap_lazyfree_thp(struct vm_area_struct *vma,
+                                          unsigned long address,
+                                          struct folio *folio)
+{
+     spinlock_t *ptl;
+     pmd_t *pmdp, orig_pmd;
+     struct mmu_notifier_range range;
+     struct mmu_gather tlb;
+     struct mm_struct *mm = vma->vm_mm;
+     struct page *page;
+     bool ret = false;
+
+     VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio);
+     VM_WARN_ON_FOLIO(folio_test_swapbacked(folio), folio);
+     VM_WARN_ON_FOLIO(!folio_test_pmd_mappable(folio), folio);
+     VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
+
+     /*
+      * If we encounter a PMD-mapped THP that marked as lazyfree, we
+      * will try to unmap it without splitting.
+      *
Quoted 8 lines

mm/rmap.c void folio_remove_rmap_pmd(struct folio *folio, struct page *page, · line 1635

Quoted 5 lines
mm/rmap.c +5 -0
+      * The folio exclusively mapped should only have two refs:
+      * one from the isolation and one from the rmap.
+      */
+     if (folio_entire_mapcount(folio) != 1 || folio_test_dirty(folio) ||
+         folio_ref_count(folio) != 2)

folio_mapcount() == 1 is a bit nicer. Bit I assume you can drop that completely and only check the refcount?

Thanks for your suggestion!

+ if (folio_test_dirty(folio) || folio_ref_count(folio) != 2)

I'm not sure if it's safe without checking the folio_mapcount.

Quoted 21 lines
Quoted 21 lines
mm/rmap.c +20 -0
+             return false;
+
+     pmdp = mm_find_pmd(mm, address);
+     if (unlikely(!pmdp))
+             return false;
+     if (pmd_dirty(*pmdp))
+             return false;
+
+     tlb_gather_mmu(&tlb, mm);
+     mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm,
+                             address & HPAGE_PMD_MASK,
+                             (address & HPAGE_PMD_MASK) + HPAGE_PMD_SIZE);
+     mmu_notifier_invalidate_range_start(&range);
+
+     ptl = pmd_lock(mm, pmdp);
+     orig_pmd = *pmdp;
+     if (unlikely(!pmd_present(orig_pmd) || !pmd_trans_huge(orig_pmd)))
+             goto out;
+
+     page = pmd_page(orig_pmd);
Quoted 8 lines

mm/rmap.c void folio_remove_rmap_pmd(struct folio *folio, struct page *page, · line 1660

Quoted 5 lines
mm/rmap.c +5 -0
+     if (unlikely(page_folio(page) != folio))
+             goto out;
+
+     orig_pmd = pmdp_huge_get_and_clear(mm, address, pmdp);
+     tlb_remove_pmd_tlb_entry(&tlb, pmdp, address);

Until this point, the page could have been pinned (including GUP-fast) and we might be in trouble if we drop it.

Thanks for pointing that out!

+ if (pmd_dirty(orig_pmd) || folio_maybe_dma_pinned(folio) || folio_ref_count(folio) != 2) {
+ set_pmd_at(mm, address, pmdp, orig_pmd);
+ } else {

Could I check the folio->_pincount using folio_maybe_dma_pinned() and then re-check the refcount here? Or should I just re-check the refcount?

IIUC, this folio has been already unlinked from the PMD and the process cannot get an additional pin on this folio.

Thanks again for the review!
Lance

Quoted 6 lines

--
Cheers,

David / dhildenb

#3 lore raw
In reply to Lance Yang
Quoted 20 lines · Lance Yang

When the user no longer requires the pages, they would use madvise(madv_free) to mark the pages as lazy free. IMO, they would not typically rewrite to the given range.

At present, a PMD-mapped THP marked as lazyfree during shrink_folio_list() is unconditionally split, which may be unnecessary. If the THP is exclusively mapped and clean, and the PMD associated with it is also clean, then we can attempt to remove the PMD mapping from it. This change will improve the efficiency of memory reclamation in this case.

On an Intel i5 CPU, reclaiming 1GiB of PMD-mapped THPs using mem_cgroup_force_empty() results in the following runtimes in seconds (shorter is better):

--------------------------------------------
| Old | New | Change |
--------------------------------------------
| 0.683426 | 0.049197 | -92.80% |
--------------------------------------------

Quoted 8 lines
---
 include/linux/huge_mm.h |  1 +
 include/linux/rmap.h    |  1 +
 mm/huge_memory.c        |  2 +-
 mm/rmap.c               | 81 +++++++++++++++++++++++++++++++++++++++++
 mm/vmscan.c             |  7 ++++
 5 files changed, 91 insertions(+), 1 deletion(-)

I'm confused why we need all this extra code. If we remove a folio from the pagecache, we can just call truncate_inode_folio() and unmap_mapping_folio() takes care of all the necessary unmappings. Why can't you call unmap_mapping_folio() here?

author
#4 lore raw

Hey Matthew,

Thanks for taking time to review!

Quoted 31 lines · Matthew Wilcox
Quoted 20 lines · Lance Yang

When the user no longer requires the pages, they would use madvise(madv_free) to mark the pages as lazy free. IMO, they would not typically rewrite to the given range.

At present, a PMD-mapped THP marked as lazyfree during shrink_folio_list() is unconditionally split, which may be unnecessary. If the THP is exclusively mapped and clean, and the PMD associated with it is also clean, then we can attempt to remove the PMD mapping from it. This change will improve the efficiency of memory reclamation in this case.

On an Intel i5 CPU, reclaiming 1GiB of PMD-mapped THPs using mem_cgroup_force_empty() results in the following runtimes in seconds (shorter is better):

--------------------------------------------
| Old | New | Change |
--------------------------------------------
| 0.683426 | 0.049197 | -92.80% |
--------------------------------------------

Quoted 8 lines
---
 include/linux/huge_mm.h |  1 +
 include/linux/rmap.h    |  1 +
 mm/huge_memory.c        |  2 +-
 mm/rmap.c               | 81 +++++++++++++++++++++++++++++++++++++++++
 mm/vmscan.c             |  7 ++++
 5 files changed, 91 insertions(+), 1 deletion(-)
Quoted 1 line

I'm confused why we need all this extra code. If we remove a folio

Thanks for pointing that out!

I've added a lot of extra code to rmap.c, and we don't need it for file pages - sorry. I'll reconsider where to place this code.

Quoted 3 lines

from the pagecache, we can just call truncate_inode_folio() and unmap_mapping_folio() takes care of all the necessary unmappings. Why can't you call unmap_mapping_folio() here?

Thanks for your suggestion.

But this change only avoids the splitting of *anon* large folios (PMD-mapped THPs) that are marked as lazyfree during shrink_folio_list().

IIUC, in some cases, we cannot unmap the THP marked as lazyfree here, such as when it's not exclusively mapped, dirty, pinned, etc. In such situations, we still need to return to try_to_unmap_one(), and then call split_huge_pmd_address() to split it.

Thanks again for the review.
Lance

author
#5 lore raw
Quoted 48 lines · Lance Yang

Hey Matthew,

Thanks for taking time to review!

Quoted 31 lines · Matthew Wilcox
Quoted 20 lines · Lance Yang

When the user no longer requires the pages, they would use madvise(madv_free) to mark the pages as lazy free. IMO, they would not typically rewrite to the given range.

At present, a PMD-mapped THP marked as lazyfree during shrink_folio_list() is unconditionally split, which may be unnecessary. If the THP is exclusively mapped and clean, and the PMD associated with it is also clean, then we can attempt to remove the PMD mapping from it. This change will improve the efficiency of memory reclamation in this case.

On an Intel i5 CPU, reclaiming 1GiB of PMD-mapped THPs using mem_cgroup_force_empty() results in the following runtimes in seconds (shorter is better):

--------------------------------------------
| Old | New | Change |
--------------------------------------------
| 0.683426 | 0.049197 | -92.80% |
--------------------------------------------

Quoted 8 lines
---
 include/linux/huge_mm.h |  1 +
 include/linux/rmap.h    |  1 +
 mm/huge_memory.c        |  2 +-
 mm/rmap.c               | 81 +++++++++++++++++++++++++++++++++++++++++
 mm/vmscan.c             |  7 ++++
 5 files changed, 91 insertions(+), 1 deletion(-)
Quoted 1 line

I'm confused why we need all this extra code. If we remove a folio

Thanks for pointing that out!

I've added a lot of extra code to rmap.c, and we don't need it for file pages - sorry. I'll reconsider where to place this code.

Quoted 3 lines

from the pagecache, we can just call truncate_inode_folio() and unmap_mapping_folio() takes care of all the necessary unmappings. Why can't you call unmap_mapping_folio() here?

Quoted 8 lines

Thanks for your suggestion.

But this change only avoids the splitting of *anon* large folios (PMD-mapped THPs) that are marked as lazyfree during shrink_folio_list().

IIUC, in some cases, we cannot unmap the THP marked as lazyfree here, such as when it's not exclusively mapped, dirty, pinned, etc.

I’d like to make a correction.

IMO, we can unmap the THP that is not exclusively mapped, but ensuring folio_ref_count() equals folio_mapcount() +1.

Thanks,
Lance

Quoted 5 lines

In such situations, we still need to return to try_to_unmap_one(), and then call split_huge_pmd_address() to split it.

Thanks again for the review.
Lance

#6 lore raw
Quoted 55 lines · Lance Yang
Quoted 54 lines · Lance Yang

Hey Matthew,

Thanks for taking time to review!

Quoted 31 lines · Matthew Wilcox
Quoted 20 lines · Lance Yang

When the user no longer requires the pages, they would use madvise(madv_free) to mark the pages as lazy free. IMO, they would not typically rewrite to the given range.

At present, a PMD-mapped THP marked as lazyfree during shrink_folio_list() is unconditionally split, which may be unnecessary. If the THP is exclusively mapped and clean, and the PMD associated with it is also clean, then we can attempt to remove the PMD mapping from it. This change will improve the efficiency of memory reclamation in this case.

On an Intel i5 CPU, reclaiming 1GiB of PMD-mapped THPs using mem_cgroup_force_empty() results in the following runtimes in seconds (shorter is better):

--------------------------------------------
| Old | New | Change |
--------------------------------------------
| 0.683426 | 0.049197 | -92.80% |
--------------------------------------------

Quoted 8 lines
---
  include/linux/huge_mm.h |  1 +
  include/linux/rmap.h    |  1 +
  mm/huge_memory.c        |  2 +-
  mm/rmap.c               | 81 +++++++++++++++++++++++++++++++++++++++++
  mm/vmscan.c             |  7 ++++
  5 files changed, 91 insertions(+), 1 deletion(-)
Quoted 1 line

I'm confused why we need all this extra code. If we remove a folio

Thanks for pointing that out!

I've added a lot of extra code to rmap.c, and we don't need it for file pages - sorry. I'll reconsider where to place this code.

Quoted 3 lines

from the pagecache, we can just call truncate_inode_folio() and unmap_mapping_folio() takes care of all the necessary unmappings. Why can't you call unmap_mapping_folio() here?

Thanks for your suggestion.

But this change only avoids the splitting of *anon* large folios (PMD-mapped THPs) that are marked as lazyfree during shrink_folio_list().

Quoted 7 lines
Quoted 2 lines

IIUC, in some cases, we cannot unmap the THP marked as lazyfree here, such as when it's not exclusively mapped, dirty, pinned, etc.

I’d like to make a correction.

IMO, we can unmap the THP that is not exclusively mapped, but ensuring folio_ref_count() equals folio_mapcount() +1.

You must follow the exact same logic as in try_to_unmap_one() I guess.

That is, unmap the page, syncing against concurrent GUP-fast. Then, check mapcount vs. refcount. If there are unexpected references, remap the page (set_pte_at).

Signature · 3 lines
Cheers,

David / dhildenb
author
#7 lore raw
Quoted 57 lines · David Hildenbrand
Quoted 55 lines · Lance Yang
Quoted 54 lines · Lance Yang

Hey Matthew,

Thanks for taking time to review!

Quoted 31 lines · Matthew Wilcox
Quoted 20 lines · Lance Yang

When the user no longer requires the pages, they would use madvise(madv_free) to mark the pages as lazy free. IMO, they would not typically rewrite to the given range.

At present, a PMD-mapped THP marked as lazyfree during shrink_folio_list() is unconditionally split, which may be unnecessary. If the THP is exclusively mapped and clean, and the PMD associated with it is also clean, then we can attempt to remove the PMD mapping from it. This change will improve the efficiency of memory reclamation in this case.

On an Intel i5 CPU, reclaiming 1GiB of PMD-mapped THPs using mem_cgroup_force_empty() results in the following runtimes in seconds (shorter is better):

--------------------------------------------
| Old | New | Change |
--------------------------------------------
| 0.683426 | 0.049197 | -92.80% |
--------------------------------------------

Quoted 8 lines
---
  include/linux/huge_mm.h |  1 +
  include/linux/rmap.h    |  1 +
  mm/huge_memory.c        |  2 +-
  mm/rmap.c               | 81 +++++++++++++++++++++++++++++++++++++++++
  mm/vmscan.c             |  7 ++++
  5 files changed, 91 insertions(+), 1 deletion(-)
Quoted 1 line

I'm confused why we need all this extra code. If we remove a folio

Thanks for pointing that out!

I've added a lot of extra code to rmap.c, and we don't need it for file pages - sorry. I'll reconsider where to place this code.

Quoted 3 lines

from the pagecache, we can just call truncate_inode_folio() and unmap_mapping_folio() takes care of all the necessary unmappings. Why can't you call unmap_mapping_folio() here?

Thanks for your suggestion.

But this change only avoids the splitting of *anon* large folios (PMD-mapped THPs) that are marked as lazyfree during shrink_folio_list().

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

IIUC, in some cases, we cannot unmap the THP marked as lazyfree here, such as when it's not exclusively mapped, dirty, pinned, etc.

I’d like to make a correction.

IMO, we can unmap the THP that is not exclusively mapped, but ensuring folio_ref_count() equals folio_mapcount() +1.

Hey David,

Thanks a lot for clarifying!

Quoted 1 line

You must follow the exact same logic as in try_to_unmap_one() I guess.

Agreed. I'll take a closer look at try_to_unmap_one() and follow the exact same logic - thanks!

Quoted 4 lines

That is, unmap the page, syncing against concurrent GUP-fast. Then, check mapcount vs. refcount. If there are unexpected references, remap the page (set_pte_at).

Yep, I understood. Could you please provide some suggestions on where to place the exact same logic?

Thanks again for your time!
Lance

Quoted 6 lines

--
Cheers,

David / dhildenb