mm/vmscan: avoid split PMD-mapped THP during shrink_folio_list()
Subsystems (5)
PATCH v2 1/1 mm/vmscan: avoid split PMD-mapped THP during shrink_folio_list()
When the user no longer requires the pages, they would use madvise(MADV_FREE) to mark the pages as lazy free. IMO, they would not typically rewrite to the given range.
At present, PMD-mapped THPs that are marked as lazyfree during shrink_folio_list() are unconditionally split, which may be unnecessary. If the THP is clean, its PMD is also clean, and there are no unexpected references, then we can attempt to remove the PMD mapping from it. This change will improve the efficiency of memory reclamation in this case.
On an Intel i5 CPU, reclaiming 1GiB of PMD-mapped THPs using mem_cgroup_force_empty() results in the following runtimes in seconds (shorter is better):
--------------------------------------------
| Old | New | Change |
--------------------------------------------
| 0.683426 | 0.049197 | -92.80% |
--------------------------------------------
---
v1 -> v2:
- Update the changelog
- Follow the exact same logic as in try_to_unmap_one() (per David Hildenbrand)
- Remove the extra code from rmap.c (per Matthew Wilcox)
- https://lore.kernel.org/linux-mm/[email protected]
include/linux/huge_mm.h | 2 + include/linux/rmap.h | 2 + mm/huge_memory.c | 88 +++++++++++++++++++++++++++++++++++++++++ mm/rmap.c | 6 +++ mm/vmscan.c | 7 ++++ 5 files changed, 105 insertions(+)
include/linux/huge_mm.h +2 -0
@@ -36,6 +36,8 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,int change_huge_pmd (struct mmu_gather *tlb,struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,pgprot_t newprot,unsigned long cp_flags); +bool discard_trans_pmd (struct vm_area_struct *vma,unsigned long addr, +struct folio *folio);vm_fault_t vmf_insert_pfn_pmd (struct vm_fault *vmf,pfn_t pfn,bool write);vm_fault_t vmf_insert_pfn_pud (struct vm_fault *vmf,pfn_t pfn,bool write);
include/linux/rmap.h +2 -0
@@ -100,6 +100,8 @@ enum ttu_flags {* do a final flush if necessary */ TTU_RMAP_LOCKED =0x80 ,/* do not grab rmap lock: * caller holds it */ + TTU_LAZYFREE_THP = 0x100, /* avoid splitting PMD-mapped THPs + * that are marked as lazyfree. */ };#ifdef CONFIG_MMU
mm/huge_memory.c +88 -0
94 lines not on this page · raw message
mm/rmap.c +6 -0
@@ -1631,6 +1631,12 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,if (flags &TTU_SYNC ) pvmw.flags =PVMW_SYNC ; +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +if (flags &TTU_LAZYFREE_THP ) +if (discard_trans_pmd (vma, address, folio)) +return true; +#endif +if (flags &TTU_SPLIT_HUGE_PMD )split_huge_pmd_address (vma, address, false, folio);
mm/vmscan.c +7 -0
@@ -1277,6 +1277,13 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (folio_test_pmd_mappable (folio)) flags |=TTU_SPLIT_HUGE_PMD ; + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +if (folio_test_anon (folio) && !was_swapbacked && + (flags &TTU_SPLIT_HUGE_PMD )) + flags |=TTU_LAZYFREE_THP ; +#endif +/* * Without TTU_SYNC, try_to_unmap will only begin to * hold PTL from the first present PTE within a large
Signature · 1 line
2.33.1
Quoted 9 lines · Lance Yang
When the user no longer requires the pages, they would use
madvise(MADV_FREE)to mark the pages as lazy free. IMO, they would not typically rewrite to the given range.At present, PMD-mapped THPs that are marked as lazyfree during
shrink_folio_list()are unconditionally split, which may be unnecessary. If the THP is clean, its PMD is also clean, and there are no unexpected references, then we can attempt to remove the PMD mapping from it. This change will improve the efficiency of memory reclamation in this case.
Does this happen outside of benchmarks? I'm really struggling to see how we end up in this situation. We have a clean THP without swap backing, so it's full of zeroes, but for some reason we haven't used the shared huge zero page? What is going on?
Quoted 11 lines · Matthew Wilcox
Quoted 9 lines · Lance Yang
When the user no longer requires the pages, they would use
madvise(MADV_FREE)to mark the pages as lazy free. IMO, they would not typically rewrite to the given range.At present, PMD-mapped THPs that are marked as lazyfree during
shrink_folio_list()are unconditionally split, which may be unnecessary. If the THP is clean, its PMD is also clean, and there are no unexpected references, then we can attempt to remove the PMD mapping from it. This change will improve the efficiency of memory reclamation in this case.
Quoted 4 lines
Does this happen outside of benchmarks? I'm really struggling to see how we end up in this situation. We have a clean THP without swap backing, so it's full of zeroes, but for some reason we haven't used the shared huge zero page? What is going on?
It's not full of zeroes.
User space called MADV_FREE on a PMD-mapped THP.
During MADV_FREE, we mark the PTEs as clean, the folio as clean and sd "lazyfree" (no swap backend). If, during memory reclaim, we detect that (a) the folio is still clean (b) the PTEs are still clean and (c) there are no unexpected references (GUP), user space didn't re-write to that memory again, so we can just discard the memory "lazily".
Signature · 3 lines
Cheers, David / dhildenb
Zi Yan
reviewerQuoted 17 lines · David Hildenbrand
Quoted 11 lines · Matthew Wilcox
Quoted 9 lines · Lance Yang
When the user no longer requires the pages, they would use
madvise(MADV_FREE)to mark the pages as lazy free. IMO, they would not typically rewrite to the given range.At present, PMD-mapped THPs that are marked as lazyfree during
shrink_folio_list()are unconditionally split, which may be unnecessary. If the THP is clean, its PMD is also clean, and there are no unexpected references, then we can attempt to remove the PMD mapping from it. This change will improve the efficiency of memory reclamation in this case.Quoted 4 lines
Does this happen outside of benchmarks? I'm really struggling to see how we end up in this situation. We have a clean THP without swap backing, so it's full of zeroes, but for some reason we haven't used the shared huge zero page? What is going on?
Quoted 5 lines
It's not full of zeroes.
User space called
MADV_FREEon a PMD-mapped THP.During
MADV_FREE, we mark the PTEs as clean, the folio as clean and sd "lazyfree" (no swap backend). If, during memory reclaim, we detect that (a) the folio is still clean (b) the PTEs are still clean and (c) there are no unexpected references (GUP), user space didn't re-write to that memory again, so we can just discard the memory "lazily".
It seems that try_to_unmap_one() does not support unmapping PMD-mapped folios. Maybe adding that support instead of a special case handling?
--
Best Regards,
Yan, Zi
Quoted 19 lines · Zi Yan
Quoted 17 lines · David Hildenbrand
Quoted 15 lines · Matthew Wilcox
Quoted 9 lines · Lance Yang
When the user no longer requires the pages, they would use
madvise(MADV_FREE)to mark the pages as lazy free. IMO, they would not typically rewrite to the given range.At present, PMD-mapped THPs that are marked as lazyfree during
shrink_folio_list()are unconditionally split, which may be unnecessary. If the THP is clean, its PMD is also clean, and there are no unexpected references, then we can attempt to remove the PMD mapping from it. This change will improve the efficiency of memory reclamation in this case.Does this happen outside of benchmarks? I'm really struggling to see how we end up in this situation. We have a clean THP without swap backing, so it's full of zeroes, but for some reason we haven't used the shared huge zero page? What is going on?
Quoted 8 lines
Quoted 5 lines
It's not full of zeroes.
User space called
MADV_FREEon a PMD-mapped THP.During
MADV_FREE, we mark the PTEs as clean, the folio as clean and sd "lazyfree" (no swap backend). If, during memory reclaim, we detect that (a) the folio is still clean (b) the PTEs are still clean and (c) there are no unexpected references (GUP), user space didn't re-write to that memory again, so we can just discard the memory "lazily".It seems that
try_to_unmap_one()does not support unmapping PMD-mapped folios. Maybe adding that support instead of a special case handling?
I was thinking the same, and finding a way to avoid TTU_LAZYFREE_THP.
Signature · 3 lines
Cheers, David / dhildenb
Hey Zi, David,
Thanks for taking time to review!
Quoted 23 lines · David Hildenbrand
Quoted 21 lines · Zi Yan
Quoted 19 lines · David Hildenbrand
Quoted 11 lines · Matthew Wilcox
Quoted 9 lines · Lance Yang
When the user no longer requires the pages, they would use
madvise(MADV_FREE)to mark the pages as lazy free. IMO, they would not typically rewrite to the given range.At present, PMD-mapped THPs that are marked as lazyfree during
shrink_folio_list()are unconditionally split, which may be unnecessary. If the THP is clean, its PMD is also clean, and there are no unexpected references, then we can attempt to remove the PMD mapping from it. This change will improve the efficiency of memory reclamation in this case.Quoted 4 lines
Does this happen outside of benchmarks? I'm really struggling to see how we end up in this situation. We have a clean THP without swap backing, so it's full of zeroes, but for some reason we haven't used the shared huge zero page? What is going on?
It's not full of zeroes.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
User space called
MADV_FREEon a PMD-mapped THP.During
MADV_FREE, we mark the PTEs as clean, the folio as clean and sd "lazyfree" (no swap backend). If, during memory reclaim, we detect that (a) the folio is still clean (b) the PTEs are still clean and (c) there are no unexpected references (GUP), user space didn't re-write to that memory again, so we can just discard the memory "lazily".It seems that
try_to_unmap_one()does not support unmapping PMD-mapped folios. Maybe adding that support instead of a special case handling?I was thinking the same, and finding a way to avoid
TTU_LAZYFREE_THP.
Thanks for the suggestions!
Yep, I completely agreed. Adding support for unmapping PMD-mapped folios to try_to_unmap_one() would make it more future-proof.
Thanks again for the review!
Lance
Quoted 6 lines
--
Cheers,David / dhildenb
Hey Zi, David,
How about this change(diff against mm-unstable) as follows?
I'd like to add __try_to_unmap_huge_pmd() as a new internal function specifically for unmapping PMD-mapped folios. If, for any reason, we cannot unmap the folio, then we'll still split it as previously done.
Currently, __try_to_unmap_huge_pmd() only handles lazyfree THPs, but it can be extended to support other large folios that are PMD-mapped in the future if needed.
include/linux/rmap.h +0 -2
@@ -100,8 +100,6 @@ enum ttu_flags {* do a final flush if necessary */ TTU_RMAP_LOCKED = 0x80, /* do not grab rmap lock:* caller holds it */ - TTU_LAZYFREE_THP = 0x100, /* avoid splitting PMD-mapped THPs - * that are marked as lazyfree. */ };#ifdef CONFIG_MMU
mm/rmap.c +16 -6
@@ -1606,6 +1606,19 @@ void folio_remove_rmap_pmd(struct folio *folio, struct page *page,#endif } +static bool __try_to_unmap_huge_pmd (struct vm_area_struct *vma, +unsigned long addr,struct folio *folio) +{ +VM_WARN_ON_FOLIO (!folio_test_pmd_mappable (folio), folio); + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +if (folio_test_anon (folio) && !folio_test_swapbacked (folio)) +return discard_trans_pmd (vma, addr, folio); +#endif + +return false; +} +/* * @arg: enum ttu_flags will be passed to this argument */
@@ -1631,14 +1644,11 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,if (flags &TTU_SYNC ) pvmw.flags =PVMW_SYNC ; -#ifdef CONFIG_TRANSPARENT_HUGEPAGE -if (flags &TTU_LAZYFREE_THP ) -if (discard_trans_pmd (vma, address, folio)) +if (flags &TTU_SPLIT_HUGE_PMD ) { +if (__try_to_unmap_huge_pmd (vma, address, folio))return true; -#endif - -if (flags &TTU_SPLIT_HUGE_PMD )split_huge_pmd_address (vma, address, false, folio); + }/* * For THP, we have to assume the worse case ie pmd for invalidation.
mm/vmscan.c +0 -7
@@ -1277,13 +1277,6 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (folio_test_pmd_mappable (folio)) flags |=TTU_SPLIT_HUGE_PMD ; - -#ifdef CONFIG_TRANSPARENT_HUGEPAGE -if (folio_test_anon (folio) && !was_swapbacked && - (flags &TTU_SPLIT_HUGE_PMD )) - flags |=TTU_LAZYFREE_THP ; -#endif -/* * Without TTU_SYNC, try_to_unmap will only begin to * hold PTL from the first present PTE within a large
Signature · 2 lines
Thanks, Lance
Quoted 1 line · Lance Yang
Hey Zi, David,
Hi,
Quoted 2 lines
How about this change(diff against mm-unstable) as follows?
goes into the right direction, please resent the whole thing, that will make it easier to review.
Quoted 52 lines
I'd like to add
__try_to_unmap_huge_pmd()as a new internal function specifically for unmapping PMD-mapped folios. If, for any reason, we cannot unmap the folio, then we'll still split it as previously done.Currently,
__try_to_unmap_huge_pmd()only handles lazyfree THPs, but it can be extended to support other large folios that are PMD-mapped in the future if needed.include/linux/rmap.h +0
-2@@ -100,8 +100,6 @@ enum ttu_flags {* do a final flush if necessary */ TTU_RMAP_LOCKED = 0x80, /* do not grab rmap lock:* caller holds it */ - TTU_LAZYFREE_THP = 0x100, /* avoid splitting PMD-mapped THPs - * that are marked as lazyfree. */ };#ifdef CONFIG_MMU mm/rmap.c +13
-2@@ -1606,6 +1606,19 @@ void folio_remove_rmap_pmd(struct folio *folio, struct page *page,#endif } +static bool __try_to_unmap_huge_pmd (struct vm_area_struct *vma, +unsigned long addr,struct folio *folio) +{ +VM_WARN_ON_FOLIO (!folio_test_pmd_mappable (folio), folio); + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +if (folio_test_anon (folio) && !folio_test_swapbacked (folio)) +return discard_trans_pmd (vma, addr, folio); +#endif + +return false; +} +/* * @arg: enum ttu_flags will be passed to this argument */ @@ -1631,14 +1644,11 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,if (flags &TTU_SYNC ) pvmw.flags =PVMW_SYNC ; -#ifdef CONFIG_TRANSPARENT_HUGEPAGE -if (flags &TTU_LAZYFREE_THP )
mm/rmap.c static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, · line 1652
Quoted 10 lines
mm/rmap.c +3
-4-if (discard_trans_pmd (vma, address, folio)) +if (flags &TTU_SPLIT_HUGE_PMD ) { +if (__try_to_unmap_huge_pmd (vma, address, folio))return true; -#endif - -if (flags &TTU_SPLIT_HUGE_PMD )split_huge_pmd_address (vma, address, false, folio); + }
I was wondering if we can better integrate that into the pagewalk below.
That is, don't do the TTU_SPLIT_HUGE_PMD immediately. Start the pagewalk first. If we walk a PMD, try to unmap it. Only if that fails, split it.
Less working on "vma + address" and instead directly on PMDs.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 11 lines · David Hildenbrand
Quoted 1 line · Lance Yang
Hey Zi, David,
Hi,
Quoted 2 lines
How about this change(diff against mm-unstable) as follows?
goes into the right direction, please resent the whole thing, that will make it easier to review.
Got it. I‘ll keep that in mind, thanks!
Quoted 66 lines
Quoted 53 lines
I'd like to add
__try_to_unmap_huge_pmd()as a new internal function specifically for unmapping PMD-mapped folios. If, for any reason, we cannot unmap the folio, then we'll still split it as previously done.Currently,
__try_to_unmap_huge_pmd()only handles lazyfree THPs, but it can be extended to support other large folios that are PMD-mapped in the future if needed.include/linux/rmap.h +0
-2@@ -100,8 +100,6 @@ enum ttu_flags {* do a final flush if necessary */ TTU_RMAP_LOCKED = 0x80, /* do not grab rmap lock:* caller holds it */ -TTU_LAZYFREE_THP =0x100 ,/* avoid splitting PMD-mapped THPs -* that are marked as lazyfree. */ };#ifdef CONFIG_MMU mm/rmap.c +13
-2@@ -1606,6 +1606,19 @@ void folio_remove_rmap_pmd(struct folio *folio, struct page *page,#endif } +static bool __try_to_unmap_huge_pmd (struct vm_area_struct *vma, +unsigned long addr,struct folio *folio) +{ +VM_WARN_ON_FOLIO (!folio_test_pmd_mappable (folio), folio); + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +if (folio_test_anon (folio) && !folio_test_swapbacked (folio)) +return discard_trans_pmd (vma, addr, folio); +#endif + +return false; +} +/* * @arg: enum ttu_flags will be passed to this argument */ @@ -1631,14 +1644,11 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,if (flags &TTU_SYNC ) pvmw.flags =PVMW_SYNC ; -#ifdef CONFIG_TRANSPARENT_HUGEPAGE -if (flags &TTU_LAZYFREE_THP )mm/rmap.c static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, · line 1652
Quoted 10 lines
mm/rmap.c +3
-4-if (discard_trans_pmd (vma, address, folio)) +if (flags &TTU_SPLIT_HUGE_PMD ) { +if (__try_to_unmap_huge_pmd (vma, address, folio))return true; -#endif - -if (flags &TTU_SPLIT_HUGE_PMD )split_huge_pmd_address (vma, address, false, folio); + }I was wondering if we can better integrate that into the pagewalk below.
Quoted 2 lines
That is, don't do the
TTU_SPLIT_HUGE_PMDimmediately. Start the pagewalk first. If we walk a PMD, try to unmap it. Only if that fails, split it.
Nice. Thanks for the suggestion!
I'll work on integrating it into the pagewalk as you suggested.
Quoted 2 lines
Less working on "vma + address" and instead directly on PMDs.
Yes, some of the work on "vma + address" can be avoided :)
Thanks again for the review!
Lance
Quoted 6 lines
--
Cheers,David / dhildenb
Quoted 5 lines
Quoted 4 lines
I was wondering if we can better integrate that into the pagewalk below.
That is, don't do the
TTU_SPLIT_HUGE_PMDimmediately. Start the pagewalk first. If we walk a PMD, try to unmap it. Only if that fails, split it.
Quoted 7 lines
Nice. Thanks for the suggestion!
I'll work on integrating it into the pagewalk as you suggested.Quoted 2 lines
Less working on "vma + address" and instead directly on PMDs.
Yes, some of the work on "vma + address" can be avoided :)
Doing the conditional split while in the pagewalk will be the interesting bit to sort out (we temporarily have to drop the PTL and start once again from that now-PTE-mapped page table). But it should be a reasonable thing to have.
Please let us know if you run into bigger issues with that!
See walk_pmd_range() as an inspiration where we call split_huge_pmd().
Signature · 3 lines
Cheers, David / dhildenb
Quoted 10 lines · David Hildenbrand
Quoted 10 lines
Quoted 5 lines
I was wondering if we can better integrate that into the pagewalk below.
That is, don't do the
TTU_SPLIT_HUGE_PMDimmediately. Start the pagewalk first. If we walk a PMD, try to unmap it. Only if that fails, split it.Nice. Thanks for the suggestion!
I'll work on integrating it into the pagewalk as you suggested.Quoted 1 line
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Less working on "vma + address" and instead directly on PMDs.
Yes, some of the work on "vma + address" can be avoided :)
Doing the conditional split while in the pagewalk will be the interesting bit to sort out (we temporarily have to drop the PTL and start once again from that now-PTE-mapped page table). But it should be a reasonable thing to have.
Yep, it's an interesting challenge for me, but definitely worth tackling ;)
Quoted 2 lines
Please let us know if you run into bigger issues with that!
Thanks, I'll do my best to sort it out and reach out for help if needed :p
Quoted 2 lines
See
walk_pmd_range()as an inspiration where we callsplit_huge_pmd().
Thanks for your suggestion! I'll take a closer look at it.
Thanks again for clarifying!
Lance
Quoted 6 lines
--
Cheers,David / dhildenb
Hey Matthew,
Thanks for taking time to review.
Quoted 9 lines · Matthew Wilcox
Quoted 7 lines · Lance Yang
When the user no longer requires the pages, they would use
madvise(MADV_FREE)to mark the pages as lazy free. IMO, they would not typically rewrite to the given range.At present, PMD-mapped THPs that are marked as lazyfree during
shrink_folio_list()are unconditionally split, which may be unnecessary. If the THP is clean, its PMD is also clean, and there are no unexpected
"If the THP is clean, its PMD is also clean" can be confusing - sorry. It should be modified to "If the THP and its PMD are both marked as clean".
Thanks,
Lance
Quoted 8 lines
Quoted 2 lines
references, then we can attempt to remove the PMD mapping from it. This change will improve the efficiency of memory reclamation in this case.
Does this happen outside of benchmarks? I'm really struggling to see how we end up in this situation. We have a clean THP without swap backing, so it's full of zeroes, but for some reason we haven't used the shared huge zero page? What is going on?
Andrew Morton
maintainerQuoted 7 lines · On Wed, 24 Apr 2024 23:46:59 +0800 Lance Yang <[email protected]> wrote:
Quoted 6 lines · Matthew Wilcox
Quoted 4 lines · Lance Yang
When the user no longer requires the pages, they would use
madvise(MADV_FREE)to mark the pages as lazy free. IMO, they would not typically rewrite to the given range.
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
At present, PMD-mapped THPs that are marked as lazyfree during
shrink_folio_list()are unconditionally split, which may be unnecessary. If the THP is clean, its PMD is also clean, and there are no unexpected"If the THP is clean, its PMD is also clean" can be confusing - sorry. It should be modified to "If the THP and its PMD are both marked as clean".
I made that changelog edit.
Quoted 10 lines · Andrew Morton
Quoted 7 lines · On Wed, 24 Apr 2024 23:46:59 +0800 Lance Yang <[email protected]> wrote:
Quoted 6 lines · Matthew Wilcox
Quoted 4 lines · Lance Yang
When the user no longer requires the pages, they would use
madvise(MADV_FREE)to mark the pages as lazy free. IMO, they would not typically rewrite to the given range.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
At present, PMD-mapped THPs that are marked as lazyfree during
shrink_folio_list()are unconditionally split, which may be unnecessary. If the THP is clean, its PMD is also clean, and there are no unexpected"If the THP is clean, its PMD is also clean" can be confusing - sorry. It should be modified to "If the THP and its PMD are both marked as clean".
I made that changelog edit.
Thanks for updating, Andrew!
Lance