mm-madvise-optimize-lazyfreeing-with-mthp-in-madvise_free.patch removed from -mm tree
The quilt patch titled
Subject: mm/madvise: optimize lazyfreeing with mTHP in madvise_free
has been removed from the -mm tree. Its filename was
mm-madvise-optimize-lazyfreeing-with-mthp-in-madvise_free.patch
This patch was dropped because it was merged into the mm-stable branch of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
------------------------------------------------------
From: Lance Yang <[email protected]>
Subject: mm/madvise: optimize lazyfreeing with mTHP in madvise_free
Date: Thu, 18 Apr 2024 21:44:35 +0800
This patch optimizes lazyfreeing with PTE-mapped mTHP[1] (Inspired by David Hildenbrand[2]). We aim to avoid unnecessary folio splitting if the large folio is fully mapped within the target range.
If a large folio is locked or shared, or if we fail to split it, we just leave it in place and advance to the next PTE in the range. But note that the behavior is changed; previously, any failure of this sort would cause the entire operation to give up. As large folios become more common, sticking to the old way could result in wasted opportunities.
On an Intel I5 CPU, lazyfreeing a 1GiB VMA backed by PTE-mapped folios of the same size results in the following runtimes for madvise(MADV_FREE) in seconds (shorter is better):
Folio Size | Old | New | Change
------------------------------------------
4KiB | 0.590251 | 0.590259 | 0%
16KiB | 2.990447 | 0.185655 | -94%
32KiB | 2.547831 | 0.104870 | -95%
64KiB | 2.457796 | 0.052812 | -97%
128KiB | 2.281034 | 0.032777 | -99%
256KiB | 2.230387 | 0.017496 | -99%
512KiB | 2.189106 | 0.010781 | -99%
1024KiB | 2.183949 | 0.007753 | -99%
2048KiB | 0.002799 | 0.002804 | 0%[1] https://lkml.kernel.org/r/[email protected]
[2] https://lore.kernel.org/linux-mm/[email protected]
---
mm/madvise.c | 85 +++++++++++++++++++++++++------------------------ 1 file changed, 44 insertions(+), 41 deletions(-)
mm/madvise.c renamed from mm/madvise.c~mm-madvise-optimize-lazyfreeing-with-mthp-in-madvise_free +44 -41
@@ -643,6 +643,7 @@ static int madvise_free_pte_range(pmd_t unsigned long end, struct mm_walk *walk) { + const cydp_t cydp_flags = CYDP_CLEAR_YOUNG | CYDP_CLEAR_DIRTY;struct mmu_gather *tlb = walk->private;struct mm_struct *mm = tlb->mm;struct vm_area_struct *vma = walk->vma;
@@ -697,44 +698,57 @@ static int madvise_free_pte_range(pmd_tcontinue ;/* -* If pmd isn't transhuge but the folio is large and -* is owned by only this process, split it and -* deactivate all pages. +* If we encounter a large folio, only split it if it is not +* fully mapped within the range we are operating on. Otherwise +* leave it as is so that it can be marked as lazyfree. If we +* fail to split a folio, leave it in place and advance to the +* next pte in the range. */ if (folio_test_large (folio)) { -int err; +bool any_young, any_dirty; -if (folio_likely_mapped_shared (folio)) -break ; -if (!folio_trylock (folio)) -break ; -folio_get (folio); -arch_leave_lazy_mmu_mode (); -pte_unmap_unlock (start_pte, ptl); - start_pte =NULL ; - err =split_folio (folio); -folio_unlock (folio); -folio_put (folio); -if (err) -break ; - start_pte = pte = -pte_offset_map_lock (mm, pmd, addr, &ptl); -if (!start_pte) -break ; -arch_enter_lazy_mmu_mode (); - pte--; - addr -=PAGE_SIZE ; -continue ; + nr =madvise_folio_pte_batch (addr, end, folio, pte, + ptent, &any_young, &any_dirty); + +if (nr <folio_nr_pages (folio)) { +int err; + +if (folio_likely_mapped_shared (folio)) +continue ; +if (!folio_trylock (folio)) +continue ; +folio_get (folio); +arch_leave_lazy_mmu_mode (); +pte_unmap_unlock (start_pte, ptl); + start_pte =NULL ; + err =split_folio (folio); +folio_unlock (folio); +folio_put (folio); + pte =pte_offset_map_lock (mm, pmd, addr, &ptl); + start_pte = pte; +if (!start_pte) +break ; +arch_enter_lazy_mmu_mode (); +if (!err) + nr =0 ; +continue ; + } + +if (any_young) + ptent =pte_mkyoung (ptent); +if (any_dirty) + ptent =pte_mkdirty (ptent); }if (folio_test_swapcache (folio) ||folio_test_dirty (folio)) {if (!folio_trylock (folio))continue ;/* -* If folio is shared with others, we mustn't clear -* the folio's dirty flag. +* If we have a large folio at this point, we know it is +* fully mapped so if its mapcount is the same as its +* number of pages, it must be exclusive. */ -if (folio_mapcount (folio) !=1 ) { +if (folio_mapcount (folio) !=folio_nr_pages (folio)) {folio_unlock (folio);continue ; }
@@ -750,19 +764,8 @@ static int madvise_free_pte_range(pmd_t }if (pte_young (ptent) ||pte_dirty (ptent)) { -/* -* Some of architecture(ex, PPC) don't update TLB -* with set_pte_at and tlb_remove_tlb_entry so for -* the portability, remap the pte with old|clean -* after pte clearing. -*/ - ptent =ptep_get_and_clear_full (mm, addr, pte, - tlb->fullmm); - - ptent =pte_mkold (ptent); - ptent =pte_mkclean (ptent); -set_pte_at (mm, addr, pte, ptent); -tlb_remove_tlb_entry (tlb, pte, addr); +clear_young_dirty_ptes (vma, addr, pte, nr, cydp_flags); +tlb_remove_tlb_entries (tlb, pte, nr, addr); }folio_mark_lazyfree (folio); }
_
Patches currently in -mm which might be from [email protected] are
Quoted 9 lines · Andrew Morton
The quilt patch titled
Subject: mm/madvise: optimize lazyfreeing with mTHP inmadvise_free
has been removed from the -mm tree. Its filename was
mm-madvise-optimize-lazyfreeing-with-mthp-in-madvise_free.patchThis patch was dropped because it was merged into the mm-stable branch of
git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
Andrew,
this series was moved to mm-stable a bit too fast for my taste.
Review is still going on, can we still remove that from mm-stable?
Signature · 3 lines
Cheers, David / dhildenb
Quoted 7 lines · On Tue, 7 May 2024 10:15:24 +0200 David Hildenbrand <[email protected]> wrote:
Quoted 6 lines · Andrew Morton
The quilt patch titled
Subject: mm/madvise: optimize lazyfreeing with mTHP inmadvise_free
has been removed from the -mm tree. Its filename was
mm-madvise-optimize-lazyfreeing-with-mthp-in-madvise_free.patch
Quoted 7 lines
Quoted 3 lines
This patch was dropped because it was merged into the mm-stable branch of
git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mmAndrew,
this series was moved to mm-stable a bit too fast for my taste.
It was nearly three weeks.
Quoted 1 line
Review is still going on, can we still remove that from mm-stable?
I can rebase if needed. Is there a reason to do so? Nothing that can't be fixed up with followup patches?
Quoted 13 lines · Andrew Morton
Quoted 11 lines · On Tue, 7 May 2024 10:15:24 +0200 David Hildenbrand <[email protected]> wrote:
Quoted 9 lines · Andrew Morton
The quilt patch titled
Subject: mm/madvise: optimize lazyfreeing with mTHP inmadvise_free
has been removed from the -mm tree. Its filename was
mm-madvise-optimize-lazyfreeing-with-mthp-in-madvise_free.patchThis patch was dropped because it was merged into the mm-stable branch of
git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
Quoted 5 lines
Quoted 3 lines
Andrew,
this series was moved to mm-stable a bit too fast for my taste.
It was nearly three weeks.
I'm sorry, I replied to the wrong mail. I thought this was:
[1] https://lkml.kernel.org/r/[email protected]
Because *that* is in mm-stable now as well. The major rewrite in v3 (sent April 29) that had no reviews but broke compilation and was resend in v4 (send May 1), to be moved to mm-stable on May 6.
That's certainly too rushed for my taste for something that is not high provide just before the merge window is about to open.
I have it on my review list, but was expecting this to not get moved to mm-stable at this point. I won't be able to review it this/next week. I see other people are reviewing it now.
Quoted 5 lines
Quoted 2 lines
Review is still going on, can we still remove that from mm-stable?
I can rebase if needed. Is there a reason to do so? Nothing that can't be fixed up with followup patches?
This here is good to go and saw sufficient review. [1] was too fast for my taste, but it's up to you to decide if we want to fix that up piece by piece as more review happens.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 10 lines · On Tue, 7 May 2024 18:54:09 +0200 David Hildenbrand <[email protected]> wrote:
Quoted 5 lines
Quoted 2 lines
Review is still going on, can we still remove that from mm-stable?
I can rebase if needed. Is there a reason to do so? Nothing that can't be fixed up with followup patches?
This here is good to go and saw sufficient review. [1] was too fast for my taste, but it's up to you to decide if we want to fix that up piece by piece as more review happens.
OK.
Both series are cleanly droppable. Maybe drop the later series "Reclaim lazyfree THP without splitting" - let's see what the required fixups look like?