mm/khugepaged: optimize collapse candidate detection
Subsystems (4)
Lance Yang
reviewerv2 cover mm/khugepaged: optimize collapse candidate detection
Hi all,
This series improves the THP scanner by teaching it to detect unsuitable collapse candidates, such as guard PTEs, earlier in the scan to avoid wasted work.
Thanks,
Lance
---
v1 -> v2:
- #01 Dropped patch that skipped mlocked VMAs from THP collapse, per discussion. Thanks a lot to Hugh, Kiryl, Lorenzo, Dev and Mika!
- #02 Pick RB from Lorenzo - thanks!
Update the changelog (per Lorenzo)
Pick AB from David - thanks!
- #03 Reuse SCAN_PTE_NON_PRESENT for guard PTEs (per Lorenzo)
- https://lore.kernel.org/linux-mm/[email protected]
Lance Yang (2):
mm: make is_guard_pte_marker() available for hugepage collapse mm/khugepaged: abort collapse scan on guard PTEs
include/linux/swapops.h | 6 ++++++ mm/khugepaged.c | 10 ++++++++++ mm/madvise.c | 6 ------ 3 files changed, 16 insertions(+), 6 deletions(-)
Signature · 1 line
2.49.0
Lance Yang
reviewerPATCH v2 1/2 mm: make is_guard_pte_marker() available for hugepage collapse
From: Lance Yang <[email protected]>
The hugepage collapsing code needs is_guard_pte_marker() to correctly handle PTE guard markers. Move the helper to a shared header and expose it.
While at it, simplify the implementation. The current code is redundant as it effectively expands to:
is_swap_pte(pte) && is_pte_marker_entry(...) && // from is_pte_marker() is_pte_marker_entry(...) // from is_guard_swp_entry()
While a modern compiler could likely optimize this away, let's have clean code and not rely on it.
--- include/linux/swapops.h | 6 ++++++ mm/madvise.c | 6 ------ 2 files changed, 6 insertions(+), 6 deletions(-)
include/linux/swapops.h +6 -0
@@ -469,6 +469,12 @@ static inline int is_guard_swp_entry(swp_entry_t entry) (pte_marker_get (entry) &PTE_MARKER_GUARD ); } +static inline bool is_guard_pte_marker (pte_t ptent) +{ +return is_swap_pte (ptent) && +is_guard_swp_entry (pte_to_swp_entry (ptent)); +} +/* * This is a special version to check pte_none() just to cover the case when * the pte is a pte marker. It existed because in many cases the pte marker
mm/madvise.c +0 -6
@@ -1069,12 +1069,6 @@ static bool is_valid_guard_vma(struct vm_area_struct *vma, bool allow_locked)return !(vma->vm_flags & disallowed); } -static bool is_guard_pte_marker (pte_t ptent) -{ -return is_pte_marker (ptent) && -is_guard_swp_entry (pte_to_swp_entry (ptent)); -} -static int guard_install_pud_entry (pud_t *pud,unsigned long addr,unsigned long next,struct mm_walk *walk) {
Signature · 1 line
2.49.0
Zi Yan
reviewerRe: [PATCH mm-new v2 1/2] mm: make is_guard_pte_marker() available for hugepage collapse
Quoted 16 lines · Lance Yang
From: Lance Yang <[email protected]>
The hugepage collapsing code needs
is_guard_pte_marker()to correctly handle PTE guard markers. Move the helper to a shared header and expose it.While at it, simplify the implementation. The current code is redundant as it effectively expands to:
is_swap_pte(pte) && is_pte_marker_entry(...) && // from is_pte_marker() is_pte_marker_entry(...) // from is_guard_swp_entry()While a modern compiler could likely optimize this away, let's have clean code and not rely on it.
Quoted 9 lines
Cc Kairui Song <[email protected]>Reviewed-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>--- include/linux/swapops.h | 6 ++++++ mm/madvise.c | 6 ------ 2 files changed, 6 insertions(+), 6 deletions(-)
LGTM. Reviewed-by: Zi Yan <[email protected]>
Best Regards,
Yan, Zi
Re: [PATCH mm-new v2 1/2] mm: make is_guard_pte_marker() available for hugepage collapse
In reply to Lance YangQuoted 13 lines · Lance Yang
From: Lance Yang <[email protected]>
The hugepage collapsing code needs
is_guard_pte_marker()to correctly handle PTE guard markers. Move the helper to a shared header and expose it.While at it, simplify the implementation. The current code is redundant as it effectively expands to:
is_swap_pte(pte)&&is_pte_marker_entry(...)&& // fromis_pte_marker()
is_pte_marker_entry(...)// fromis_guard_swp_entry()
Quoted 7 lines
While a modern compiler could likely optimize this away, let's have clean code and not rely on it.
Cc Kairui Song <[email protected]>Reviewed-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>
Signature · 2 lines
Wei Yang Help you, Help me
Lance Yang
reviewerPATCH v2 2/2 mm/khugepaged: abort collapse scan on guard PTEs
In reply to Lance YangFrom: Lance Yang <[email protected]>
Guard PTE markers are installed via MADV_GUARD_INSTALL to create lightweight guard regions.
Currently, any collapse path (khugepaged or MADV_COLLAPSE) will fail when encountering such a range.
MADV_COLLAPSE fails deep inside the collapse logic when trying to swap-in the special marker in __collapse_huge_page_swapin().
hpage_collapse_scan_pmd()
`- collapse_huge_page()
`- __collapse_huge_page_swapin() -> fails!
khugepaged's behavior is slightly different due to its max_ptes_swap limit (default 64). It won't fail as deep, but it will still needlessly scan up to 64 swap entries before bailing out.
IMHO, we can and should detect this much earlier.
This patch adds a check directly inside the PTE scan loop. If a guard marker is found, the scan is aborted immediately with SCAN_PTE_NON_PRESENT, avoiding wasted work.
--- mm/khugepaged.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
mm/khugepaged.c +10 -0
@@ -1306,6 +1306,16 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, result =SCAN_PTE_UFFD_WP ; goto out_unmap; } +/* +* Guard PTE markers are installed by +* MADV_GUARD_INSTALL. Any collapse path must +* not touch them, so abort the scan immediately +* if one is found. +*/ +if (is_guard_pte_marker (pteval)) { + result =SCAN_PTE_NON_PRESENT ; + goto out_unmap; + }continue ; }else { result =SCAN_EXCEED_SWAP_PTE ;
Signature · 1 line
2.49.0
Dev Jain
reviewerRe: [PATCH mm-new v2 2/2] mm/khugepaged: abort collapse scan on guard PTEs
Quoted 44 lines · Lance Yang
From: Lance Yang <[email protected]>
Guard PTE markers are installed via
MADV_GUARD_INSTALLto create lightweight guard regions.Currently, any collapse path (khugepaged or
MADV_COLLAPSE) will fail when encountering such a range.
MADV_COLLAPSEfails deep inside the collapse logic when trying to swap-in the special marker in__collapse_huge_page_swapin().
hpage_collapse_scan_pmd()
`-collapse_huge_page()`- __collapse_huge_page_swapin() -> fails!khugepaged's behavior is slightly different due to its
max_ptes_swaplimit (default 64). It won't fail as deep, but it will still needlessly scan up to 64 swap entries before bailing out.IMHO, we can and should detect this much earlier.
This patch adds a check directly inside the PTE scan loop. If a guard marker is found, the scan is aborted immediately with
SCAN_PTE_NON_PRESENT, avoiding wasted work.Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>--- mm/khugepaged.c | 10 ++++++++++ 1 file changed, 10 insertions(+)mm/khugepaged.c +3
-0@@ -1306,6 +1306,16 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, result =SCAN_PTE_UFFD_WP ; goto out_unmap; } +/* +* Guard PTE markers are installed by +* MADV_GUARD_INSTALL. Any collapse path must
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1321
Quoted 10 lines
mm/khugepaged.c +7
-0+* not touch them, so abort the scan immediately +* if one is found. +*/ +if (is_guard_pte_marker (pteval)) { + result =SCAN_PTE_NON_PRESENT ; + goto out_unmap; + }continue ; }else { result =SCAN_EXCEED_SWAP_PTE ;
I would like to hear everyone else's thoughts on
https://lore.kernel.org/linux-mm/[email protected]/
wherein I suggest that we should not continue to try collapsing other regions but immediately exit. The SCAN_PTE_NON_PRESENT case does not exit.
Lance Yang
reviewerRe: [PATCH mm-new v2 2/2] mm/khugepaged: abort collapse scan on guard PTEs
Quoted 54 lines · Dev Jain
Quoted 52 lines · Lance Yang
From: Lance Yang <[email protected]>
Guard PTE markers are installed via
MADV_GUARD_INSTALLto create lightweight guard regions.Currently, any collapse path (khugepaged or
MADV_COLLAPSE) will fail when encountering such a range.
MADV_COLLAPSEfails deep inside the collapse logic when trying to swap-in the special marker in__collapse_huge_page_swapin().
hpage_collapse_scan_pmd()
`-collapse_huge_page()
`-__collapse_huge_page_swapin()-> fails!khugepaged's behavior is slightly different due to its
max_ptes_swaplimit (default 64). It won't fail as deep, but it will still needlessly scan up to 64 swap entries before bailing out.IMHO, we can and should detect this much earlier.
This patch adds a check directly inside the PTE scan loop. If a guard marker is found, the scan is aborted immediately with
SCAN_PTE_NON_PRESENT, avoiding wasted work.Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>---
mm/khugepaged.c| 10 ++++++++++1 file changed, 10 insertions(+)mm/khugepaged.c +10
-0@@ -1306,6 +1306,16 @@ static int hpage_collapse_scan_pmd(structmm_struct *mm, result = SCAN_PTE_UFFD_WP; goto out_unmap; } + /* + * Guard PTE markers are installed by + * MADV_GUARD_INSTALL. Any collapse path must + * not touch them, so abort the scan immediately + * if one is found. + */ + if (is_guard_pte_marker(pteval)) { + result = SCAN_PTE_NON_PRESENT; + goto out_unmap; + } continue; } else {
Quoted 8 lines
Quoted 3 lines
result =
SCAN_EXCEED_SWAP_PTE;I would like to hear everyone else's thoughts on
https://lore.kernel.org/linux-mm/750a06dc-db3d-43c6- [email protected]/ wherein I suggest that we should not continue to try collapsing other regions but immediately exit. TheSCAN_PTE_NON_PRESENTcase does not exit.
Yes! Let's hear what other folks think on that[1].
[1] https://lore.kernel.org/linux-mm/[email protected]
Cheers,
Lance
Re: [PATCH mm-new v2 2/2] mm/khugepaged: abort collapse scan on guard PTEs
Quoted 70 lines · Lance Yang
Quoted 58 lines · Dev Jain
Quoted 56 lines · Lance Yang
From: Lance Yang <[email protected]>
Guard PTE markers are installed via
MADV_GUARD_INSTALLto create lightweight guard regions.Currently, any collapse path (khugepaged or
MADV_COLLAPSE) will fail when encountering such a range.
MADV_COLLAPSEfails deep inside the collapse logic when trying to swap-in the special marker in__collapse_huge_page_swapin().
hpage_collapse_scan_pmd()
`-collapse_huge_page()
`-__collapse_huge_page_swapin()-> fails!khugepaged's behavior is slightly different due to its
max_ptes_swaplimit
(default 64). It won't fail as deep, but it will still needlessly scan up to 64 swap entries before bailing out.IMHO, we can and should detect this much earlier.
This patch adds a check directly inside the PTE scan loop. If a guard marker is found, the scan is aborted immediately with
SCAN_PTE_NON_PRESENT,
avoiding wasted work.Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>---
mm/khugepaged.c| 10 ++++++++++1 file changed, 10 insertions(+)mm/khugepaged.c +10
-0@@ -1306,6 +1306,16 @@ static int hpage_collapse_scan_pmd(structmm_struct *mm, result = SCAN_PTE_UFFD_WP; goto out_unmap; } + /* + * Guard PTE markers are installed by + * MADV_GUARD_INSTALL. Any collapse path must + * not touch them, so abort the scan immediately + * if one is found. + */ + if (is_guard_pte_marker(pteval)) { + result = SCAN_PTE_NON_PRESENT; + goto out_unmap; + } continue; } else {result =
SCAN_EXCEED_SWAP_PTE;Quoted 8 lines
Quoted 1 line
I would like to hear everyone else's thoughts on
https://lore.kernel.org/linux-mm/750a06dc-db3d-43c6-
[email protected]/
wherein I suggest that we should not continue to try collapsing other regions
but immediately exit. TheSCAN_PTE_NON_PRESENTcase does not exit.
Quoted 3 lines
Yes! Let's hear what other folks think on that[1].
Since the code has changed let's discuss on this thread.
Dev - You can have guard regions in a range that prevent one PMD from being collapsed, I'm struggling to understand why you'd want to abort the whole thing?
Your reasoning there isn't clear at all, so if I had a guard region in one page in a giant range I was trying to collapse, you're saying we should just abort the whole thing?
I really don't understand why we would do that? You just skip over what you can't collapse right?
There's no reason at all to assume that overlapping regions here matter, we can't predict how users will use this.
As Lance says, it's best effort. And also note we already do this with UFFD WP. And note this is also a non-present, PTE marker.
And also this would change existing behaviour which treats this as a swap entry then just fails later down the line right?
So yeah I don't agree, I think it's fine as is, unless I'm missing something here.
Cheers, Lorenzo
Dev Jain
reviewerRe: [PATCH mm-new v2 2/2] mm/khugepaged: abort collapse scan on guard PTEs
Quoted 72 lines · Lorenzo Stoakes
Quoted 66 lines · Lance Yang
Quoted 56 lines · Dev Jain
Quoted 55 lines · Lance Yang
From: Lance Yang <[email protected]>
Guard PTE markers are installed via
MADV_GUARD_INSTALLto create lightweight guard regions.Currently, any collapse path (khugepaged or
MADV_COLLAPSE) will fail when encountering such a range.
MADV_COLLAPSEfails deep inside the collapse logic when trying to swap-in the special marker in__collapse_huge_page_swapin().
hpage_collapse_scan_pmd()
`-collapse_huge_page()
`-__collapse_huge_page_swapin()-> fails!khugepaged's behavior is slightly different due to its
max_ptes_swaplimit
(default 64). It won't fail as deep, but it will still needlessly scan up to 64 swap entries before bailing out.IMHO, we can and should detect this much earlier.
This patch adds a check directly inside the PTE scan loop. If a guard marker is found, the scan is aborted immediately with
SCAN_PTE_NON_PRESENT,
avoiding wasted work.Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>--- mm/khugepaged.c | 10 ++++++++++ 1 file changed, 10 insertions(+)mm/khugepaged.c +10
-0@@ -1306,6 +1306,16 @@ static int hpage_collapse_scan_pmd(structmm_struct *mm, result = SCAN_PTE_UFFD_WP; goto out_unmap; } + /* + * Guard PTE markers are installed by + * MADV_GUARD_INSTALL. Any collapse path must + * not touch them, so abort the scan immediately + * if one is found. + */ + if (is_guard_pte_marker(pteval)) { + result = SCAN_PTE_NON_PRESENT; + goto out_unmap; + } continue; } else {result =
SCAN_EXCEED_SWAP_PTE;Quoted 8 lines
Quoted 2 lines
I would like to hear everyone else's thoughts on
https://lore.kernel.org/linux-mm/750a06dc-db3d-43c6-
[email protected]/
wherein I suggest that we should not continue to try collapsing other regions
but immediately exit. TheSCAN_PTE_NON_PRESENTcase does not exit.Quoted 3 lines
Yes! Let's hear what other folks think on that[1].
Since the code has changed let's discuss on this thread.
Quoted 7 lines
Dev - You can have guard regions in a range that prevent one PMD from being collapsed, I'm struggling to understand why you'd want to abort the whole thing?
Your reasoning there isn't clear at all, so if I had a guard region in one page in a giant range I was trying to collapse, you're saying we should just abort the whole thing?
My reasoning was that it doesn't seem correct that the user will operate in any capacity on a guard region when it knows it is a guard region.
But, as you say, we then won't be able to collapse a large region in one go and will have to do multiple madvise() calls to prevent overlapping with a guard region. So I agree with you.
Quoted 6 lines
I really don't understand why we would do that? You just skip over what you can't collapse right?
There's no reason at all to assume that overlapping regions here matter, we can't predict how users will use this.
True.
Quoted 9 lines
As Lance says, it's best effort. And also note we already do this with UFFD WP. And note this is also a non-present, PTE marker.
And also this would change existing behaviour which treats this as a swap entry then just fails later down the line right?
So yeah I don't agree, I think it's fine as is, unless I'm missing something here.
Thanks for your explanation!
Quoted 2 lines
Cheers, Lorenzo
Re: [PATCH mm-new v2 2/2] mm/khugepaged: abort collapse scan on guard PTEs
In reply to Lance YangQuoted 19 lines · Lance Yang
From: Lance Yang <[email protected]>
Guard PTE markers are installed via
MADV_GUARD_INSTALLto create lightweight guard regions.Currently, any collapse path (khugepaged or
MADV_COLLAPSE) will fail when encountering such a range.
MADV_COLLAPSEfails deep inside the collapse logic when trying to swap-in the special marker in__collapse_huge_page_swapin().
hpage_collapse_scan_pmd()
`-collapse_huge_page()`- __collapse_huge_page_swapin() -> fails!khugepaged's behavior is slightly different due to its
max_ptes_swaplimit (default 64). It won't fail as deep, but it will still needlessly scan up to 64 swap entries before bailing out.
Quoted 8 lines
IMHO, we can and should detect this much earlier.
This patch adds a check directly inside the PTE scan loop. If a guard marker is found, the scan is aborted immediately with
SCAN_PTE_NON_PRESENT, avoiding wasted work.Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>
LGTM, so:
Quoted 28 lines
--- mm/khugepaged.c | 10 ++++++++++ 1 file changed, 10 insertions(+)mm/khugepaged.c +10
-1@@ -1306,6 +1306,16 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, result =SCAN_PTE_UFFD_WP ; goto out_unmap; } +/* +* Guard PTE markers are installed by +* MADV_GUARD_INSTALL. Any collapse path must +* not touch them, so abort the scan immediately +* if one is found. +*/ +if (is_guard_pte_marker (pteval)) { + result =SCAN_PTE_NON_PRESENT ; + goto out_unmap; + }continue ; }else { result =SCAN_EXCEED_SWAP_PTE ; --2.49.0
Re: [PATCH mm-new v2 2/2] mm/khugepaged: abort collapse scan on guard PTEs
In reply to Lance YangQuoted 21 lines · Lance Yang
From: Lance Yang <[email protected]>
Guard PTE markers are installed via
MADV_GUARD_INSTALLto create lightweight guard regions.Currently, any collapse path (khugepaged or
MADV_COLLAPSE) will fail when encountering such a range.
MADV_COLLAPSEfails deep inside the collapse logic when trying to swap-in the special marker in__collapse_huge_page_swapin().
hpage_collapse_scan_pmd()
`-collapse_huge_page()`- __collapse_huge_page_swapin() -> fails!khugepaged's behavior is slightly different due to its
max_ptes_swaplimit (default 64). It won't fail as deep, but it will still needlessly scan up to 64 swap entries before bailing out.IMHO, we can and should detect this much earlier.
Quoted 7 lines
This patch adds a check directly inside the PTE scan loop. If a guard marker is found, the scan is aborted immediately with
SCAN_PTE_NON_PRESENT, avoiding wasted work.Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>---
Re: [PATCH mm-new v2 2/2] mm/khugepaged: abort collapse scan on guard PTEs
In reply to Lance YangQuoted 39 lines · Lance Yang
From: Lance Yang <[email protected]>
Guard PTE markers are installed via
MADV_GUARD_INSTALLto create lightweight guard regions.Currently, any collapse path (khugepaged or
MADV_COLLAPSE) will fail when encountering such a range.
MADV_COLLAPSEfails deep inside the collapse logic when trying to swap-in the special marker in__collapse_huge_page_swapin().
hpage_collapse_scan_pmd()
`-collapse_huge_page()`- __collapse_huge_page_swapin() -> fails!khugepaged's behavior is slightly different due to its
max_ptes_swaplimit (default 64). It won't fail as deep, but it will still needlessly scan up to 64 swap entries before bailing out.IMHO, we can and should detect this much earlier.
This patch adds a check directly inside the PTE scan loop. If a guard marker is found, the scan is aborted immediately with
SCAN_PTE_NON_PRESENT, avoiding wasted work.Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>--- mm/khugepaged.c | 10 ++++++++++ 1 file changed, 10 insertions(+)mm/khugepaged.c +0
-0@@ -1306,6 +1306,16 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, result =SCAN_PTE_UFFD_WP ; goto out_unmap; }
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1318
Quoted 10 lines
mm/khugepaged.c +10
-0+/* +* Guard PTE markers are installed by +* MADV_GUARD_INSTALL. Any collapse path must +* not touch them, so abort the scan immediately +* if one is found. +*/ +if (is_guard_pte_marker (pteval)) { + result =SCAN_PTE_NON_PRESENT ; + goto out_unmap; + }
Thinking about it, this is interesting.
Essentially we track any non-swap swap entries towards khugepaged_max_ptes_swap, which is rather weird.
I think we might also run into migration entries here and hwpoison entries?
So what about just generalizing this:
mm/khugepaged.c +19 -13
@@ -1293,7 +1293,24 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,for (_address = address, _pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, _address +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (is_swap_pte (pteval)) { + +if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { + ++none_or_zero; +if (!userfaultfd_armed (vma) && + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) { +continue ; + }else { + result =SCAN_EXCEED_NONE_PTE ; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); + goto out_unmap; + } + }else if (!pte_present (pteval)) { +if (non_swap_entry (pte_to_swp_entry (pteval))) { + result =SCAN_PTE_NON_PRESENT ; + goto out_unmap; + } + ++unmapped;if (!cc->is_khugepaged || unmapped <= khugepaged_max_ptes_swap) {
@@ -1313,18 +1330,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, goto out_unmap; } } -if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { - ++none_or_zero; -if (!userfaultfd_armed (vma) && - (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { -continue ; - }else { - result =SCAN_EXCEED_NONE_PTE ; -count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); - goto out_unmap; - } - } +if (pte_uffd_wp (pteval)) {/* * Don't collapse the page if any of the small
With that, the function flow looks more similar to __collapse_huge_page_isolate(), except that we handle swap entries in there now.
And with that in place, couldn't we factor out a huge chunk of both scanning functions into some helper (passing whether swap entries are allowed or not?).
Yes, I know, refactoring khugepaged, crazy idea.
Signature · 3 lines
Cheers David / dhildenb
Lance Yang
reviewerRe: [PATCH mm-new v2 2/2] mm/khugepaged: abort collapse scan on guard PTEs
Quoted 114 lines · David Hildenbrand
Quoted 40 lines · Lance Yang
From: Lance Yang <[email protected]>
Guard PTE markers are installed via
MADV_GUARD_INSTALLto create lightweight guard regions.Currently, any collapse path (khugepaged or
MADV_COLLAPSE) will fail when encountering such a range.
MADV_COLLAPSEfails deep inside the collapse logic when trying to swap-in the special marker in__collapse_huge_page_swapin().
hpage_collapse_scan_pmd()
`-collapse_huge_page()
`-__collapse_huge_page_swapin()-> fails!khugepaged's behavior is slightly different due to its
max_ptes_swaplimit (default 64). It won't fail as deep, but it will still needlessly scan up to 64 swap entries before bailing out.IMHO, we can and should detect this much earlier.
This patch adds a check directly inside the PTE scan loop. If a guard marker is found, the scan is aborted immediately with
SCAN_PTE_NON_PRESENT, avoiding wasted work.Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>---
mm/khugepaged.c| 10 ++++++++++1 file changed, 10 insertions(+)mm/khugepaged.c +0
-0@@ -1306,6 +1306,16 @@ static int hpage_collapse_scan_pmd(structmm_struct *mm, result = SCAN_PTE_UFFD_WP; goto out_unmap; }mm/khugepaged.c static int hpage_collapse_scan_pmd(struct · line 1319
Quoted 10 lines
mm/khugepaged.c +10
-0+ /* + * Guard PTE markers are installed by + * MADV_GUARD_INSTALL. Any collapse path must + * not touch them, so abort the scan immediately + * if one is found. + */ + if (is_guard_pte_marker(pteval)) { + result = SCAN_PTE_NON_PRESENT; + goto out_unmap; + }Thinking about it, this is interesting.
Essentially we track any non-swap swap entries towards
khugepaged_max_ptes_swap, which is rather weird.I think we might also run into migration entries here and hwpoison entries?
So what about just generalizing this:
mm/khugepaged.c +18
-1@@ -1293,7 +1293,24 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, for (_address = address, _pte = pte; _pte < pte + HPAGE_PMD_NR; _pte++, _address += PAGE_SIZE) { pte_t pteval = ptep_get(_pte); - if (is_swap_pte(pteval)) { + + if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { + ++none_or_zero; + if (!userfaultfd_armed(vma) && + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) { + continue; + } else { + result = SCAN_EXCEED_NONE_PTE; + count_vm_event(THP_SCAN_EXCEED_NONE_PTE); + goto out_unmap; + } + } else if (!pte_present(pteval)) { + if (non_swap_entry(pte_to_swp_entry(pteval))) { + result = SCAN_PTE_NON_PRESENT; + goto out_unmap; + } + ++unmapped; if (!cc->is_khugepaged ||unmapped <=
khugepaged_max_ptes_swap) {mm/khugepaged.c +1
-12@@ -1313,18 +1330,7 @@ static int hpage_collapse_scan_pmd(structmm_struct *mm, goto out_unmap; } } - if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { - ++none_or_zero; - if (!userfaultfd_armed(vma) && - (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { - continue; - } else { - result = SCAN_EXCEED_NONE_PTE; - count_vm_event(THP_SCAN_EXCEED_NONE_PTE); - goto out_unmap; - } - } + if (pte_uffd_wp(pteval)) { /** Don't collapse the page if any of the small
Quoted 2 lines
With that, the function flow looks more similar to
__collapse_huge_page_isolate(), except that we handle swap entries in there now.
Ah, indeed. I like this crazy idea ;p
Quoted 4 lines
And with that in place, couldn't we factor out a huge chunk of both scanning functions into some helper (passing whether swap entries are allowed or not?).
Yes. Factoring out the common scanning logic into a new helper is a good suggestion. It would clean things up ;)
Quoted 2 lines
Yes, I know, refactoring khugepaged, crazy idea.
I'll look into that. But let's do this separately :)
Cheers,
Lance
Re: [PATCH mm-new v2 2/2] mm/khugepaged: abort collapse scan on guard PTEs
Quoted 133 lines · Lance Yang
Quoted 117 lines · David Hildenbrand
Quoted 42 lines · Lance Yang
From: Lance Yang <[email protected]>
Guard PTE markers are installed via
MADV_GUARD_INSTALLto create lightweight guard regions.Currently, any collapse path (khugepaged or
MADV_COLLAPSE) will fail when encountering such a range.
MADV_COLLAPSEfails deep inside the collapse logic when trying to swap-in the special marker in__collapse_huge_page_swapin().
hpage_collapse_scan_pmd()
`-collapse_huge_page()
`-__collapse_huge_page_swapin()-> fails!khugepaged's behavior is slightly different due to its
max_ptes_swaplimit
(default 64). It won't fail as deep, but it will still needlessly scan up to 64 swap entries before bailing out.IMHO, we can and should detect this much earlier.
This patch adds a check directly inside the PTE scan loop. If a guard marker is found, the scan is aborted immediately with
SCAN_PTE_NON_PRESENT,
avoiding wasted work.Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>--- mm/khugepaged.c | 10 ++++++++++ 1 file changed, 10 insertions(+)mm/khugepaged.c +0
-0@@ -1306,6 +1306,16 @@ static int hpage_collapse_scan_pmd(structmm_struct *mm, result = SCAN_PTE_UFFD_WP; goto out_unmap; }mm/khugepaged.c static int hpage_collapse_scan_pmd(struct · line 1319
Quoted 10 lines
mm/khugepaged.c +10
-0+ /* + * Guard PTE markers are installed by + * MADV_GUARD_INSTALL. Any collapse path must + * not touch them, so abort the scan immediately + * if one is found. + */ + if (is_guard_pte_marker(pteval)) { + result = SCAN_PTE_NON_PRESENT; + goto out_unmap; + }Thinking about it, this is interesting.
Essentially we track any non-swap swap entries towards
khugepaged_max_ptes_swap, which is rather weird.I think we might also run into migration entries here and hwpoison entries?
So what about just generalizing this:
mm/khugepaged.c +18
-1@@ -1293,7 +1293,24 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, for (_address = address, _pte = pte; _pte < pte + HPAGE_PMD_NR; _pte++, _address += PAGE_SIZE) { pte_t pteval = ptep_get(_pte); - if (is_swap_pte(pteval)) { + + if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { + ++none_or_zero; + if (!userfaultfd_armed(vma) && + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) { + continue; + } else { + result = SCAN_EXCEED_NONE_PTE; + count_vm_event(THP_SCAN_EXCEED_NONE_PTE); + goto out_unmap; + } + } else if (!pte_present(pteval)) { + if (non_swap_entry(pte_to_swp_entry(pteval))) { + result = SCAN_PTE_NON_PRESENT; + goto out_unmap; + } + ++unmapped; if (!cc->is_khugepaged ||unmapped <= khugepaged_max_ptes_swap) {mm/khugepaged.c +1
-12@@ -1313,18 +1330,7 @@ static int hpage_collapse_scan_pmd(structmm_struct *mm, goto out_unmap; } } - if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { - ++none_or_zero; - if (!userfaultfd_armed(vma) && - (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { - continue; - } else { - result = SCAN_EXCEED_NONE_PTE; - count_vm_event(THP_SCAN_EXCEED_NONE_PTE); - goto out_unmap; - } - } + if (pte_uffd_wp(pteval)) { /** Don't collapse the page if any of the small
Quoted 3 lines
With that, the function flow looks more similar to
__collapse_huge_page_isolate(),
except that we handle swap entries in there now.Ah, indeed. I like this crazy idea ;p
Quoted 6 lines
And with that in place, couldn't we factor out a huge chunk of both scanning
functions into some helper (passing whether swap entries are allowed or not?).
Quoted 7 lines
Yes. Factoring out the common scanning logic into a new helper is a good suggestion. It would clean things up ;)
Quoted 2 lines
Yes, I know, refactoring khugepaged, crazy idea.
I'll look into that. But let's do this separately :)
Right, but let's just skip any non-swap entries early in this patch instead of special-casing only guard ptes.
Signature · 3 lines
Cheers David / dhildenb
Lance Yang
reviewerRe: [PATCH mm-new v2 2/2] mm/khugepaged: abort collapse scan on guard PTEs
Quoted 142 lines · David Hildenbrand
Quoted 141 lines · Lance Yang
Quoted 121 lines · David Hildenbrand
Quoted 42 lines · Lance Yang
From: Lance Yang <[email protected]>
Guard PTE markers are installed via
MADV_GUARD_INSTALLto create lightweight guard regions.Currently, any collapse path (khugepaged or
MADV_COLLAPSE) will fail when encountering such a range.
MADV_COLLAPSEfails deep inside the collapse logic when trying to swap-in the special marker in__collapse_huge_page_swapin().
hpage_collapse_scan_pmd()
`-collapse_huge_page()
`-__collapse_huge_page_swapin()-> fails!khugepaged's behavior is slightly different due to its
max_ptes_swaplimit
(default 64). It won't fail as deep, but it will still needlessly scan up to 64 swap entries before bailing out.IMHO, we can and should detect this much earlier.
This patch adds a check directly inside the PTE scan loop. If a guard marker is found, the scan is aborted immediately with
SCAN_PTE_NON_PRESENT,
avoiding wasted work.Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>---
mm/khugepaged.c| 10 ++++++++++1 file changed, 10 insertions(+)mm/khugepaged.c +0
-0@@ -1306,6 +1306,16 @@ static int hpage_collapse_scan_pmd(structmm_struct *mm, result = SCAN_PTE_UFFD_WP; goto out_unmap; }mm/khugepaged.c static int hpage_collapse_scan_pmd(struct · line 1319
Quoted 10 lines
mm/khugepaged.c +10
-0+ /* + * Guard PTE markers are installed by + * MADV_GUARD_INSTALL. Any collapse path must + * not touch them, so abort the scan immediately + * if one is found. + */ + if (is_guard_pte_marker(pteval)) { + result = SCAN_PTE_NON_PRESENT; + goto out_unmap; + }Thinking about it, this is interesting.
Essentially we track any non-swap swap entries towards
khugepaged_max_ptes_swap, which is rather weird.I think we might also run into migration entries here and hwpoison entries?
So what about just generalizing this:
mm/khugepaged.c +18
-1@@ -1293,7 +1293,24 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, for (_address = address, _pte = pte; _pte < pte + HPAGE_PMD_NR; _pte++, _address += PAGE_SIZE) { pte_t pteval = ptep_get(_pte); - if (is_swap_pte(pteval)) { + + if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { + ++none_or_zero; + if (!userfaultfd_armed(vma) && + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) { + continue; + } else { + result = SCAN_EXCEED_NONE_PTE; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); + goto out_unmap; + } + } else if (!pte_present(pteval)) { + if (non_swap_entry(pte_to_swp_entry(pteval))) { + result = SCAN_PTE_NON_PRESENT; + goto out_unmap; + } +++unmapped;
if (!cc->is_khugepaged || unmapped <=khugepaged_max_ptes_swap) {mm/khugepaged.c +1
-12@@ -1313,18 +1330,7 @@ static int hpage_collapse_scan_pmd(structmm_struct *mm, goto out_unmap; } } - if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { - ++none_or_zero; - if (!userfaultfd_armed(vma) && - (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { - continue; - } else { - result = SCAN_EXCEED_NONE_PTE; -count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); - goto out_unmap; - } - } +if (
pte_uffd_wp(pteval)) {
/*
* Don't collapse the page if any of the smallQuoted 3 lines
With that, the function flow looks more similar to
__collapse_huge_page_isolate(),
except that we handle swap entries in there now.Ah, indeed. I like this crazy idea ;p
Quoted 6 lines
And with that in place, couldn't we factor out a huge chunk of both scanning
functions into some helper (passing whether swap entries are allowed or not?).Yes. Factoring out the common scanning logic into a new helper is a good suggestion. It would clean things up ;)
Quoted 1 line
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Yes, I know, refactoring khugepaged, crazy idea.
I'll look into that. But let's do this separately :)
Right, but let's just skip any non-swap entries early in this patch instead of special-casing only guard ptes.
Ah, right! I missed the other non-swap entries. Will rework this patch as you suggested!
Cheers,
Lance
Dev Jain
reviewerRe: [PATCH mm-new v2 2/2] mm/khugepaged: abort collapse scan on guard PTEs
In reply to David HildenbrandQuoted 114 lines · David Hildenbrand
Quoted 40 lines · Lance Yang
From: Lance Yang <[email protected]>
Guard PTE markers are installed via
MADV_GUARD_INSTALLto create lightweight guard regions.Currently, any collapse path (khugepaged or
MADV_COLLAPSE) will fail when encountering such a range.
MADV_COLLAPSEfails deep inside the collapse logic when trying to swap-in the special marker in__collapse_huge_page_swapin().
hpage_collapse_scan_pmd()
`-collapse_huge_page()
`-__collapse_huge_page_swapin()-> fails!khugepaged's behavior is slightly different due to its
max_ptes_swaplimit (default 64). It won't fail as deep, but it will still needlessly scan up to 64 swap entries before bailing out.IMHO, we can and should detect this much earlier.
This patch adds a check directly inside the PTE scan loop. If a guard marker is found, the scan is aborted immediately with
SCAN_PTE_NON_PRESENT, avoiding wasted work.Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>---
mm/khugepaged.c| 10 ++++++++++1 file changed, 10 insertions(+)mm/khugepaged.c +0
-0@@ -1306,6 +1306,16 @@ static int hpage_collapse_scan_pmd(structmm_struct *mm, result = SCAN_PTE_UFFD_WP; goto out_unmap; }mm/khugepaged.c static int hpage_collapse_scan_pmd(struct · line 1319
Quoted 10 lines
mm/khugepaged.c +10
-0+ /* + * Guard PTE markers are installed by + * MADV_GUARD_INSTALL. Any collapse path must + * not touch them, so abort the scan immediately + * if one is found. + */ + if (is_guard_pte_marker(pteval)) { + result = SCAN_PTE_NON_PRESENT; + goto out_unmap; + }Thinking about it, this is interesting.
Essentially we track any non-swap swap entries towards
khugepaged_max_ptes_swap, which is rather weird.I think we might also run into migration entries here and hwpoison entries?
So what about just generalizing this:
mm/khugepaged.c +18
-1@@ -1293,7 +1293,24 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, for (_address = address, _pte = pte; _pte < pte + HPAGE_PMD_NR; _pte++, _address += PAGE_SIZE) { pte_t pteval = ptep_get(_pte); - if (is_swap_pte(pteval)) { + + if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { + ++none_or_zero; + if (!userfaultfd_armed(vma) && + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) { + continue; + } else { + result = SCAN_EXCEED_NONE_PTE; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); + goto out_unmap; + } + } else if (!pte_present(pteval)) { + if (non_swap_entry(pte_to_swp_entry(pteval))) { + result = SCAN_PTE_NON_PRESENT; + goto out_unmap; + } + ++unmapped;if (!cc->is_khugepaged || unmapped <=
khugepaged_max_ptes_swap) {mm/khugepaged.c +1
-12@@ -1313,18 +1330,7 @@ static int hpage_collapse_scan_pmd(structmm_struct *mm, goto out_unmap; } } - if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { - ++none_or_zero; - if (!userfaultfd_armed(vma) && - (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { - continue; - } else { - result = SCAN_EXCEED_NONE_PTE; -count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); - goto out_unmap; - } - } + if (pte_uffd_wp(pteval)) { /** Don't collapse the page if any of the small
Quoted 2 lines
With that, the function flow looks more similar to
__collapse_huge_page_isolate(), except that we handle swap entries in there now.
This looks good!
Quoted 7 lines
And with that in place, couldn't we factor out a huge chunk of both scanning functions into some helper (passing whether swap entries are allowed or not?).
Yes, I know, refactoring khugepaged, crazy idea.
Zi Yan
reviewerRe: [PATCH mm-new v2 2/2] mm/khugepaged: abort collapse scan on guard PTEs
In reply to Lance YangQuoted 36 lines · Lance Yang
From: Lance Yang <[email protected]>
Guard PTE markers are installed via
MADV_GUARD_INSTALLto create lightweight guard regions.Currently, any collapse path (khugepaged or
MADV_COLLAPSE) will fail when encountering such a range.
MADV_COLLAPSEfails deep inside the collapse logic when trying to swap-in the special marker in__collapse_huge_page_swapin().
hpage_collapse_scan_pmd()
`-collapse_huge_page()`- __collapse_huge_page_swapin() -> fails!khugepaged's behavior is slightly different due to its
max_ptes_swaplimit (default 64). It won't fail as deep, but it will still needlessly scan up to 64 swap entries before bailing out.IMHO, we can and should detect this much earlier.
This patch adds a check directly inside the PTE scan loop. If a guard marker is found, the scan is aborted immediately with
SCAN_PTE_NON_PRESENT, avoiding wasted work.Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>--- mm/khugepaged.c | 10 ++++++++++ 1 file changed, 10 insertions(+)mm/khugepaged.c +0
-0
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1315
Quoted 10 lines
mm/khugepaged.c +7
-0@@ -1306,6 +1306,16 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, result =SCAN_PTE_UFFD_WP ; goto out_unmap; } +/* +* Guard PTE markers are installed by +* MADV_GUARD_INSTALL. Any collapse path must +* not touch them, so abort the scan immediately +* if one is found. +*/ +if (is_guard_pte_marker (pteval)) {
pteval is already is_swap_pte(), would is_guard_swp_entry() be a better choice here? Save one is_swap_pte() call.
Quoted 8 lines
mm/khugepaged.c +3
-0+ result =SCAN_PTE_NON_PRESENT ; + goto out_unmap; + }continue ; }else { result =SCAN_EXCEED_SWAP_PTE ;Signature · 1 line
2.49.0
Best Regards,
Yan, Zi
Lance Yang
reviewerRe: [PATCH mm-new v2 2/2] mm/khugepaged: abort collapse scan on guard PTEs
Quoted 43 lines · Zi Yan
Quoted 41 lines · Lance Yang
From: Lance Yang <[email protected]>
Guard PTE markers are installed via
MADV_GUARD_INSTALLto create lightweight guard regions.Currently, any collapse path (khugepaged or
MADV_COLLAPSE) will fail when encountering such a range.
MADV_COLLAPSEfails deep inside the collapse logic when trying to swap-in the special marker in__collapse_huge_page_swapin().
hpage_collapse_scan_pmd()
`-collapse_huge_page()`- __collapse_huge_page_swapin() -> fails!khugepaged's behavior is slightly different due to its
max_ptes_swaplimit (default 64). It won't fail as deep, but it will still needlessly scan up to 64 swap entries before bailing out.IMHO, we can and should detect this much earlier.
This patch adds a check directly inside the PTE scan loop. If a guard marker is found, the scan is aborted immediately with
SCAN_PTE_NON_PRESENT, avoiding wasted work.Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>--- mm/khugepaged.c | 10 ++++++++++ 1 file changed, 10 insertions(+)mm/khugepaged.c +2
-0@@ -1306,6 +1306,16 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, result =SCAN_PTE_UFFD_WP ; goto out_unmap; } +/* +* Guard PTE markers are installed by
Quoted 8 lines
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1315
Quoted 5 lines
mm/khugepaged.c +5
-0+* MADV_GUARD_INSTALL. Any collapse path must +* not touch them, so abort the scan immediately +* if one is found. +*/ +if (is_guard_pte_marker (pteval)) {pteval is already
is_swap_pte(), wouldis_guard_swp_entry()be a better choice here? Save oneis_swap_pte()call.
Yeah. Good spot! Will do ;)
Thanks,
Lance
Quoted 13 lines
Quoted 9 lines
mm/khugepaged.c +3
-0+ result =SCAN_PTE_NON_PRESENT ; + goto out_unmap; + }continue ; }else { result =SCAN_EXCEED_SWAP_PTE ;Signature · 1 line
2.49.0Best Regards,
Yan, Zi
Baolin Wang
reviewerRe: [PATCH mm-new v2 2/2] mm/khugepaged: abort collapse scan on guard PTEs
Quoted 47 lines · Lance Yang
Quoted 44 lines · Zi Yan
Quoted 42 lines · Lance Yang
From: Lance Yang <[email protected]>
Guard PTE markers are installed via
MADV_GUARD_INSTALLto create lightweight guard regions.Currently, any collapse path (khugepaged or
MADV_COLLAPSE) will fail when encountering such a range.
MADV_COLLAPSEfails deep inside the collapse logic when trying to swap-in the special marker in__collapse_huge_page_swapin().
hpage_collapse_scan_pmd()
`-collapse_huge_page()
`-__collapse_huge_page_swapin()-> fails!khugepaged's behavior is slightly different due to its
max_ptes_swaplimit (default 64). It won't fail as deep, but it will still needlessly scan up to 64 swap entries before bailing out.IMHO, we can and should detect this much earlier.
This patch adds a check directly inside the PTE scan loop. If a guard marker is found, the scan is aborted immediately with
SCAN_PTE_NON_PRESENT, avoiding wasted work.Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>---
mm/khugepaged.c| 10 ++++++++++1 file changed, 10 insertions(+)mm/khugepaged.c +2
-0@@ -1306,6 +1306,16 @@ static int hpage_collapse_scan_pmd(structmm_struct *mm, result = SCAN_PTE_UFFD_WP; goto out_unmap; } + /* +* Guard PTE markers are installed by
Quoted 8 lines
Quoted 8 lines
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1315
Quoted 5 lines
mm/khugepaged.c +5
-0+* MADV_GUARD_INSTALL. Any collapse path must +* not touch them, so abort the scan immediately +* if one is found. +*/ + if (is_guard_pte_marker(pteval)) {pteval is already
is_swap_pte(), wouldis_guard_swp_entry()be a better choice here? Save oneis_swap_pte()call.
Agree. Then seems we don't need to move the is_guard_pte_marker() into a header file.
Lance Yang
reviewerRe: [PATCH mm-new v2 2/2] mm/khugepaged: abort collapse scan on guard PTEs
Quoted 59 lines · Baolin Wang
Quoted 47 lines · Lance Yang
Quoted 44 lines · Zi Yan
Quoted 42 lines · Lance Yang
From: Lance Yang <[email protected]>
Guard PTE markers are installed via
MADV_GUARD_INSTALLto create lightweight guard regions.Currently, any collapse path (khugepaged or
MADV_COLLAPSE) will fail when encountering such a range.
MADV_COLLAPSEfails deep inside the collapse logic when trying to swap-in the special marker in__collapse_huge_page_swapin().
hpage_collapse_scan_pmd()
`-collapse_huge_page()
`-__collapse_huge_page_swapin()-> fails!khugepaged's behavior is slightly different due to its
max_ptes_swaplimit (default 64). It won't fail as deep, but it will still needlessly scan up to 64 swap entries before bailing out.IMHO, we can and should detect this much earlier.
This patch adds a check directly inside the PTE scan loop. If a guard marker is found, the scan is aborted immediately with
SCAN_PTE_NON_PRESENT, avoiding wasted work.Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>---
mm/khugepaged.c| 10 ++++++++++1 file changed, 10 insertions(+)mm/khugepaged.c +2
-0@@ -1306,6 +1306,16 @@ static int hpage_collapse_scan_pmd(structmm_struct *mm, result = SCAN_PTE_UFFD_WP; goto out_unmap; } + /* +* Guard PTE markers are installed by Quoted 8 lines
Quoted 8 lines
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1315
Quoted 5 lines
mm/khugepaged.c +5
-0+* MADV_GUARD_INSTALL. Any collapse path must +* not touch them, so abort the scan immediately +* if one is found. +*/ + if (is_guard_pte_marker(pteval)) {pteval is already
is_swap_pte(), wouldis_guard_swp_entry()be a better choice here? Save oneis_swap_pte()call.
Quoted 1 line
Agree. Then seems we don't need to move the
is_guard_pte_marker()into a header file.
Got it. I'll only remove the redundant check in #01 patch.