mm/khugepaged: abort collapse scan on non-swap entries
Subsystems (2)
Lance Yang
reviewerPATCH 1/1 mm/khugepaged: abort collapse scan on non-swap entries
From: Lance Yang <[email protected]>
The existing check in hpage_collapse_scan_pmd() is specific to uffd-wp markers. Other special markers (e.g., GUARD, POISONED) would not be caught early, leading to failures deeper in the swap-in logic.
hpage_collapse_scan_pmd()
`- collapse_huge_page()
`- __collapse_huge_page_swapin() -> fails!
As David suggested[1], this patch skips any such non-swap entries early. If a special marker is found, the scan is aborted immediately with the SCAN_PTE_NON_PRESENT result, as Lorenzo suggested[2], avoiding wasted work.
[1] https://lore.kernel.org/linux-mm/[email protected]
[2] https://lore.kernel.org/linux-mm/[email protected]
--- mm/khugepaged.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
mm/khugepaged.c +8 -5
@@ -1285,16 +1285,19 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, _pte++, addr += PAGE_SIZE) {pte_t pteval =ptep_get (_pte);if (is_swap_pte (pteval)) { +swp_entry_t swp =pte_to_swp_entry (pteval); ++unmapped;if (!cc->is_khugepaged || unmapped <= khugepaged_max_ptes_swap) {/* -* Always be strict with uffd-wp -* enabled swap entries. Please see -* comment below for pte_uffd_wp(). +* Always be strict with PTE markers, which are +* special non-swap entries (e.g., for UFFD_WP, +* POISONED, GUARD). We cannot collapse over +* them, so just abort the scan here. */ -if (pte_swp_uffd_wp_any (pteval)) { - result =SCAN_PTE_UFFD_WP ; +if (is_pte_marker_entry (swp) && +pte_marker_get (swp)) { + result =SCAN_PTE_NON_PRESENT ; goto out_unmap; }continue ;
Signature · 1 line
2.49.0
Quoted 40 lines · Lance Yang
From: Lance Yang <[email protected]>
The existing check in
hpage_collapse_scan_pmd()is specific to uffd-wp markers. Other special markers (e.g., GUARD, POISONED) would not be caught early, leading to failures deeper in the swap-in logic.
hpage_collapse_scan_pmd()
`-collapse_huge_page()`- __collapse_huge_page_swapin() -> fails!As David suggested[1], this patch skips any such non-swap entries early. If a special marker is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENTresult, as Lorenzo suggested[2], avoiding wasted work.[1] https://lore.kernel.org/linux-mm/[email protected]
[2] https://lore.kernel.org/linux-mm/[email protected]Suggested-by David Hildenbrand <[email protected]>Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>--- mm/khugepaged.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)mm/khugepaged.c +1
-2@@ -1285,16 +1285,19 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, _pte++, addr += PAGE_SIZE) {pte_t pteval =ptep_get (_pte);if (is_swap_pte (pteval)) { +swp_entry_t swp =pte_to_swp_entry (pteval); ++unmapped;if (!cc->is_khugepaged || unmapped <= khugepaged_max_ptes_swap) {/* -* Always be strict with uffd-wp -* enabled swap entries. Please see
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1299
Quoted 10 lines
mm/khugepaged.c +6
-3-* comment below for pte_uffd_wp(). +* Always be strict with PTE markers, which are +* special non-swap entries (e.g., for UFFD_WP, +* POISONED, GUARD). We cannot collapse over +* them, so just abort the scan here. */ -if (pte_swp_uffd_wp_any (pteval)) { - result =SCAN_PTE_UFFD_WP ; +if (is_pte_marker_entry (swp) && +pte_marker_get (swp)) {
Hm. Can we have a marker that have zero pte_marker_get()?
Quoted 7 lines
mm/khugepaged.c +1
-0+ result =SCAN_PTE_NON_PRESENT ; goto out_unmap; }continue ;Signature · 1 line
2.49.0
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Quoted 6 lines · Lance Yang
From: Lance Yang <[email protected]>
The existing check in
hpage_collapse_scan_pmd()is specific to uffd-wp markers. Other special markers (e.g., GUARD, POISONED) would not be caught early, leading to failures deeper in the swap-in logic.
Quoted 8 lines
hpage_collapse_scan_pmd()
`-collapse_huge_page()`- __collapse_huge_page_swapin() -> fails!As David suggested[1], this patch skips any such non-swap entries early. If a special marker is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENTresult, as Lorenzo suggested[2], avoiding wasted work.
Note that I suggested to skip all non-present entries except swap entries, which includes migration entries, hwpoisoned entries etc.
Signature · 3 lines
Cheers David / dhildenb
Quoted 11 lines · David Hildenbrand
Quoted 10 lines · Lance Yang
From: Lance Yang <[email protected]>
The existing check in
hpage_collapse_scan_pmd()is specific to uffd-wp markers. Other special markers (e.g., GUARD, POISONED) would not be caught early, leading to failures deeper in the swap-in logic.
hpage_collapse_scan_pmd()
`-collapse_huge_page()`- __collapse_huge_page_swapin() -> fails!
Quoted 7 lines
Quoted 4 lines
As David suggested[1], this patch skips any such non-swap entries early. If a special marker is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENTresult, as Lorenzo suggested[2], avoiding wasted work.Note that I suggested to skip all non-present entries except swap entries, which includes migration entries, hwpoisoned entries etc.
Hm. So swap in is fine, but wait for migration to complete is not? Seems odd.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Quoted 17 lines · Kiryl Shutsemau
Quoted 16 lines · David Hildenbrand
Quoted 14 lines · Lance Yang
From: Lance Yang <[email protected]>
The existing check in
hpage_collapse_scan_pmd()is specific to uffd-wp markers. Other special markers (e.g., GUARD, POISONED) would not be caught early, leading to failures deeper in the swap-in logic.
hpage_collapse_scan_pmd()
`-collapse_huge_page()`- __collapse_huge_page_swapin() -> fails!As David suggested[1], this patch skips any such non-swap entries early. If a special marker is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENTresult, as Lorenzo suggested[2], avoiding wasted work.
Quoted 4 lines
Quoted 2 lines
Note that I suggested to skip all non-present entries except swap entries, which includes migration entries, hwpoisoned entries etc.
Hm. So swap in is fine, but wait for migration to complete is not?
If so we'd have to add the logic to actually wait for migration entries, and not count them towards max swap entries.
But that's a different discussion and could be added on top of cleanly handling all non-swap entries.
Signature · 3 lines
Cheers David / dhildenb
Lance Yang
reviewerQuoted 11 lines · David Hildenbrand
Quoted 10 lines · Lance Yang
From: Lance Yang <[email protected]>
The existing check in
hpage_collapse_scan_pmd()is specific to uffd-wp markers. Other special markers (e.g., GUARD, POISONED) would not be caught early, leading to failures deeper in the swap-in logic.
hpage_collapse_scan_pmd()
`-collapse_huge_page()
`-__collapse_huge_page_swapin()-> fails!
Quoted 6 lines
Quoted 4 lines
As David suggested[1], this patch skips any such non-swap entries early. If a special marker is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENTresult, as Lorenzo suggested[2], avoiding wasted work.Note that I suggested to skip all non-present entries except swap entries, which includes migration entries, hwpoisoned entries etc.
Oops, I completely misunderstood your suggestion :(
It should be to handle all special non-present entries (migration, hwpoison, markers), not just a specific type of marker ...
How about this version, which handles all non-swap entries as you suggested?
mm/khugepaged.c +18 -14
@@ -1284,7 +1284,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,for (addr = start_addr, _pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, addr +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (is_swap_pte (pteval)) { +if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { + ++none_or_zero; +if (!userfaultfd_armed (vma) && + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) { +continue ; + }else { + result =SCAN_EXCEED_NONE_PTE ; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); + goto out_unmap; + } + }else if (!pte_present (pteval)) { +if (non_swap_entry (pte_to_swp_entry (pteval))) { + result =SCAN_PTE_NON_PRESENT ; + goto out_unmap; + } + ++unmapped;if (!cc->is_khugepaged || unmapped <= khugepaged_max_ptes_swap) {
@@ -1293,7 +1309,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,* enabled swap entries. Please see * comment below for pte_uffd_wp(). */ -if (pte_swp_uffd_wp_any (pteval)) { +if (pte_swp_uffd_wp (pteval)) { result =SCAN_PTE_UFFD_WP ; goto out_unmap; }
@@ -1304,18 +1320,6 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, goto out_unmap; } } -if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { - ++none_or_zero; -if (!userfaultfd_armed (vma) && - (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { -continue ; - }else { - result =SCAN_EXCEED_NONE_PTE ; -count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); - goto out_unmap; - } - }if (pte_uffd_wp (pteval)) {/* * Don't collapse the page if any of the small
---
Thanks,
Lance
Lance Yang
reviewerQuoted 21 lines · Lance Yang
Quoted 11 lines · David Hildenbrand
Quoted 10 lines · Lance Yang
From: Lance Yang <[email protected]>
The existing check in
hpage_collapse_scan_pmd()is specific to uffd-wp markers. Other special markers (e.g., GUARD, POISONED) would not be caught early, leading to failures deeper in the swap-in logic.
hpage_collapse_scan_pmd()
`-collapse_huge_page()
`-__collapse_huge_page_swapin()-> fails!Quoted 6 lines
Quoted 4 lines
As David suggested[1], this patch skips any such non-swap entries early. If a special marker is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENTresult, as Lorenzo suggested[2], avoiding wasted work.Note that I suggested to skip all non-present entries except swap entries, which includes migration entries, hwpoisoned entries etc.
Quoted 7 lines
Oops, I completely misunderstood your suggestion :(
It should be to handle all special non-present entries (migration, hwpoison, markers), not just a specific type of marker ...
How about this version, which handles all non-swap entries as you suggested?
@David
Gently ping ...
Quoted 65 lines
mm/khugepaged.c +17
-1@@ -1284,7 +1284,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, for (addr = start_addr, _pte = pte; _pte < pte + HPAGE_PMD_NR; _pte++, addr += PAGE_SIZE) { pte_t pteval = ptep_get(_pte); - if (is_swap_pte(pteval)) { + if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { + ++none_or_zero; + if (!userfaultfd_armed(vma) && + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) { + continue; + } else { + result = SCAN_EXCEED_NONE_PTE; + count_vm_event(THP_SCAN_EXCEED_NONE_PTE); + goto out_unmap; + } + } else if (!pte_present(pteval)) { + if (non_swap_entry(pte_to_swp_entry(pteval))) { + result = SCAN_PTE_NON_PRESENT; + goto out_unmap; + } + ++unmapped; if (!cc->is_khugepaged ||unmapped <=
khugepaged_max_ptes_swap) {mm/khugepaged.c +1
-1@@ -1293,7 +1309,7 @@ static int hpage_collapse_scan_pmd(structmm_struct *mm, * enabled swap entries. Please see * comment below for pte_uffd_wp(). */ - if (pte_swp_uffd_wp_any(pteval)) { + if (pte_swp_uffd_wp(pteval)) { result = SCAN_PTE_UFFD_WP; goto out_unmap;}
mm/khugepaged.c +0
-12@@ -1304,18 +1320,6 @@ static int hpage_collapse_scan_pmd(structmm_struct *mm, goto out_unmap; } } - if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { - ++none_or_zero; - if (!userfaultfd_armed(vma) && - (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { - continue; - } else { - result = SCAN_EXCEED_NONE_PTE; - count_vm_event(THP_SCAN_EXCEED_NONE_PTE); - goto out_unmap; - } - } if (pte_uffd_wp(pteval)) { /** Don't collapse the page if any of the small
---Thanks,
Lance
Quoted 79 lines · Lance Yang
Quoted 12 lines · David Hildenbrand
Quoted 11 lines · Lance Yang
From: Lance Yang <[email protected]>
The existing check in
hpage_collapse_scan_pmd()is specific to uffd-wp markers. Other special markers (e.g., GUARD, POISONED) would not be caught
early, leading to failures deeper in the swap-in logic.
hpage_collapse_scan_pmd()
`-collapse_huge_page()
`-__collapse_huge_page_swapin()-> fails!Quoted 7 lines
Quoted 4 lines
As David suggested[1], this patch skips any such non-swap entries early. If a special marker is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENTresult, as Lorenzo suggested[2], avoiding wasted work.Note that I suggested to skip all non-present entries except swap entries, which includes migration entries, hwpoisoned entries etc.
Oops, I completely misunderstood your suggestion :(
It should be to handle all special non-present entries (migration, hwpoison, markers), not just a specific type of marker ...
How about this version, which handles all non-swap entries as you suggested?
mm/khugepaged.c +17
-1@@ -1284,7 +1284,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, for (addr = start_addr, _pte = pte; _pte < pte + HPAGE_PMD_NR; _pte++, addr += PAGE_SIZE) {pte_t pteval =ptep_get (_pte); -if (is_swap_pte (pteval)) { +if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { + ++none_or_zero; +if (!userfaultfd_armed (vma) && + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) { +continue ; + }else { + result =SCAN_EXCEED_NONE_PTE ; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); + goto out_unmap; + } + }else if (!pte_present (pteval)) { +if (non_swap_entry (pte_to_swp_entry (pteval))) { + result =SCAN_PTE_NON_PRESENT ; + goto out_unmap; + } + ++unmapped; if (!cc->is_khugepaged ||unmapped <= khugepaged_max_ptes_swap) {mm/khugepaged.c +1
-1@@ -1293,7 +1309,7 @@ static int hpage_collapse_scan_pmd(structmm_struct *mm, * enabled swap entries. Please see * comment below for pte_uffd_wp(). */ -if (pte_swp_uffd_wp_any (pteval)) { +if (pte_swp_uffd_wp (pteval)) { result =SCAN_PTE_UFFD_WP ; goto out_unmap;}mm/khugepaged.c +0
-3@@ -1304,18 +1320,6 @@ static int hpage_collapse_scan_pmd(structmm_struct *mm, goto out_unmap; } } -if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { - ++none_or_zero; - if (!userfaultfd_armed(vma) &&
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct · line 1324
Quoted 10 lines
mm/khugepaged.c +0
-9- (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { -continue ; - }else { - result =SCAN_EXCEED_NONE_PTE ; -count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); - goto out_unmap; - } - }if (pte_uffd_wp (pteval)) {
From a quick glimpse, this should work. And as raised, we might be able to unify later the scanning with the almost-duplicated code when we do the second scan.
Signature · 3 lines
Cheers David / dhildenb
Lance Yang
reviewerQuoted 85 lines · David Hildenbrand
Quoted 84 lines · Lance Yang
Quoted 12 lines · David Hildenbrand
Quoted 11 lines · Lance Yang
From: Lance Yang <[email protected]>
The existing check in
hpage_collapse_scan_pmd()is specific to uffd-wp markers. Other special markers (e.g., GUARD, POISONED) would not be caught
early, leading to failures deeper in the swap-in logic.
hpage_collapse_scan_pmd()
`-collapse_huge_page()
`-__collapse_huge_page_swapin()-> fails!Quoted 7 lines
Quoted 4 lines
As David suggested[1], this patch skips any such non-swap entries early. If a special marker is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENTresult, as Lorenzo suggested[2], avoiding wasted work.Note that I suggested to skip all non-present entries except swap entries, which includes migration entries, hwpoisoned entries etc.
Oops, I completely misunderstood your suggestion :(
It should be to handle all special non-present entries (migration, hwpoison, markers), not just a specific type of marker ...
How about this version, which handles all non-swap entries as you suggested?
mm/khugepaged.c +17
-1@@ -1284,7 +1284,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, for (addr = start_addr, _pte = pte; _pte < pte + HPAGE_PMD_NR; _pte++, addr += PAGE_SIZE) { pte_t pteval = ptep_get(_pte); - if (is_swap_pte(pteval)) { + if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { + ++none_or_zero; + if (!userfaultfd_armed(vma) && + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) { + continue; + } else { + result = SCAN_EXCEED_NONE_PTE; + count_vm_event(THP_SCAN_EXCEED_NONE_PTE); + goto out_unmap; + } + } else if (!pte_present(pteval)) { + if (non_swap_entry(pte_to_swp_entry(pteval))) { + result = SCAN_PTE_NON_PRESENT; + goto out_unmap; + } + ++unmapped;if (!cc->is_khugepaged || unmapped <=
khugepaged_max_ptes_swap) {mm/khugepaged.c +1
-1@@ -1293,7 +1309,7 @@ static int hpage_collapse_scan_pmd(structmm_struct *mm, * enabled swap entries. Please see * comment below for pte_uffd_wp(). */ - if (pte_swp_uffd_wp_any(pteval)) { + if (pte_swp_uffd_wp(pteval)) { result = SCAN_PTE_UFFD_WP; goto out_unmap;}
mm/khugepaged.c +0
-8@@ -1304,18 +1320,6 @@ static int hpage_collapse_scan_pmd(structmm_struct *mm, goto out_unmap; } } - if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { - ++none_or_zero; - if (!userfaultfd_armed(vma) && - (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { - continue; - } else { - result = SCAN_EXCEED_NONE_PTE;
Quoted 8 lines
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1323
Quoted 5 lines
mm/khugepaged.c +0
-4- count_vm_event(THP_SCAN_EXCEED_NONE_PTE); - goto out_unmap; - } - } if (pte_uffd_wp(pteval)) {From a quick glimpse, this should work. And as raised, we might be able
to unify later the scanning with the almost-duplicated code when we do the second scan.
Sounds good! Let's get this one merged first, and I'll send a follow-up patch to unify the duplicated code as you suggested ;)