mm/khugepaged: abort collapse scan on non-swap entries
Subsystems (2)
Lance Yang
reviewerPATCH v2 1/1 mm/khugepaged: abort collapse scan on non-swap entries
From: Lance Yang <[email protected]>
Currently, special non-swap entries (like migration, hwpoison, or PTE markers) are not caught early in hpage_collapse_scan_pmd(), leading to failures deep in the swap-in logic.
hpage_collapse_scan_pmd()
`- collapse_huge_page()
`- __collapse_huge_page_swapin() -> fails!
As David suggested[1], this patch skips any such non-swap entries early. If any one is found, the scan is aborted immediately with the SCAN_PTE_NON_PRESENT result, as Lorenzo suggested[2], avoiding wasted work.
[1] https://lore.kernel.org/linux-mm/[email protected]
[2] https://lore.kernel.org/linux-mm/[email protected]
---
v1 -> v2:
- Skip all non-present entries except swap entries (per David) thanks!
- https://lore.kernel.org/linux-mm/[email protected]/
mm/khugepaged.c | 32 ++++++++++++++++++-------------- 1 file changed, 18 insertions(+), 14 deletions(-)
mm/khugepaged.c +18 -14
@@ -1284,7 +1284,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,for (addr = start_addr, _pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, addr +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (is_swap_pte (pteval)) { +if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { + ++none_or_zero; +if (!userfaultfd_armed (vma) && + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) { +continue ; + }else { + result =SCAN_EXCEED_NONE_PTE ; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); + goto out_unmap; + } + }else if (!pte_present (pteval)) { +if (non_swap_entry (pte_to_swp_entry (pteval))) { + result =SCAN_PTE_NON_PRESENT ; + goto out_unmap; + } + ++unmapped;if (!cc->is_khugepaged || unmapped <= khugepaged_max_ptes_swap) {
@@ -1293,7 +1309,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,* enabled swap entries. Please see * comment below for pte_uffd_wp(). */ -if (pte_swp_uffd_wp_any (pteval)) { +if (pte_swp_uffd_wp (pteval)) { result =SCAN_PTE_UFFD_WP ; goto out_unmap; }
@@ -1304,18 +1320,6 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, goto out_unmap; } } -if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { - ++none_or_zero; -if (!userfaultfd_armed (vma) && - (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { -continue ; - }else { - result =SCAN_EXCEED_NONE_PTE ; -count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); - goto out_unmap; - } - }if (pte_uffd_wp (pteval)) {/* * Don't collapse the page if any of the small
Signature · 1 line
2.49.0
Quoted 15 lines · Lance Yang
From: Lance Yang <[email protected]>
Currently, special non-swap entries (like migration, hwpoison, or PTE markers) are not caught early in
hpage_collapse_scan_pmd(), leading to failures deep in the swap-in logic.
hpage_collapse_scan_pmd()
`-collapse_huge_page()`- __collapse_huge_page_swapin() -> fails!As David suggested[1], this patch skips any such non-swap entries early. If any one is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENTresult, as Lorenzo suggested[2], avoiding wasted work.
Quoted 7 lines
[1] https://lore.kernel.org/linux-mm/[email protected]
[2] https://lore.kernel.org/linux-mm/[email protected]Suggested-by David Hildenbrand <[email protected]>Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>---
Could have mentioned you adjust the flow of code to resemble what we have in __collapse_huge_page_isolate().
Signature · 3 lines
Cheers David / dhildenb
Lance Yang
reviewerQuoted 19 lines · David Hildenbrand
Quoted 18 lines · Lance Yang
From: Lance Yang <[email protected]>
Currently, special non-swap entries (like migration, hwpoison, or PTE markers) are not caught early in
hpage_collapse_scan_pmd(), leading to failures deep in the swap-in logic.
hpage_collapse_scan_pmd()
`-collapse_huge_page()
`-__collapse_huge_page_swapin()-> fails!As David suggested[1], this patch skips any such non-swap entries early. If any one is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENTresult, as Lorenzo suggested[2], avoiding wasted work.[1] https://lore.kernel.org/linux-mm/7840f68e-7580-42cb- [email protected]
[2] https://lore.kernel.org/linux-mm/7df49fe7-c6b7-426a-8680- [email protected]
Quoted 6 lines
Quoted 4 lines
Suggested-by David Hildenbrand <[email protected]>Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>---
Could have mentioned you adjust the flow of code to resemble what we have in
__collapse_huge_page_isolate().
Thanks for pointing that out! Let's do that separately. I'm working on a follow-up patch that will unify the scanning with the almost-duplicated code as you suggested ;)
Quoted 2 lines
Cheers!
Quoted 34 lines · Lance Yang
From: Lance Yang <[email protected]>
Currently, special non-swap entries (like migration, hwpoison, or PTE markers) are not caught early in
hpage_collapse_scan_pmd(), leading to failures deep in the swap-in logic.
hpage_collapse_scan_pmd()
`-collapse_huge_page()`- __collapse_huge_page_swapin() -> fails!As David suggested[1], this patch skips any such non-swap entries early. If any one is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENTresult, as Lorenzo suggested[2], avoiding wasted work.[1] https://lore.kernel.org/linux-mm/[email protected]
[2] https://lore.kernel.org/linux-mm/[email protected]Suggested-by David Hildenbrand <[email protected]>Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>---
v1 -> v2:
- Skip all non-present entries except swap entries (per David) thanks!
- https://lore.kernel.org/linux-mm/[email protected]/
mm/khugepaged.c| 32 ++++++++++++++++++--------------1 file changed, 18 insertions(+), 14 deletions(-)mm/khugepaged.c +0
-0
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1287
Quoted 4 lines
mm/khugepaged.c +0
-1@@ -1284,7 +1284,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,for (addr = start_addr, _pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, addr +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (is_swap_pte (pteval)) {
It looks is_swap_pte() is mis-leading?
Quoted 21 lines
mm/khugepaged.c +17
-0+if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { + ++none_or_zero; +if (!userfaultfd_armed (vma) && + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) { +continue ; + }else { + result =SCAN_EXCEED_NONE_PTE ; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); + goto out_unmap; + } + }else if (!pte_present (pteval)) { +if (non_swap_entry (pte_to_swp_entry (pteval))) { + result =SCAN_PTE_NON_PRESENT ; + goto out_unmap; + } + ++unmapped;if (!cc->is_khugepaged || unmapped <= khugepaged_max_ptes_swap) {
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1312
Quoted 5 lines
mm/khugepaged.c +1
-1@@ -1293,7 +1309,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,* enabled swap entries. Please see * comment below for pte_uffd_wp(). */ -if (pte_swp_uffd_wp_any (pteval)) { +if (pte_swp_uffd_wp (pteval)) {
I am not sure why we want to change this. There is no description in the change log.
Would you mind giving some hint on this?
Quoted 24 lines
mm/khugepaged.c +0
-12result =SCAN_PTE_UFFD_WP ; goto out_unmap; }@@ -1304,18 +1320,6 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, goto out_unmap; } } -if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { - ++none_or_zero; -if (!userfaultfd_armed (vma) && - (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { -continue ; - }else { - result =SCAN_EXCEED_NONE_PTE ; -count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); - goto out_unmap; - } - }if (pte_uffd_wp (pteval)) {/* * Don't collapse the page if any of the small Signature · 1 line
2.49.0
Signature · 2 lines
Wei Yang Help you, Help me
Quoted 30 lines · Wei Yang
Quoted 29 lines · Lance Yang
From: Lance Yang <[email protected]>
Currently, special non-swap entries (like migration, hwpoison, or PTE markers) are not caught early in
hpage_collapse_scan_pmd(), leading to failures deep in the swap-in logic.
hpage_collapse_scan_pmd()
`-collapse_huge_page()`- __collapse_huge_page_swapin() -> fails!As David suggested[1], this patch skips any such non-swap entries early. If any one is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENTresult, as Lorenzo suggested[2], avoiding wasted work.[1] https://lore.kernel.org/linux-mm/[email protected]
[2] https://lore.kernel.org/linux-mm/[email protected]Suggested-by David Hildenbrand <[email protected]>Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>---
v1 -> v2:
- Skip all non-present entries except swap entries (per David) thanks!
- https://lore.kernel.org/linux-mm/[email protected]/
mm/khugepaged.c| 32 ++++++++++++++++++--------------1 file changed, 18 insertions(+), 14 deletions(-)
Quoted 11 lines
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1287
Quoted 9 lines
mm/khugepaged.c +0
-1@@ -1284,7 +1284,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,for (addr = start_addr, _pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, addr +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (is_swap_pte (pteval)) {It looks
is_swap_pte()is mis-leading?
Yeah, it nowadays includes non-swap entries as well.
Maybe we should consider renaming it at some point to something better.
Signature · 3 lines
Cheers David / dhildenb
Lance Yang
reviewerQuoted 30 lines · Wei Yang
Quoted 29 lines · Lance Yang
From: Lance Yang <[email protected]>
Currently, special non-swap entries (like migration, hwpoison, or PTE markers) are not caught early in
hpage_collapse_scan_pmd(), leading to failures deep in the swap-in logic.
hpage_collapse_scan_pmd()
`-collapse_huge_page()`- __collapse_huge_page_swapin() -> fails!As David suggested[1], this patch skips any such non-swap entries early. If any one is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENTresult, as Lorenzo suggested[2], avoiding wasted work.[1] https://lore.kernel.org/linux-mm/[email protected]
[2] https://lore.kernel.org/linux-mm/[email protected]Suggested-by David Hildenbrand <[email protected]>Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>---
v1 -> v2:
- Skip all non-present entries except swap entries (per David) thanks!
- https://lore.kernel.org/linux-mm/[email protected]/
mm/khugepaged.c| 32 ++++++++++++++++++--------------1 file changed, 18 insertions(+), 14 deletions(-)
Quoted 11 lines
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1287
Quoted 9 lines
mm/khugepaged.c +0
-1@@ -1284,7 +1284,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,for (addr = start_addr, _pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, addr +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (is_swap_pte (pteval)) {It looks
is_swap_pte()is mis-leading?
Hmm.. not to me, IMO. is_swap_pte() just means:
!pte_none(pte) && !pte_present(pte)
Quoted 31 lines
Quoted 22 lines
mm/khugepaged.c +17
-0+if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { + ++none_or_zero; +if (!userfaultfd_armed (vma) && + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) { +continue ; + }else { + result =SCAN_EXCEED_NONE_PTE ; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); + goto out_unmap; + } + }else if (!pte_present (pteval)) { +if (non_swap_entry (pte_to_swp_entry (pteval))) { + result =SCAN_PTE_NON_PRESENT ; + goto out_unmap; + } + ++unmapped;if (!cc->is_khugepaged || unmapped <= khugepaged_max_ptes_swap) {mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1312
Quoted 5 lines
mm/khugepaged.c +1
-1@@ -1293,7 +1309,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,* enabled swap entries. Please see * comment below for pte_uffd_wp(). */ -if (pte_swp_uffd_wp_any (pteval)) { +if (pte_swp_uffd_wp (pteval)) {I am not sure why we want to change this. There is no description in the change log.
Quoted 1 line
Would you mind giving some hint on this?
The reason is that pte_swp_uffd_wp_any(pte) is broader than what we need :)
static inline bool pte_swp_uffd_wp_any(pte_t pte)
{
#ifdef CONFIG_PTE_MARKER_UFFD_WP if (!is_swap_pte(pte)) return false; if (pte_swp_uffd_wp(pte)) return true; if (pte_marker_uffd_wp(pte)) return true; #endif return false; }
In the context within hpage_collapse_scan_pmd(), we are already inside an is_swap_pte() block, and we have just handled all non-swap entries (which would include pte_marker_uffd_wp()).
So we only need to check if the swap entry itself is write-protected for userfaultfd ;)
Hope that explains it. I skipped it in the changelog as it's a tiny cleanup ...
Thanks,
Lance
Quoted 26 lines
Quoted 25 lines
mm/khugepaged.c +0
-12result =SCAN_PTE_UFFD_WP ; goto out_unmap; }@@ -1304,18 +1320,6 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, goto out_unmap; } } -if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { - ++none_or_zero; -if (!userfaultfd_armed (vma) && - (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { -continue ; - }else { - result =SCAN_EXCEED_NONE_PTE ; -count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); - goto out_unmap; - } - }if (pte_uffd_wp (pteval)) {/* * Don't collapse the page if any of the small Signature · 1 line
2.49.0
Wei Yang
Reviewed-byQuoted 104 lines · Lance Yang
Quoted 30 lines · Wei Yang
Quoted 29 lines · Lance Yang
From: Lance Yang <[email protected]>
Currently, special non-swap entries (like migration, hwpoison, or PTE markers) are not caught early in
hpage_collapse_scan_pmd(), leading to failures deep in the swap-in logic.
hpage_collapse_scan_pmd()
`-collapse_huge_page()`- __collapse_huge_page_swapin() -> fails!As David suggested[1], this patch skips any such non-swap entries early. If any one is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENTresult, as Lorenzo suggested[2], avoiding wasted work.[1] https://lore.kernel.org/linux-mm/[email protected]
[2] https://lore.kernel.org/linux-mm/[email protected]Suggested-by David Hildenbrand <[email protected]>Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>---
v1 -> v2:
- Skip all non-present entries except swap entries (per David) thanks!
- https://lore.kernel.org/linux-mm/[email protected]/
mm/khugepaged.c| 32 ++++++++++++++++++--------------1 file changed, 18 insertions(+), 14 deletions(-)Quoted 11 lines
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1287
Quoted 9 lines
mm/khugepaged.c +0
-1@@ -1284,7 +1284,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,for (addr = start_addr, _pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, addr +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (is_swap_pte (pteval)) {It looks
is_swap_pte()is mis-leading?Hmm.. not to me, IMO.
is_swap_pte()just means:!pte_none(pte) && !pte_present(pte)
Quoted 31 lines
Quoted 22 lines
mm/khugepaged.c +17
-0+if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { + ++none_or_zero; +if (!userfaultfd_armed (vma) && + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) { +continue ; + }else { + result =SCAN_EXCEED_NONE_PTE ; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); + goto out_unmap; + } + }else if (!pte_present (pteval)) { +if (non_swap_entry (pte_to_swp_entry (pteval))) { + result =SCAN_PTE_NON_PRESENT ; + goto out_unmap; + } + ++unmapped;if (!cc->is_khugepaged || unmapped <= khugepaged_max_ptes_swap) {mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1312
Quoted 5 lines
mm/khugepaged.c +1
-1@@ -1293,7 +1309,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,* enabled swap entries. Please see * comment below for pte_uffd_wp(). */ -if (pte_swp_uffd_wp_any (pteval)) { +if (pte_swp_uffd_wp (pteval)) {I am not sure why we want to change this. There is no description in the change log.
Quoted 1 line
Would you mind giving some hint on this?
The reason is that
pte_swp_uffd_wp_any(pte)is broader than what we need :)static inline bool
pte_swp_uffd_wp_any(pte_t pte)
{#ifdef CONFIG_PTE_MARKER_UFFD_WP if (!is_swap_pte(pte)) return false; if (pte_swp_uffd_wp(pte)) return true; if (pte_marker_uffd_wp(pte)) return true; #endif return false; }In the context within
hpage_collapse_scan_pmd(), we are already inside anis_swap_pte()block, and we have just handled all non-swap entries (which would includepte_marker_uffd_wp()).
Quoted 5 lines
So we only need to check if the swap entry itself is write-protected for userfaultfd ;)
Hope that explains it. I skipped it in the changelog as it's a tiny cleanup ...
Thanks, I got it.
Generally, looks good to me.
Signature · 2 lines
Wei Yang Help you, Help me
Quoted 45 lines · Lance Yang
Quoted 30 lines · Wei Yang
Quoted 29 lines · Lance Yang
From: Lance Yang <[email protected]>
Currently, special non-swap entries (like migration, hwpoison, or PTE markers) are not caught early in
hpage_collapse_scan_pmd(), leading to failures deep in the swap-in logic.
hpage_collapse_scan_pmd()
`-collapse_huge_page()`- __collapse_huge_page_swapin() -> fails!As David suggested[1], this patch skips any such non-swap entries early. If any one is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENTresult, as Lorenzo suggested[2], avoiding wasted work.[1] https://lore.kernel.org/linux-mm/[email protected]
[2] https://lore.kernel.org/linux-mm/[email protected]Suggested-by David Hildenbrand <[email protected]>Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>---
v1 -> v2:
- Skip all non-present entries except swap entries (per David) thanks!
- https://lore.kernel.org/linux-mm/[email protected]/
mm/khugepaged.c| 32 ++++++++++++++++++--------------1 file changed, 18 insertions(+), 14 deletions(-)Quoted 11 lines
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1287
Quoted 9 lines
mm/khugepaged.c +0
-1@@ -1284,7 +1284,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,for (addr = start_addr, _pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, addr +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (is_swap_pte (pteval)) {It looks
is_swap_pte()is mis-leading?
Quoted 4 lines
Hmm.. not to me, IMO.
is_swap_pte()just means:!pte_none(pte) && !pte_present(pte)
Maybe it has some reason.
I took another look into __collapse_huge_page_swapin(), which just check is_swap_pte() before do_swap_page().
We have filtered non-swap entries in hpage_collapse_scan_pmd(), but we drop mmap lock before isolation. This looks we may have a chance to get non-swap entry.
Do you think it is reasonable to add a non_swap_entry() check before do_swap_page()?
Signature · 2 lines
Wei Yang Help you, Help me
Lance Yang
reviewerQuoted 51 lines · Wei Yang
Quoted 45 lines · Lance Yang
Quoted 30 lines · Wei Yang
Quoted 29 lines · Lance Yang
From: Lance Yang <[email protected]>
Currently, special non-swap entries (like migration, hwpoison, or PTE markers) are not caught early in
hpage_collapse_scan_pmd(), leading to failures deep in the swap-in logic.
hpage_collapse_scan_pmd()
`-collapse_huge_page()`- __collapse_huge_page_swapin() -> fails!As David suggested[1], this patch skips any such non-swap entries early. If any one is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENTresult, as Lorenzo suggested[2], avoiding wasted work.[1] https://lore.kernel.org/linux-mm/[email protected]
[2] https://lore.kernel.org/linux-mm/[email protected]Suggested-by David Hildenbrand <[email protected]>Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>---
v1 -> v2:
- Skip all non-present entries except swap entries (per David) thanks!
- https://lore.kernel.org/linux-mm/[email protected]/
mm/khugepaged.c| 32 ++++++++++++++++++--------------1 file changed, 18 insertions(+), 14 deletions(-)Quoted 11 lines
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1287
Quoted 9 lines
mm/khugepaged.c +0
-1@@ -1284,7 +1284,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,for (addr = start_addr, _pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, addr +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (is_swap_pte (pteval)) {It looks
is_swap_pte()is mis-leading?Quoted 4 lines
Hmm.. not to me, IMO.
is_swap_pte()just means:!pte_none(pte) && !pte_present(pte)
Quoted 8 lines
Maybe it has some reason.
I took another look into
__collapse_huge_page_swapin(), which just checkis_swap_pte()beforedo_swap_page().We have filtered non-swap entries in
hpage_collapse_scan_pmd(), but we drop mmap lock before isolation. This looks we may have a chance to get non-swap entry.
Thanks for pointing that out!
Yep, there is a theoretical window between dropping the mmap lock after the initial scan and re-acquiring it for isolation.
Quoted 3 lines
Do you think it is reasonable to add a
non_swap_entry()check beforedo_swap_page()?
However, that seems unlikely in practice. IMHO, the early check in hpage_collapse_scan_pmd() is sufficient for now, so I'd prefer to keep it as-is :)
Quoted 51 lines · Lance Yang
Quoted 48 lines · Wei Yang
Quoted 47 lines · Lance Yang
Quoted 30 lines · Wei Yang
Quoted 29 lines · Lance Yang
From: Lance Yang <[email protected]>
Currently, special non-swap entries (like migration, hwpoison, or PTE markers) are not caught early in
hpage_collapse_scan_pmd(), leading to failures deep in the swap-in logic.
hpage_collapse_scan_pmd()
`-collapse_huge_page()`- __collapse_huge_page_swapin() -> fails!As David suggested[1], this patch skips any such non-swap entries early. If any one is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENTresult, as Lorenzo suggested[2], avoiding wasted work.[1] https://lore.kernel.org/linux-mm/[email protected]
[2] https://lore.kernel.org/linux-mm/[email protected]Suggested-by David Hildenbrand <[email protected]>Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>---
v1 -> v2:
- Skip all non-present entries except swap entries (per David) thanks!
- https://lore.kernel.org/linux-mm/[email protected]/
mm/khugepaged.c| 32 ++++++++++++++++++--------------1 file changed, 18 insertions(+), 14 deletions(-)Quoted 11 lines
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1287
Quoted 9 lines
mm/khugepaged.c +0
-1@@ -1284,7 +1284,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,for (addr = start_addr, _pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, addr +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (is_swap_pte (pteval)) {It looks
is_swap_pte()is mis-leading?Hmm.. not to me, IMO.
is_swap_pte()just means:
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
!pte_none(pte) && !pte_present(pte)
Maybe it has some reason.
I took another look into
__collapse_huge_page_swapin(), which just checkis_swap_pte()beforedo_swap_page().
Thanks for pointing that out.
A function that is called __collapse_huge_page_swapin() and documented to "Bring missing pages in from swap" will handle other types as well.
Unbelievable horrible.
So let's think this through so we can document it in the changelog properly.
We could have currently ended up in do_swap_page() with
(1) Migration entries. We would have waited.
-> Maybe worth it to wait, maybe not. I suspect we don't stumble into
that frequently such that we don't care. We could always unlock this separately later.
(2) Device-exclusive entries. We would have converted to non-exclusive.
-> See make_device_exclusive(), we cannot tolerate PMD entries and have
to split them through FOLL_SPLIT_PMD. As popped up during a recent discussion, collapsing here is actually counter-productive, because the next conversion will PTE-map it again. (until recently, it would not have worked with large folios at all IIRC).
-> Ok to not collapse.
(3) Device-private entries. We would have migrated to RAM.
-> Device-private still does not support THPs, so collapsing right now just means that the next device access would split the folio again.
-> Ok to not collapse.
(4) HWPoison entries
-> Cannot collapse
(5) Markers
-> Cannot collapse
I suggest we add that in some form to the patch description, stating that we can unlock later what we really need, and not account it towards max_swap_ptes.
Quoted 11 lines
Quoted 4 lines
We have filtered non-swap entries in
hpage_collapse_scan_pmd(), but we drop mmap lock before isolation. This looks we may have a chance to get non-swap entry.Thanks for pointing that out!
Yep, there is a theoretical window between dropping the mmap lock after the initial scan and re-acquiring it for isolation.
Quoted 1 line
Quoted 6 lines
Quoted 2 lines
Do you think it is reasonable to add a
non_swap_entry()check beforedo_swap_page()?However, that seems unlikely in practice. IMHO, the early check in
hpage_collapse_scan_pmd()is sufficient for now, so I'd prefer to keep it as-is :)
I think we really should add that check, as per reasoning above.
I was looking into some possible races with uffd-wp being set before we enter do_swap_page(), but I think it might be okay (although very confusing).
Signature · 3 lines
Cheers David / dhildenb
Lance Yang
reviewerQuoted 99 lines · David Hildenbrand
Quoted 54 lines · Lance Yang
Quoted 51 lines · Wei Yang
Quoted 50 lines · Lance Yang
Quoted 32 lines · Wei Yang
Quoted 31 lines · Lance Yang
From: Lance Yang <[email protected]>
Currently, special non-swap entries (like migration, hwpoison, or PTE markers) are not caught early in
hpage_collapse_scan_pmd(), leading to failures deep in the swap-in logic.
hpage_collapse_scan_pmd()
`-collapse_huge_page()
`-__collapse_huge_page_swapin()-> fails!As David suggested[1], this patch skips any such non-swap entries early. If any one is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENTresult, as Lorenzo suggested[2], avoiding wasted work.[1] https://lore.kernel.org/linux-mm/7840f68e-7580-42cb- [email protected]
[2] https://lore.kernel.org/linux-mm/7df49fe7-c6b7-426a-8680- [email protected]Suggested-by David Hildenbrand <[email protected]>Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>---
v1 -> v2:
- Skip all non-present entries except swap entries (per David) thanks!
- https://lore.kernel.org/linux-mm/20250924100207.28332-1-
[email protected]/
mm/khugepaged.c| 32 ++++++++++++++++++--------------1 file changed, 18 insertions(+), 14 deletions(-)Quoted 12 lines
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct · line 1288
Quoted 10 lines
mm/khugepaged.c +0
-1@@ -1284,7 +1284,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, for (addr = start_addr, _pte = pte; _pte < pte + HPAGE_PMD_NR; _pte++, addr += PAGE_SIZE) { pte_t pteval = ptep_get(_pte); - if (is_swap_pte(pteval)) {It looks
is_swap_pte()is mis-leading?Hmm.. not to me, IMO.
is_swap_pte()just means:Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
!pte_none(pte) && !pte_present(pte)
Maybe it has some reason.
I took another look into
__collapse_huge_page_swapin(), which just checkis_swap_pte()beforedo_swap_page().Thanks for pointing that out.
A function that is called
__collapse_huge_page_swapin()and documented to "Bring missing pages in from swap" will handle other types as well.Unbelievable horrible.
So let's think this through so we can document it in the changelog properly.
We could have currently ended up in
do_swap_page()with(1) Migration entries. We would have waited.
-> Maybe worth it to wait, maybe not. I suspect we don't stumble into
that frequently such that we don't care. We could always unlock this separately later.(2) Device-exclusive entries. We would have converted to non-exclusive.
-> See
make_device_exclusive(), we cannot tolerate PMD entries and have
to split them throughFOLL_SPLIT_PMD. As popped up during a recent discussion, collapsing here is actually counter-productive, because the next conversion will PTE-map it again. (until recently, it would not have worked with large folios at all IIRC).-> Ok to not collapse.
(3) Device-private entries. We would have migrated to RAM.
-> Device-private still does not support THPs, so collapsing right now just means that the next device access would split the folio again.
-> Ok to not collapse.
(4) HWPoison entries
Quoted 8 lines
-> Cannot collapse
(5) Markers
-> Cannot collapse
I suggest we add that in some form to the patch description, stating that we can unlock later what we really need, and not account it towards
max_swap_ptes.
Cool!
I'll take a closer look and adjust the patch description accordingly ;)
Thanks a lot for the lesson!
Quoted 15 lines
Quoted 15 lines
Quoted 5 lines
We have filtered non-swap entries in
hpage_collapse_scan_pmd(), but we drop mmap lock before isolation. This looks we may have a chance to get non-swap entry.Thanks for pointing that out!
Yep, there is a theoretical window between dropping the mmap lock after the initial scan and re-acquiring it for isolation.
Quoted 3 lines
Do you think it is reasonable to add a
non_swap_entry()check beforedo_swap_page()?
Quoted 7 lines
Quoted 3 lines
However, that seems unlikely in practice. IMHO, the early check in
hpage_collapse_scan_pmd()is sufficient for now, so I'd prefer to keep it as-is :)I think we really should add that check, as per reasoning above.
I was looking into some possible races with uffd-wp being set before we enter
do_swap_page(), but I think it might be okay (although very confusing).
Ah, I see ;p
@Wei could you send a patch to add the non_swap_entry() check there?
Lance Yang
reviewerQuoted 122 lines · David Hildenbrand
Quoted 54 lines · Lance Yang
Quoted 51 lines · Wei Yang
Quoted 50 lines · Lance Yang
Quoted 32 lines · Wei Yang
Quoted 31 lines · Lance Yang
From: Lance Yang <[email protected]>
Currently, special non-swap entries (like migration, hwpoison, or PTE markers) are not caught early in
hpage_collapse_scan_pmd(), leading to failures deep in the swap-in logic.
hpage_collapse_scan_pmd()
`-collapse_huge_page()
`-__collapse_huge_page_swapin()-> fails!As David suggested[1], this patch skips any such non-swap entries early. If any one is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENTresult, as Lorenzo suggested[2], avoiding wasted work.[1] https://lore.kernel.org/linux-mm/7840f68e-7580-42cb- [email protected]
[2] https://lore.kernel.org/linux-mm/7df49fe7-c6b7-426a-8680- [email protected]Suggested-by David Hildenbrand <[email protected]>Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>---
v1 -> v2:
- Skip all non-present entries except swap entries (per David) thanks!
- https://lore.kernel.org/linux-mm/20250924100207.28332-1-
[email protected]/
mm/khugepaged.c| 32 ++++++++++++++++++--------------1 file changed, 18 insertions(+), 14 deletions(-)Quoted 12 lines
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct · line 1288
Quoted 10 lines
mm/khugepaged.c +0
-1@@ -1284,7 +1284,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, for (addr = start_addr, _pte = pte; _pte < pte + HPAGE_PMD_NR; _pte++, addr += PAGE_SIZE) { pte_t pteval = ptep_get(_pte); - if (is_swap_pte(pteval)) {It looks
is_swap_pte()is mis-leading?Hmm.. not to me, IMO.
is_swap_pte()just means:Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
!pte_none(pte) && !pte_present(pte)
Maybe it has some reason.
I took another look into
__collapse_huge_page_swapin(), which just checkis_swap_pte()beforedo_swap_page().Thanks for pointing that out.
A function that is called
__collapse_huge_page_swapin()and documented to "Bring missing pages in from swap" will handle other types as well.Unbelievable horrible.
So let's think this through so we can document it in the changelog properly.
We could have currently ended up in
do_swap_page()with(1) Migration entries. We would have waited.
-> Maybe worth it to wait, maybe not. I suspect we don't stumble into
that frequently such that we don't care. We could always unlock this separately later.(2) Device-exclusive entries. We would have converted to non-exclusive.
-> See
make_device_exclusive(), we cannot tolerate PMD entries and have
to split them throughFOLL_SPLIT_PMD. As popped up during a recent discussion, collapsing here is actually counter-productive, because the next conversion will PTE-map it again. (until recently, it would not have worked with large folios at all IIRC).-> Ok to not collapse.
(3) Device-private entries. We would have migrated to RAM.
-> Device-private still does not support THPs, so collapsing right now just means that the next device access would split the folio again.
-> Ok to not collapse.
(4) HWPoison entries
-> Cannot collapse
(5) Markers
-> Cannot collapse
I suggest we add that in some form to the patch description, stating that we can unlock later what we really need, and not account it towards
max_swap_ptes.Quoted 14 lines
Quoted 4 lines
We have filtered non-swap entries in
hpage_collapse_scan_pmd(), but we drop mmap lock before isolation. This looks we may have a chance to get non-swap entry.Thanks for pointing that out!
Yep, there is a theoretical window between dropping the mmap lock after the initial scan and re-acquiring it for isolation.
Quoted 3 lines
Do you think it is reasonable to add a
non_swap_entry()check beforedo_swap_page()?
Quoted 7 lines
Quoted 3 lines
However, that seems unlikely in practice. IMHO, the early check in
hpage_collapse_scan_pmd()is sufficient for now, so I'd prefer to keep it as-is :)I think we really should add that check, as per reasoning above.
I was looking into some possible races with uffd-wp being set before we enter
do_swap_page(), but I think it might be okay (although very confusing).
How about the version below?
```
Currently, special non-swap entries (like PTE markers) are not caught
early in hpage_collapse_scan_pmd(), leading to failures deep in the
swap-in logic.
A function that is called __collapse_huge_page_swapin() and documented
to "Bring missing pages in from swap" will handle other types as well.
As analyzed by David[1], we could have ended up with the following
entry types right before do_swap_page():
(1) Migration entries. We would have waited.
-> Maybe worth it to wait, maybe not. We suspect we don't stumble
into that frequently such that we don't care. We could always
unlock this separately later.
(2) Device-exclusive entries. We would have converted to non-exclusive.
-> See make_device_exclusive(), we cannot tolerate PMD entries and
have to split them through FOLL_SPLIT_PMD. As popped up during
a recent discussion, collapsing here is actually
counter-productive, because the next conversion will PTE-map
it again.
-> Ok to not collapse.
(3) Device-private entries. We would have migrated to RAM.
-> Device-private still does not support THPs, so collapsing right
now just means that the next device access would split the
folio again.
-> Ok to not collapse.
(4) HWPoison entries
-> Cannot collapse
(5) Markers
-> Cannot collapse
First, this patch adds an early check for these non-swap entries. If
any one is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENT result, as Lorenzo suggested[2], avoiding wasted
work.
Second, as Wei pointed out[3], we may have a chance to get a non-swap
entry, since we will drop and re-acquire the mmap lock before
__collapse_huge_page_swapin(). To handle this, we also add a
non_swap_entry() check there.
Note that we can unlock later what we really need, and not account it
towards max_swap_ptes.
[1] https://lore.kernel.org/linux-mm/[email protected]
[2] https://lore.kernel.org/linux-mm/[email protected]
[3] https://lore.kernel.org/linux-mm/20251005010511.ysek2nqojebqngf3@master
```I also think it makes sense to fold the change that adds the non_swap_entry() check in __collapse_huge_page_swapin() into this patch, rather than creating a new patch just for that :)
Hmmm... one thing I'm not sure about: regarding the uffd-wp race you mentioned, is the pte_swp_uffd_wp() check needed after non_swap_entry()? It seems like it might not be ...
```
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index f4f57ba69d72..bec3e268dc76 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -1020,6 +1020,11 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm,
if (!is_swap_pte(vmf.orig_pte))
continue;
+ if (non_swap_entry(pte_to_swp_entry(vmf.orig_pte))) {
+ result = SCAN_PTE_NON_PRESENT;
+ goto out;
+ }
+
vmf.pte = pte;
vmf.ptl = ptl;
ret = do_swap_page(&vmf);
```@David does that sound good to you?
Quoted 4 lines · Lance Yang
Quoted 1 line · David Hildenbrand
On 05.10.25 04:12, Lance Yang wrote:
[...]
Quoted 61 lines
Quoted 4 lines
I was looking into some possible races with uffd-wp being set before we enter
do_swap_page(), but I think it might be okay (although very confusing).How about the version below?
``` Currently, special non-swap entries (like PTE markers) are not caught early in hpage_collapse_scan_pmd(), leading to failures deep in the swap-in logic. A function that is called __collapse_huge_page_swapin() and documented to "Bring missing pages in from swap" will handle other types as well. As analyzed by David[1], we could have ended up with the following entry types right before do_swap_page(): (1) Migration entries. We would have waited. -> Maybe worth it to wait, maybe not. We suspect we don't stumble into that frequently such that we don't care. We could always unlock this separately later. (2) Device-exclusive entries. We would have converted to non-exclusive. -> See make_device_exclusive(), we cannot tolerate PMD entries and have to split them through FOLL_SPLIT_PMD. As popped up during a recent discussion, collapsing here is actually counter-productive, because the next conversion will PTE-map it again. -> Ok to not collapse. (3) Device-private entries. We would have migrated to RAM. -> Device-private still does not support THPs, so collapsing right now just means that the next device access would split the folio again. -> Ok to not collapse. (4) HWPoison entries -> Cannot collapse (5) Markers -> Cannot collapse First, this patch adds an early check for these non-swap entries. If any one is found, the scan is aborted immediately with the SCAN_PTE_NON_PRESENT result, as Lorenzo suggested[2], avoiding wasted work. Second, as Wei pointed out[3], we may have a chance to get a non-swap entry, since we will drop and re-acquire the mmap lock before __collapse_huge_page_swapin(). To handle this, we also add a non_swap_entry() check there. Note that we can unlock later what we really need, and not account it towards max_swap_ptes. [1] https://lore.kernel.org/linux-mm/[email protected] [2] https://lore.kernel.org/linux-mm/[email protected] [3] https://lore.kernel.org/linux-mm/20251005010511.ysek2nqojebqngf3@master ```
Quoted 4 lines
I also think it makes sense to fold the change that adds the
non_swap_entry()check in__collapse_huge_page_swapin()into this patch, rather than creating a new patch just for that :)
Agree.
Quoted 25 lines
Hmmm... one thing I'm not sure about: regarding the uffd-wp race you mentioned, is the
pte_swp_uffd_wp()check needed afternon_swap_entry()? It seems like it might not be ...```
mm/khugepaged.c +5
-0@@ -1020,6 +1020,11 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, if (!is_swap_pte(vmf.orig_pte)) continue; +if (non_swap_entry (pte_to_swp_entry (vmf.orig_pte))) { + result =SCAN_PTE_NON_PRESENT ; + goto out; + } + vmf.pte = pte; vmf.ptl = ptl;ret = do_swap_page(&vmf);```
@David does that sound good to you?
Signature · 2 lines
Wei Yang Help you, Help me
Quoted 8 lines
Second, as Wei pointed out[3], we may have a chance to get a non-swap entry, since we will drop and re-acquire the mmap lock before
__collapse_huge_page_swapin(). To handle this, we also add anon_swap_entry()check there.Note that we can unlock later what we really need, and not account it towards
max_swap_ptes.
Quoted 7 lines
[1]
https://lore.kernel.org/linux-mm/[email protected]
[2]
https://lore.kernel.org/linux-mm/[email protected]
[3] https://lore.kernel.org/linux-mm/20251005010511.ysek2nqojebqngf3@master
```
I replied to the new version.
Quoted 8 lines
I also think it makes sense to fold the change that adds the
non_swap_entry()check in__collapse_huge_page_swapin()into this patch, rather than creating a new patch just for that :)Hmmm... one thing I'm not sure about: regarding the uffd-wp race you mentioned, is the
pte_swp_uffd_wp()check needed afternon_swap_entry()? It seems like it might not be ...
If we check non_swap_entry(), there is no need to check for other markers, agreed.
Signature · 3 lines
Cheers David / dhildenb
Quoted 40 lines · Lance Yang
From: Lance Yang <[email protected]>
Currently, special non-swap entries (like migration, hwpoison, or PTE markers) are not caught early in
hpage_collapse_scan_pmd(), leading to failures deep in the swap-in logic.
hpage_collapse_scan_pmd()
`-collapse_huge_page()`- __collapse_huge_page_swapin() -> fails!As David suggested[1], this patch skips any such non-swap entries early. If any one is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENTresult, as Lorenzo suggested[2], avoiding wasted work.[1] https://lore.kernel.org/linux-mm/[email protected]
[2] https://lore.kernel.org/linux-mm/[email protected]Suggested-by David Hildenbrand <[email protected]>Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>---
v1 -> v2:
- Skip all non-present entries except swap entries (per David) thanks!
- https://lore.kernel.org/linux-mm/[email protected]/mm/khugepaged.c | 32 ++++++++++++++++++-------------- 1 file changed, 18 insertions(+), 14 deletions(-)mm/khugepaged.c +2
-1@@ -1284,7 +1284,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,for (addr = start_addr, _pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, addr +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (is_swap_pte (pteval)) { +if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { + ++none_or_zero;
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1298
Quoted 10 lines
mm/khugepaged.c +10
-0+if (!userfaultfd_armed (vma) && + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) { +continue ; + }else { + result =SCAN_EXCEED_NONE_PTE ; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); + goto out_unmap; + } + }else if (!pte_present (pteval)) {
If you are trying to merge this with the _isolate() conditions, we can do a micro-optimization here - is_swap_pte, (pte_none && is_zero_pfn), and pte_uffd_wp are disjoint conditions, so we can use if-else-if-else-if to write them.
Quoted 9 lines
mm/khugepaged.c +5
-0+if (non_swap_entry (pte_to_swp_entry (pteval))) { + result =SCAN_PTE_NON_PRESENT ; + goto out_unmap; + } + ++unmapped;if (!cc->is_khugepaged || unmapped <= khugepaged_max_ptes_swap) {
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1313
Quoted 6 lines
mm/khugepaged.c +1
-1@@ -1293,7 +1309,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,* enabled swap entries. Please see * comment below for pte_uffd_wp(). */ -if (pte_swp_uffd_wp_any (pteval)) { +if (pte_swp_uffd_wp (pteval)) { result =SCAN_PTE_UFFD_WP ;
Could have mentioned in the changelog "while at it, convert pte_swp_uffd_wp_any to pte_swp_uffd_wp since we are in the swap pte branch".
Quoted 11 lines
mm/khugepaged.c +0
-5goto out_unmap; }@@ -1304,18 +1320,6 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, goto out_unmap; } } -if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { - ++none_or_zero; -if (!userfaultfd_armed (vma) && - (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) {
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1325
Quoted 10 lines
mm/khugepaged.c +0
-7-continue ; - }else { - result =SCAN_EXCEED_NONE_PTE ; -count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); - goto out_unmap; - } - }if (pte_uffd_wp (pteval)) {/* * Don't collapse the page if any of the small
Otherwise LGTM
Lance Yang
reviewerQuoted 49 lines · Dev Jain
Quoted 47 lines · Lance Yang
From: Lance Yang <[email protected]>
Currently, special non-swap entries (like migration, hwpoison, or PTE markers) are not caught early in
hpage_collapse_scan_pmd(), leading to failures deep in the swap-in logic.
hpage_collapse_scan_pmd()
`-collapse_huge_page()
`-__collapse_huge_page_swapin()-> fails!As David suggested[1], this patch skips any such non-swap entries early. If any one is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENTresult, as Lorenzo suggested[2], avoiding wasted work.[1] https://lore.kernel.org/linux-mm/7840f68e-7580-42cb- [email protected]
[2] https://lore.kernel.org/linux-mm/7df49fe7-c6b7-426a-8680- [email protected]Suggested-by David Hildenbrand <[email protected]>Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>---
v1 -> v2:
- Skip all non-present entries except swap entries (per David) thanks!
- https://lore.kernel.org/linux-mm/20250924100207.28332-1- [email protected]/
mm/khugepaged.c| 32 ++++++++++++++++++--------------1 file changed, 18 insertions(+), 14 deletions(-)mm/khugepaged.c +8
-1@@ -1284,7 +1284,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, for (addr = start_addr, _pte = pte; _pte < pte + HPAGE_PMD_NR; _pte++, addr += PAGE_SIZE) { pte_t pteval = ptep_get(_pte); - if (is_swap_pte(pteval)) { + if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { + ++none_or_zero; + if (!userfaultfd_armed(vma) && + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) { + continue; + } else { + result = SCAN_EXCEED_NONE_PTE;
Quoted 8 lines
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1298
Quoted 4 lines
mm/khugepaged.c +4
-0+ count_vm_event(THP_SCAN_EXCEED_NONE_PTE); + goto out_unmap; + } + } else if (!pte_present(pteval)) {If you are trying to merge this with the
_isolate()conditions, we can do a micro-optimization here -is_swap_pte, (pte_none&&is_zero_pfn), andpte_uffd_wpare disjoint conditions, so we can use if-else-if-else-if to write them.
Ah, indeed, thanks!
I think it would fit better into the follow-up patch that unifies the scanning logic, and I'll make sure to include it there ;p
Quoted 12 lines
Quoted 12 lines
mm/khugepaged.c +5
-0+ if (non_swap_entry(pte_to_swp_entry(pteval))) { + result = SCAN_PTE_NON_PRESENT; + goto out_unmap; + } + ++unmapped;if (!cc->is_khugepaged ||
unmapped <=khugepaged_max_ptes_swap) {mm/khugepaged.c +0
-0@@ -1293,7 +1309,7 @@ static int hpage_collapse_scan_pmd(structmm_struct *mm, * enabled swap entries. Please see
Quoted 8 lines
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1313
Quoted 5 lines
mm/khugepaged.c +1
-1* comment below for pte_uffd_wp(). */ - if (pte_swp_uffd_wp_any(pteval)) { + if (pte_swp_uffd_wp(pteval)) { result = SCAN_PTE_UFFD_WP;Could have mentioned in the changelog "while at it, convert
pte_swp_uffd_wp_anytopte_swp_uffd_wpsince we are in the swap pte branch".
Right, that would have been clearer.
I'll add that if a next version is needed :)
Quoted 26 lines
Quoted 22 lines
mm/khugepaged.c +0
-0goto out_unmap;}
mm/khugepaged.c +0
-12@@ -1304,18 +1320,6 @@ static int hpage_collapse_scan_pmd(structmm_struct *mm, goto out_unmap; } } - if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { - ++none_or_zero; - if (!userfaultfd_armed(vma) && - (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { - continue; - } else { - result = SCAN_EXCEED_NONE_PTE; - count_vm_event(THP_SCAN_EXCEED_NONE_PTE); - goto out_unmap; - } - } if (pte_uffd_wp(pteval)) { /*Quoted 1 line
* Don't collapse the page if any of the small
Otherwise LGTM
Quoted 1 line
Reviewed-by Dev Jain <[email protected]>
Cheers!