mm/khugepaged: abort collapse scan on non-swap entries
Subsystems (2)
Lance Yang
reviewerPATCH v3 1/1 mm/khugepaged: abort collapse scan on non-swap entries
From: Lance Yang <[email protected]>
Currently, special non-swap entries (like PTE markers) are not caught early in hpage_collapse_scan_pmd(), leading to failures deep in the swap-in logic.
A function that is called __collapse_huge_page_swapin() and documented to "Bring missing pages in from swap" will handle other types as well.
As analyzed by David[1], we could have ended up with the following entry types right before do_swap_page():
(1) Migration entries. We would have waited.
-> Maybe worth it to wait, maybe not. We suspect we don't stumble
into that frequently such that we don't care. We could always unlock this separately later.
(2) Device-exclusive entries. We would have converted to non-exclusive. -> See make_device_exclusive(), we cannot tolerate PMD entries and have to split them through FOLL_SPLIT_PMD. As popped up during a recent discussion, collapsing here is actually counter-productive, because the next conversion will PTE-map it again.
-> Ok to not collapse.
(3) Device-private entries. We would have migrated to RAM.
-> Device-private still does not support THPs, so collapsing right
now just means that the next device access would split the
folio again.
-> Ok to not collapse.(4) HWPoison entries
-> Cannot collapse
(5) Markers
-> Cannot collapse
First, this patch adds an early check for these non-swap entries. If any one is found, the scan is aborted immediately with the SCAN_PTE_NON_PRESENT result, as Lorenzo suggested[2], avoiding wasted work. While at it, convert pte_swp_uffd_wp_any() to pte_swp_uffd_wp() since we are in the swap pte branch.
Second, as Wei pointed out[3], we may have a chance to get a non-swap entry, since we will drop and re-acquire the mmap lock before __collapse_huge_page_swapin(). To handle this, we also add a non_swap_entry() check there.
Note that we can unlock later what we really need, and not account it towards max_swap_ptes.
[1] https://lore.kernel.org/linux-mm/[email protected]
[2] https://lore.kernel.org/linux-mm/[email protected]
[3] https://lore.kernel.org/linux-mm/20251005010511.ysek2nqojebqngf3@master
---
v2 -> v3:
- Collect Acked-by from David - thanks!
- Collect Reviewed-by from Wei and Dev - thanks!
- Add a non_swap_entry() check in __collapse_huge_page_swapin() (per Wei and David) - thanks!
- Rework the changelog to incorporate David's detailed analysis of non-swap entry types - thanks!!!
- https://lore.kernel.org/linux-mm/[email protected]/
v1 -> v2:
- Skip all non-present entries except swap entries (per David) thanks!
- https://lore.kernel.org/linux-mm/[email protected]/
mm/khugepaged.c | 37 +++++++++++++++++++++++-------------- 1 file changed, 23 insertions(+), 14 deletions(-)
mm/khugepaged.c +23 -14
@@ -1020,6 +1020,11 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm,if (!is_swap_pte (vmf.orig_pte))continue ; +if (non_swap_entry (pte_to_swp_entry (vmf.orig_pte))) { + result =SCAN_PTE_NON_PRESENT ; + goto out; + } + vmf.pte = pte; vmf.ptl = ptl; ret =do_swap_page (&vmf);
@@ -1281,7 +1286,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,for (addr = start_addr, _pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, addr +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (is_swap_pte (pteval)) { +if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { + ++none_or_zero; +if (!userfaultfd_armed (vma) && + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) { +continue ; + }else { + result =SCAN_EXCEED_NONE_PTE ; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); + goto out_unmap; + } + }else if (!pte_present (pteval)) { +if (non_swap_entry (pte_to_swp_entry (pteval))) { + result =SCAN_PTE_NON_PRESENT ; + goto out_unmap; + } + ++unmapped;if (!cc->is_khugepaged || unmapped <= khugepaged_max_ptes_swap) {
@@ -1290,7 +1311,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,* enabled swap entries. Please see * comment below for pte_uffd_wp(). */ -if (pte_swp_uffd_wp_any (pteval)) { +if (pte_swp_uffd_wp (pteval)) { result =SCAN_PTE_UFFD_WP ; goto out_unmap; }
@@ -1301,18 +1322,6 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, goto out_unmap; } } -if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { - ++none_or_zero; -if (!userfaultfd_armed (vma) && - (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { -continue ; - }else { - result =SCAN_EXCEED_NONE_PTE ; -count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); - goto out_unmap; - } - }if (pte_uffd_wp (pteval)) {/* * Don't collapse the page if any of the small
Signature · 1 line
2.49.0
Quoted 55 lines · Lance Yang
From: Lance Yang <[email protected]>
Currently, special non-swap entries (like PTE markers) are not caught early in
hpage_collapse_scan_pmd(), leading to failures deep in the swap-in logic.A function that is called
__collapse_huge_page_swapin()and documented to "Bring missing pages in from swap" will handle other types as well.As analyzed by David[1], we could have ended up with the following entry types right before
do_swap_page():(1) Migration entries. We would have waited.
-> Maybe worth it to wait, maybe not. We suspect we don't stumble
into that frequently such that we don't care. We could always unlock this separately later.(2) Device-exclusive entries. We would have converted to non-exclusive. -> See
make_device_exclusive(), we cannot tolerate PMD entries and have to split them throughFOLL_SPLIT_PMD. As popped up during a recent discussion, collapsing here is actually counter-productive, because the next conversion will PTE-map it again.
-> Ok to not collapse.(3) Device-private entries. We would have migrated to RAM.
-> Device-private still does not support THPs, so collapsing right now just means that the next device access would split the folio again. -> Ok to not collapse.(4) HWPoison entries -> Cannot collapse (5) Markers -> Cannot collapseFirst, this patch adds an early check for these non-swap entries. If any one is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENTresult, as Lorenzo suggested[2], avoiding wasted work. While at it, convertpte_swp_uffd_wp_any()topte_swp_uffd_wp()since we are in the swap pte branch.Second, as Wei pointed out[3], we may have a chance to get a non-swap entry, since we will drop and re-acquire the mmap lock before
__collapse_huge_page_swapin(). To handle this, we also add anon_swap_entry()check there.Note that we can unlock later what we really need, and not account it towards
max_swap_ptes.[1] https://lore.kernel.org/linux-mm/[email protected]
[2] https://lore.kernel.org/linux-mm/[email protected]
[3] https://lore.kernel.org/linux-mm/20251005010511.ysek2nqojebqngf3@master
Quoted 7 lines
Reviewed-by Wei Yang <[email protected]>Reviewed-by Dev Jain <[email protected]>Suggested-by David Hildenbrand <[email protected]>Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>---
Sorry for not replying earlier to your other mail.
LGTM.
We can always handle migration entries later if this shows up to be a problem (this time, in a clean way ...) and not count them towards actual "swap" entries.
Signature · 3 lines
Cheers David / dhildenb
Quoted 10 lines · Lance Yang
From: Lance Yang <[email protected]>
Currently, special non-swap entries (like PTE markers) are not caught early in
hpage_collapse_scan_pmd(), leading to failures deep in the swap-in logic.A function that is called
__collapse_huge_page_swapin()and documented to "Bring missing pages in from swap" will handle other types as well.
Quoted 8 lines
As analyzed by David[1], we could have ended up with the following entry types right before
do_swap_page():(1) Migration entries. We would have waited.
-> Maybe worth it to wait, maybe not. We suspect we don't stumble
into that frequently such that we don't care. We could always unlock this separately later.
On phones, we’ve actually observed user-visible stutters when the migration process gets preempted and a foreground app is waiting for do_swap_page() to complete. Anyway, that’s a separate issue we might address later.
Quoted 38 lines
(2) Device-exclusive entries. We would have converted to non-exclusive. -> See
make_device_exclusive(), we cannot tolerate PMD entries and have to split them throughFOLL_SPLIT_PMD. As popped up during a recent discussion, collapsing here is actually counter-productive, because the next conversion will PTE-map it again.
-> Ok to not collapse.(3) Device-private entries. We would have migrated to RAM.
-> Device-private still does not support THPs, so collapsing right now just means that the next device access would split the folio again. -> Ok to not collapse.(4) HWPoison entries
-> Cannot collapse(5) Markers
-> Cannot collapseFirst, this patch adds an early check for these non-swap entries. If any one is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENTresult, as Lorenzo suggested[2], avoiding wasted work. While at it, convertpte_swp_uffd_wp_any()topte_swp_uffd_wp()since we are in the swap pte branch.Second, as Wei pointed out[3], we may have a chance to get a non-swap entry, since we will drop and re-acquire the mmap lock before
__collapse_huge_page_swapin(). To handle this, we also add anon_swap_entry()check there.Note that we can unlock later what we really need, and not account it towards
max_swap_ptes.[1] https://lore.kernel.org/linux-mm/[email protected]
[2] https://lore.kernel.org/linux-mm/[email protected]
[3] https://lore.kernel.org/linux-mm/20251005010511.ysek2nqojebqngf3@master
Quoted 7 lines
Reviewed-by Wei Yang <[email protected]>Reviewed-by Dev Jain <[email protected]>Suggested-by David Hildenbrand <[email protected]>Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>---
Thanks
Barry
Zi Yan
reviewerQuoted 120 lines · Lance Yang
From: Lance Yang <[email protected]>
Currently, special non-swap entries (like PTE markers) are not caught early in
hpage_collapse_scan_pmd(), leading to failures deep in the swap-in logic.A function that is called
__collapse_huge_page_swapin()and documented to "Bring missing pages in from swap" will handle other types as well.As analyzed by David[1], we could have ended up with the following entry types right before
do_swap_page():(1) Migration entries. We would have waited.
-> Maybe worth it to wait, maybe not. We suspect we don't stumble
into that frequently such that we don't care. We could always unlock this separately later.(2) Device-exclusive entries. We would have converted to non-exclusive. -> See
make_device_exclusive(), we cannot tolerate PMD entries and have to split them throughFOLL_SPLIT_PMD. As popped up during a recent discussion, collapsing here is actually counter-productive, because the next conversion will PTE-map it again.
-> Ok to not collapse.(3) Device-private entries. We would have migrated to RAM.
-> Device-private still does not support THPs, so collapsing right now just means that the next device access would split the folio again. -> Ok to not collapse.(4) HWPoison entries
-> Cannot collapse(5) Markers
-> Cannot collapseFirst, this patch adds an early check for these non-swap entries. If any one is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENTresult, as Lorenzo suggested[2], avoiding wasted work. While at it, convertpte_swp_uffd_wp_any()topte_swp_uffd_wp()since we are in the swap pte branch.Second, as Wei pointed out[3], we may have a chance to get a non-swap entry, since we will drop and re-acquire the mmap lock before
__collapse_huge_page_swapin(). To handle this, we also add anon_swap_entry()check there.Note that we can unlock later what we really need, and not account it towards
max_swap_ptes.[1] https://lore.kernel.org/linux-mm/[email protected]
[2] https://lore.kernel.org/linux-mm/[email protected]
[3] https://lore.kernel.org/linux-mm/20251005010511.ysek2nqojebqngf3@masterReviewed-by Wei Yang <[email protected]>Reviewed-by Dev Jain <[email protected]>Suggested-by David Hildenbrand <[email protected]>Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>---
v2 -> v3:
- Collect Acked-by from David - thanks!
- Collect Reviewed-by from Wei and Dev - thanks!
- Add anon_swap_entry()check in__collapse_huge_page_swapin()(per Wei and David) - thanks!
- Rework the changelog to incorporate David's detailed analysis of non-swap entry types - thanks!!!
- https://lore.kernel.org/linux-mm/[email protected]/v1 -> v2:
- Skip all non-present entries except swap entries (per David) thanks!
- https://lore.kernel.org/linux-mm/[email protected]/mm/khugepaged.c | 37 +++++++++++++++++++++++-------------- 1 file changed, 23 insertions(+), 14 deletions(-)mm/khugepaged.c +22
-1@@ -1020,6 +1020,11 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm,if (!is_swap_pte (vmf.orig_pte))continue ; +if (non_swap_entry (pte_to_swp_entry (vmf.orig_pte))) { + result =SCAN_PTE_NON_PRESENT ; + goto out; + } + vmf.pte = pte; vmf.ptl = ptl; ret =do_swap_page (&vmf);@@ -1281,7 +1286,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,for (addr = start_addr, _pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, addr +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (is_swap_pte (pteval)) { +if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { + ++none_or_zero; +if (!userfaultfd_armed (vma) && + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) { +continue ; + }else { + result =SCAN_EXCEED_NONE_PTE ; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); + goto out_unmap; + } + }else if (!pte_present (pteval)) { +if (non_swap_entry (pte_to_swp_entry (pteval))) { + result =SCAN_PTE_NON_PRESENT ; + goto out_unmap; + } + ++unmapped;if (!cc->is_khugepaged || unmapped <= khugepaged_max_ptes_swap) {
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1314
Quoted 5 lines
mm/khugepaged.c +1
-1@@ -1290,7 +1311,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,* enabled swap entries. Please see * comment below for pte_uffd_wp(). */ -if (pte_swp_uffd_wp_any (pteval)) { +if (pte_swp_uffd_wp (pteval)) {
pte_swp_uffd_wp_any() returns true for both pte_swp_uffd_wp() and pte_marker_uffd_wp(). Why is it OK to just check pte_swp_uffd_wp() here?
Quoted 24 lines
mm/khugepaged.c +0
-12result =SCAN_PTE_UFFD_WP ; goto out_unmap; }@@ -1301,18 +1322,6 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, goto out_unmap; } } -if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { - ++none_or_zero; -if (!userfaultfd_armed (vma) && - (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { -continue ; - }else { - result =SCAN_EXCEED_NONE_PTE ; -count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); - goto out_unmap; - } - }if (pte_uffd_wp (pteval)) {/* * Don't collapse the page if any of the small Signature · 1 line
2.49.0
--
Best Regards,
Yan, Zi
Lance Yang
reviewerQuoted 122 lines · Zi Yan
Quoted 120 lines · Lance Yang
From: Lance Yang <[email protected]>
Currently, special non-swap entries (like PTE markers) are not caught early in
hpage_collapse_scan_pmd(), leading to failures deep in the swap-in logic.A function that is called
__collapse_huge_page_swapin()and documented to "Bring missing pages in from swap" will handle other types as well.As analyzed by David[1], we could have ended up with the following entry types right before
do_swap_page():(1) Migration entries. We would have waited.
-> Maybe worth it to wait, maybe not. We suspect we don't stumble
into that frequently such that we don't care. We could always unlock this separately later.(2) Device-exclusive entries. We would have converted to non-exclusive. -> See
make_device_exclusive(), we cannot tolerate PMD entries and have to split them throughFOLL_SPLIT_PMD. As popped up during a recent discussion, collapsing here is actually counter-productive, because the next conversion will PTE-map it again.
-> Ok to not collapse.(3) Device-private entries. We would have migrated to RAM.
-> Device-private still does not support THPs, so collapsing right now just means that the next device access would split the folio again. -> Ok to not collapse.(4) HWPoison entries -> Cannot collapse (5) Markers -> Cannot collapseFirst, this patch adds an early check for these non-swap entries. If any one is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENTresult, as Lorenzo suggested[2], avoiding wasted work. While at it, convertpte_swp_uffd_wp_any()topte_swp_uffd_wp()since we are in the swap pte branch.Second, as Wei pointed out[3], we may have a chance to get a non-swap entry, since we will drop and re-acquire the mmap lock before
__collapse_huge_page_swapin(). To handle this, we also add anon_swap_entry()check there.Note that we can unlock later what we really need, and not account it towards
max_swap_ptes.[1] https://lore.kernel.org/linux-mm/[email protected]
[2] https://lore.kernel.org/linux-mm/[email protected]
[3] https://lore.kernel.org/linux-mm/20251005010511.ysek2nqojebqngf3@masterReviewed-by Wei Yang <[email protected]>Reviewed-by Dev Jain <[email protected]>Suggested-by David Hildenbrand <[email protected]>Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>---
v2 -> v3:
- Collect Acked-by from David - thanks!
- Collect Reviewed-by from Wei and Dev - thanks!
- Add anon_swap_entry()check in__collapse_huge_page_swapin()(per Wei and David) - thanks!
- Rework the changelog to incorporate David's detailed analysis of non-swap entry types - thanks!!!
- https://lore.kernel.org/linux-mm/[email protected]/v1 -> v2:
- Skip all non-present entries except swap entries (per David) thanks!
- https://lore.kernel.org/linux-mm/[email protected]/mm/khugepaged.c | 37 +++++++++++++++++++++++-------------- 1 file changed, 23 insertions(+), 14 deletions(-)mm/khugepaged.c +22
-1@@ -1020,6 +1020,11 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm,if (!is_swap_pte (vmf.orig_pte))continue ; +if (non_swap_entry (pte_to_swp_entry (vmf.orig_pte))) { + result =SCAN_PTE_NON_PRESENT ; + goto out; + } + vmf.pte = pte; vmf.ptl = ptl; ret =do_swap_page (&vmf);@@ -1281,7 +1286,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,for (addr = start_addr, _pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, addr +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (is_swap_pte (pteval)) { +if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { + ++none_or_zero; +if (!userfaultfd_armed (vma) && + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) { +continue ; + }else { + result =SCAN_EXCEED_NONE_PTE ; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); + goto out_unmap; + } + }else if (!pte_present (pteval)) { +if (non_swap_entry (pte_to_swp_entry (pteval))) { + result =SCAN_PTE_NON_PRESENT ; + goto out_unmap; + } + ++unmapped;if (!cc->is_khugepaged || unmapped <= khugepaged_max_ptes_swap) {@@ -1290,7 +1311,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,
Quoted 8 lines
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1314
Quoted 5 lines
mm/khugepaged.c +1
-1* enabled swap entries. Please see * comment below for pte_uffd_wp(). */ -if (pte_swp_uffd_wp_any (pteval)) { +if (pte_swp_uffd_wp (pteval)) {
pte_swp_uffd_wp_any()returns true for bothpte_swp_uffd_wp()andpte_marker_uffd_wp(). Why is it OK to just checkpte_swp_uffd_wp()here?
+ } else if (!pte_present(pteval)) {
+ if (non_swap_entry(pte_to_swp_entry(pteval))) { <--
+ result = SCAN_PTE_NON_PRESENT;
+ goto out_unmap;
+ }
IIUC, we have just handled all non-swap entries above (which would include pte_marker_uffd_wp()), right?
Quoted 30 lines
Quoted 25 lines
mm/khugepaged.c +0
-12result =SCAN_PTE_UFFD_WP ; goto out_unmap; }@@ -1301,18 +1322,6 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, goto out_unmap; } } -if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { - ++none_or_zero; -if (!userfaultfd_armed (vma) && - (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { -continue ; - }else { - result =SCAN_EXCEED_NONE_PTE ; -count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); - goto out_unmap; - } - }if (pte_uffd_wp (pteval)) {/* * Don't collapse the page if any of the small Signature · 1 line
2.49.0--
Best Regards,
Yan, Zi
Quoted 137 lines · Lance Yang
Quoted 129 lines · Zi Yan
Quoted 119 lines · Lance Yang
From: Lance Yang <[email protected]>
Currently, special non-swap entries (like PTE markers) are not caught early in
hpage_collapse_scan_pmd(), leading to failures deep in the swap-in logic.A function that is called
__collapse_huge_page_swapin()and documented to "Bring missing pages in from swap" will handle other types as well.As analyzed by David[1], we could have ended up with the following entry types right before
do_swap_page():(1) Migration entries. We would have waited.
-> Maybe worth it to wait, maybe not. We suspect we don't stumble
into that frequently such that we don't care. We could always unlock this separately later.(2) Device-exclusive entries. We would have converted to non-exclusive. -> See
make_device_exclusive(), we cannot tolerate PMD entries and have to split them throughFOLL_SPLIT_PMD. As popped up during a recent discussion, collapsing here is actually counter-productive, because the next conversion will PTE-map it again.
-> Ok to not collapse.(3) Device-private entries. We would have migrated to RAM.
-> Device-private still does not support THPs, so collapsing right now just means that the next device access would split the folio again. -> Ok to not collapse.(4) HWPoison entries -> Cannot collapse (5) Markers -> Cannot collapseFirst, this patch adds an early check for these non-swap entries. If any one is found, the scan is aborted immediately with the
SCAN_PTE_NON_PRESENTresult, as Lorenzo suggested[2], avoiding wasted work. While at it, convertpte_swp_uffd_wp_any()topte_swp_uffd_wp()since we are in the swap pte branch.Second, as Wei pointed out[3], we may have a chance to get a non-swap entry, since we will drop and re-acquire the mmap lock before
__collapse_huge_page_swapin(). To handle this, we also add anon_swap_entry()check there.Note that we can unlock later what we really need, and not account it towards
max_swap_ptes.[1] https://lore.kernel.org/linux-mm/[email protected]
[2] https://lore.kernel.org/linux-mm/[email protected]
[3] https://lore.kernel.org/linux-mm/20251005010511.ysek2nqojebqngf3@masterReviewed-by Wei Yang <[email protected]>Reviewed-by Dev Jain <[email protected]>Suggested-by David Hildenbrand <[email protected]>Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Lance Yang <[email protected]>---
v2 -> v3:
- Collect Acked-by from David - thanks!
- Collect Reviewed-by from Wei and Dev - thanks!
- Add anon_swap_entry()check in__collapse_huge_page_swapin()(per Wei and David) - thanks!
- Rework the changelog to incorporate David's detailed analysis of non-swap entry types - thanks!!!
- https://lore.kernel.org/linux-mm/[email protected]/v1 -> v2:
- Skip all non-present entries except swap entries (per David) thanks!
- https://lore.kernel.org/linux-mm/[email protected]/mm/khugepaged.c | 37 +++++++++++++++++++++++-------------- 1 file changed, 23 insertions(+), 14 deletions(-)mm/khugepaged.c +21
-1@@ -1020,6 +1020,11 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm,if (!is_swap_pte (vmf.orig_pte))continue ; +if (non_swap_entry (pte_to_swp_entry (vmf.orig_pte))) { + result =SCAN_PTE_NON_PRESENT ; + goto out; + } + vmf.pte = pte; vmf.ptl = ptl; ret =do_swap_page (&vmf);@@ -1281,7 +1286,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,for (addr = start_addr, _pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, addr +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (is_swap_pte (pteval)) { +if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { + ++none_or_zero; +if (!userfaultfd_armed (vma) && + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) { +continue ; + }else { + result =SCAN_EXCEED_NONE_PTE ; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); + goto out_unmap; + } + }else if (!pte_present (pteval)) { +if (non_swap_entry (pte_to_swp_entry (pteval))) { + result =SCAN_PTE_NON_PRESENT ; + goto out_unmap; + } + ++unmapped;if (!cc->is_khugepaged || unmapped <= khugepaged_max_ptes_swap) {mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1314
Quoted 5 lines
mm/khugepaged.c +1
-1@@ -1290,7 +1311,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,* enabled swap entries. Please see * comment below for pte_uffd_wp(). */ -if (pte_swp_uffd_wp_any (pteval)) { +if (pte_swp_uffd_wp (pteval)) {
pte_swp_uffd_wp_any()returns true for bothpte_swp_uffd_wp()andpte_marker_uffd_wp(). Why is it OK to just checkpte_swp_uffd_wp()here?mm/khugepaged.c +5
-0+ }else if (!pte_present (pteval)) { + if (non_swap_entry(pte_to_swp_entry(pteval))) { <-- + result =SCAN_PTE_NON_PRESENT ; + goto out_unmap; + }
Quoted 2 lines
IIUC, we have just handled all non-swap entries above (which would include
pte_marker_uffd_wp()), right?
Got it. Thanks for the explanation.
--
Best Regards,
Yan, Zi
mm/khugepaged.c static int __collapse_huge_page_swapin(struct mm_struct *mm, · line 1026
Quoted 12 lines · Lance Yang
mm/khugepaged.c +4
-0@@ -1020,6 +1020,11 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm,if (!is_swap_pte (vmf.orig_pte))continue ; +if (non_swap_entry (pte_to_swp_entry (vmf.orig_pte))) { + result =SCAN_PTE_NON_PRESENT ; + goto out; + }
OK seems in line with what we were discussing before...
Quoted 12 lines
mm/khugepaged.c +4
-1+ vmf.pte = pte; vmf.ptl = ptl; ret =do_swap_page (&vmf);@@ -1281,7 +1286,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,for (addr = start_addr, _pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, addr +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (is_swap_pte (pteval)) { +if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { + ++none_or_zero; + if (!userfaultfd_armed(vma) &&
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1301
Quoted 10 lines
mm/khugepaged.c +10
-0+ (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) { +continue ; + }else { + result =SCAN_EXCEED_NONE_PTE ; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); + goto out_unmap; + } + } else if (!pte_present(pteval)) { +if (non_swap_entry (pte_to_swp_entry (pteval))) {
Hm but can't this be pte_protnone() at this stage (or something else)? And then we're just assuming pte_to_swp_entry() is operating on a swap entry when it in fact might not be?
Couldn't we end up with false positives here?
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1314
Quoted 13 lines
mm/khugepaged.c +5
-1+ result =SCAN_PTE_NON_PRESENT ; + goto out_unmap; + } + ++unmapped;if (!cc->is_khugepaged || unmapped <= khugepaged_max_ptes_swap) {@@ -1290,7 +1311,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,* enabled swap entries. Please see * comment below for pte_uffd_wp(). */ -if (pte_swp_uffd_wp_any (pteval)) { +if (pte_swp_uffd_wp (pteval)) {
Again you're assuming it's a swap entry but you're not asserting this is a swap entry in this branch?
Also an aside - I hate, hate, hate how this uffd wp stuff has infiltrated all kinds of open-coded stuff. It's so gross (not your fault, just a general comment...)
Quoted 23 lines
mm/khugepaged.c +0
-13result =SCAN_PTE_UFFD_WP ; goto out_unmap; }@@ -1301,18 +1322,6 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, goto out_unmap; } } -if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { - ++none_or_zero; -if (!userfaultfd_armed (vma) && - (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { -continue ; - }else { - result =SCAN_EXCEED_NONE_PTE ; -count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); - goto out_unmap; - } - }if (pte_uffd_wp (pteval)) {/* * Don't collapse the page if any of the small --
Quoted 2 lines
2.49.0
Cheers, Lorenzo
Lance Yang
reviewerQuoted 9 lines · Lorenzo Stoakes
Quoted 8 lines · Lance Yang
mm/khugepaged.c +0
-0@@ -1020,6 +1020,11 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm,if (!is_swap_pte (vmf.orig_pte))continue ;
Quoted 6 lines
mm/khugepaged.c static int __collapse_huge_page_swapin(struct mm_struct *mm, · line 1026
Quoted 4 lines
mm/khugepaged.c +4
-0+if (non_swap_entry (pte_to_swp_entry (vmf.orig_pte))) { + result =SCAN_PTE_NON_PRESENT ; + goto out; + }OK seems in line with what we were discussing before...
Yep. That's the idea :)
Quoted 15 lines
Quoted 15 lines
mm/khugepaged.c +6
-1+ vmf.pte = pte; vmf.ptl = ptl; ret =do_swap_page (&vmf);@@ -1281,7 +1286,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,for (addr = start_addr, _pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, addr +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (is_swap_pte (pteval)) { +if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { + ++none_or_zero; +if (!userfaultfd_armed (vma) && + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) {
Quoted 9 lines
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1301
Quoted 8 lines
mm/khugepaged.c +8
-0+continue ; + }else { + result =SCAN_EXCEED_NONE_PTE ; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); + goto out_unmap; + } + }else if (!pte_present (pteval)) { +if (non_swap_entry (pte_to_swp_entry (pteval))) {
Thanks for pointing that out!
Quoted 1 line
Hm but can't this be
pte_protnone()at this stage (or something else)? And then
Yeah. The funny thing is, a protnone pte cannot actually get here, IIUC.
```
static inline int pte_protnone(pte_t pte)
{
return (pte_flags(pte) & (_PAGE_PROTNONE | _PAGE_PRESENT))
== _PAGE_PROTNONE;
}
static inline int pte_present(pte_t a)
{
return pte_flags(a) & (_PAGE_PRESENT | _PAGE_PROTNONE);
}
```On x86, pte_present() returns true for a protnone pte. And I'd assume other archs behave similarly ...
Quoted 4 lines
we're just assuming
pte_to_swp_entry()is operating on a swap entry when it in fact might not be?Couldn't we end up with false positives here?
Emm, I think we're good here and the code is doing the right thing.
Quoted 17 lines
Quoted 9 lines
mm/khugepaged.c +4
-0+ result =SCAN_PTE_NON_PRESENT ; + goto out_unmap; + } + ++unmapped;if (!cc->is_khugepaged || unmapped <= khugepaged_max_ptes_swap) {mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1314
Quoted 5 lines
mm/khugepaged.c +1
-1@@ -1290,7 +1311,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,* enabled swap entries. Please see * comment below for pte_uffd_wp(). */ -if (pte_swp_uffd_wp_any (pteval)) { +if (pte_swp_uffd_wp (pteval)) {Again you're assuming it's a swap entry but you're not asserting this is a swap entry in this branch?
As we discussed above, the non_swap_entry() check has already kicked out anything that isn't a true swap entry, right?
Quoted 4 lines
Also an aside - I hate, hate, hate how this uffd wp stuff has infiltrated all kinds of open-coded stuff. It's so gross (not your fault, just a general comment...)
Haha, tell me about it. No argument from me there ;)
Thanks,
Lance
Quoted 7 lines
static inline int
pte_protnone(pte_t pte){ return (pte_flags(pte) & (_PAGE_PROTNONE | _PAGE_PRESENT)) == _PAGE_PROTNONE; }
Quoted 8 lines
static inline int
pte_present(pte_t a){ return pte_flags(a) & (_PAGE_PRESENT | _PAGE_PROTNONE); }```
On x86,
pte_present()returns true for a protnone pte. And I'd assume other archs behave similarly ...
Applies to all architecture. prot-none entries must be present, otherwise we'd have a lot of other issues :)
Signature · 3 lines
Cheers David / dhildenb
Lance Yang
reviewerQuoted 13 lines · David Hildenbrand
Quoted 13 lines
static inline int
pte_protnone(pte_t pte)
{
return (pte_flags(pte)& (_PAGE_PROTNONE|_PAGE_PRESENT)) ==_PAGE_PROTNONE;
}static inline int
pte_present(pte_t a)
{
returnpte_flags(a)& (_PAGE_PRESENT|_PAGE_PROTNONE); }
```
Quoted 4 lines
Quoted 2 lines
On x86,
pte_present()returns true for a protnone pte. And I'd assume other archs behave similarly ...Applies to all architecture. prot-none entries must be present, otherwise we'd have a lot of other issues :)
Thanks for confirming, David! That's good to know ;p
So @Lorenzo, looks like we're good here on the protnone front. Does that clear up your concern?
Quoted 46 lines · Lance Yang
Quoted 9 lines · Lorenzo Stoakes
Quoted 8 lines · Lance Yang
mm/khugepaged.c +0
-0@@ -1020,6 +1020,11 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm,if (!is_swap_pte (vmf.orig_pte))continue ;Quoted 6 lines
mm/khugepaged.c static int __collapse_huge_page_swapin(struct mm_struct *mm, · line 1026
Quoted 4 lines
mm/khugepaged.c +4
-0+if (non_swap_entry (pte_to_swp_entry (vmf.orig_pte))) { + result =SCAN_PTE_NON_PRESENT ; + goto out; + }OK seems in line with what we were discussing before...
Yep. That's the idea :)
Quoted 15 lines
Quoted 15 lines
mm/khugepaged.c +6
-1+ vmf.pte = pte; vmf.ptl = ptl; ret =do_swap_page (&vmf);@@ -1281,7 +1286,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,for (addr = start_addr, _pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, addr +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (is_swap_pte (pteval)) { +if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { + ++none_or_zero; +if (!userfaultfd_armed (vma) && + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) {Quoted 9 lines
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1301
Quoted 8 lines
mm/khugepaged.c +8
-0+continue ; + }else { + result =SCAN_EXCEED_NONE_PTE ; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); + goto out_unmap; + } + }else if (!pte_present (pteval)) { +if (non_swap_entry (pte_to_swp_entry (pteval))) {
Quoted 1 line
Thanks for pointing that out!
You've deleted what I've said here and also not indicated whether you'll do what I asked :)
Please be clearer...
Quoted 18 lines
Quoted 2 lines
Hm but can't this be
pte_protnone()at this stage (or something else)? And thenYeah. The funny thing is, a protnone pte cannot actually get here, IIUC.
``` static inline int pte_protnone(pte_t pte) { return (pte_flags(pte) & (_PAGE_PROTNONE | _PAGE_PRESENT)) == _PAGE_PROTNONE; } static inline int pte_present(pte_t a) { return pte_flags(a) & (_PAGE_PRESENT | _PAGE_PROTNONE); } ```
Quoted 2 lines
On x86,
pte_present()returns true for a protnone pte. And I'd assume other archs behave similarly ...
This was one example, we may make changes in the future that result in entries that are non-present but also non-swap.
I don't see the point in eliminating this check based on an implicit, open-coded assumption that this can never be the case, this is just asking for trouble.
Quoted 7 lines
Quoted 5 lines
we're just assuming
pte_to_swp_entry()is operating on a swap entry when it in fact might not be?Couldn't we end up with false positives here?
Emm, I think we're good here and the code is doing the right thing.
I mean sorry but just - NO - to doing swap operations based on open-coded checks that you implicitly feel must imply a swap entry.
This makes the code a lot more confusing, it opens us up to accidentally breaking things in future and has little to no benefit, I don't see why we're doing it.
I don't think every little 'aha X must imply Y so just eliminate Z' idea need be implemented, this feels like a sort of 'mathematical reduction of code ignoring all other factors'.
Quoted 19 lines
Quoted 18 lines
Quoted 10 lines
mm/khugepaged.c +4
-0+ result =SCAN_PTE_NON_PRESENT ; + goto out_unmap; + } + ++unmapped;if (!cc->is_khugepaged || unmapped <= khugepaged_max_ptes_swap) {mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1314
Quoted 5 lines
mm/khugepaged.c +1
-1@@ -1290,7 +1311,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,* enabled swap entries. Please see * comment below for pte_uffd_wp(). */ -if (pte_swp_uffd_wp_any (pteval)) { +if (pte_swp_uffd_wp (pteval)) {Again you're assuming it's a swap entry but you're not asserting this is a swap entry in this branch?
Quoted 2 lines
As we discussed above, the
non_swap_entry()check has already kicked out anything that isn't a true swap entry, right?
This is a different function?
Actually I'm mistaken here I think - you check in the code above:
if (is_swap_pte(pteval)) {
...
}So this is fine, please ignore sorry :)
Quoted 7 lines
Quoted 5 lines
Also an aside - I hate, hate, hate how this uffd wp stuff has infiltrated all kinds of open-coded stuff. It's so gross (not your fault, just a general comment...)
Haha, tell me about it. No argument from me there ;)
:)
Quoted 3 lines
Thanks,
Lance
Cheers, Lorenzo
Quoted 19 lines · Lance Yang
Quoted 16 lines · David Hildenbrand
Quoted 15 lines
static inline int
pte_protnone(pte_t pte)
{
return (pte_flags(pte)& (_PAGE_PROTNONE|_PAGE_PRESENT)) ==_PAGE_PROTNONE;
}static inline int
pte_present(pte_t a)
{
returnpte_flags(a)& (_PAGE_PRESENT|_PAGE_PROTNONE); }
```On x86,
pte_present()returns true for a protnone pte. And I'd assume other archs behave similarly ...
Quoted 8 lines
Quoted 2 lines
Applies to all architecture. prot-none entries must be present, otherwise we'd have a lot of other issues :)
Thanks for confirming, David! That's good to know ;p
So @Lorenzo, looks like we're good here on the protnone front. Does that clear up your concern?
No absolutely not, I go into detail as to why in my reply. I don't want that upstreamed.
Lance Yang
reviewerQuoted 47 lines · Lorenzo Stoakes
Quoted 46 lines · Lance Yang
Quoted 9 lines · Lorenzo Stoakes
Quoted 8 lines · Lance Yang
mm/khugepaged.c +0
-0@@ -1020,6 +1020,11 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm,if (!is_swap_pte (vmf.orig_pte))continue ;Quoted 6 lines
mm/khugepaged.c static int __collapse_huge_page_swapin(struct mm_struct *mm, · line 1026
Quoted 4 lines
mm/khugepaged.c +4
-0+if (non_swap_entry (pte_to_swp_entry (vmf.orig_pte))) { + result =SCAN_PTE_NON_PRESENT ; + goto out; + }OK seems in line with what we were discussing before...
Yep. That's the idea :)
Quoted 24 lines
Quoted 23 lines
mm/khugepaged.c +14
-1+ vmf.pte = pte; vmf.ptl = ptl; ret =do_swap_page (&vmf);@@ -1281,7 +1286,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,for (addr = start_addr, _pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, addr +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (is_swap_pte (pteval)) { +if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { + ++none_or_zero; +if (!userfaultfd_armed (vma) && + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) { +continue ; + }else { + result =SCAN_EXCEED_NONE_PTE ; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); + goto out_unmap; + } + }else if (!pte_present (pteval)) { +if (non_swap_entry (pte_to_swp_entry (pteval))) {
Quoted 6 lines
Quoted 1 line
Thanks for pointing that out!
You've deleted what I've said here and also not indicated whether you'll do what I asked :)
Please be clearer...
Oh, I didn't delete your comment at all ... It's just below ...
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Hm but can't this be
pte_protnone()at this stage (or something
else)? And then <-- Here!
Quoted 36 lines
Quoted 16 lines
Yeah. The funny thing is, a protnone pte cannot actually get here, IIUC.
``` static inline int pte_protnone(pte_t pte) { return (pte_flags(pte) & (_PAGE_PROTNONE | _PAGE_PRESENT)) == _PAGE_PROTNONE; } static inline int pte_present(pte_t a) { return pte_flags(a) & (_PAGE_PRESENT | _PAGE_PROTNONE); } ```Quoted 2 lines
On x86,
pte_present()returns true for a protnone pte. And I'd assume other archs behave similarly ...This was one example, we may make changes in the future that result in entries that are non-present but also non-swap.
I don't see the point in eliminating this check based on an implicit, open-coded assumption that this can never be the case, this is just asking for trouble.
Quoted 7 lines
Quoted 5 lines
we're just assuming
pte_to_swp_entry()is operating on a swap entry when it in fact might not be?Couldn't we end up with false positives here?
Emm, I think we're good here and the code is doing the right thing.
I mean sorry but just - NO - to doing swap operations based on open-coded checks that you implicitly feel must imply a swap entry.
Quoted 7 lines
This makes the code a lot more confusing, it opens us up to accidentally breaking things in future and has little to no benefit, I don't see why we're doing it.
I don't think every little 'aha X must imply Y so just eliminate Z' idea need be implemented, this feels like a sort of 'mathematical reduction of code ignoring all other factors'.
Understood. Changing !pte_present() to is_swap_pte() will resolve all your concerns, right?
```
if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) {
[...]
} else if (is_swap_pte(pteval)) { <-- Here
if (non_swap_entry(pte_to_swp_entry(pteval))) {
[...]
}
[...]}
```Quoted 25 lines
Quoted 20 lines
Quoted 19 lines
Quoted 11 lines
mm/khugepaged.c +4
-0+ result =SCAN_PTE_NON_PRESENT ; + goto out_unmap; + } + ++unmapped;if (!cc->is_khugepaged || unmapped <= khugepaged_max_ptes_swap) {mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1314
Quoted 5 lines
mm/khugepaged.c +1
-1@@ -1290,7 +1311,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,* enabled swap entries. Please see * comment below for pte_uffd_wp(). */ -if (pte_swp_uffd_wp_any (pteval)) { +if (pte_swp_uffd_wp (pteval)) {Again you're assuming it's a swap entry but you're not asserting this is a swap entry in this branch?
Quoted 2 lines
As we discussed above, the
non_swap_entry()check has already kicked out anything that isn't a true swap entry, right?This is a different function?
Quoted 7 lines
Actually I'm mistaken here I think - you check in the code above:
if (is_swap_pte(pteval)) { ... }So this is fine, please ignore sorry :)
No worries at all, thanks for double-checking and clarifying!
Quoted 16 lines
Quoted 8 lines
Quoted 6 lines
Also an aside - I hate, hate, hate how this uffd wp stuff has infiltrated all kinds of open-coded stuff. It's so gross (not your fault, just a general comment...)
Haha, tell me about it. No argument from me there ;)
:)
Quoted 3 lines
Thanks,
LanceCheers, Lorenzo
Quoted 92 lines · Lorenzo Stoakes
Quoted 46 lines · Lance Yang
Quoted 9 lines · Lorenzo Stoakes
Quoted 8 lines · Lance Yang
mm/khugepaged.c +0
-0@@ -1020,6 +1020,11 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm,if (!is_swap_pte (vmf.orig_pte))continue ;Quoted 6 lines
mm/khugepaged.c static int __collapse_huge_page_swapin(struct mm_struct *mm, · line 1026
Quoted 4 lines
mm/khugepaged.c +4
-0+if (non_swap_entry (pte_to_swp_entry (vmf.orig_pte))) { + result =SCAN_PTE_NON_PRESENT ; + goto out; + }OK seems in line with what we were discussing before...
Yep. That's the idea :)
Quoted 15 lines
Quoted 15 lines
mm/khugepaged.c +6
-1+ vmf.pte = pte; vmf.ptl = ptl; ret =do_swap_page (&vmf);@@ -1281,7 +1286,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,for (addr = start_addr, _pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, addr +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (is_swap_pte (pteval)) { +if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { + ++none_or_zero; +if (!userfaultfd_armed (vma) && + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) {Quoted 9 lines
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1301
Quoted 8 lines
mm/khugepaged.c +8
-0+continue ; + }else { + result =SCAN_EXCEED_NONE_PTE ; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); + goto out_unmap; + } + }else if (!pte_present (pteval)) { +if (non_swap_entry (pte_to_swp_entry (pteval))) {Quoted 1 line
Thanks for pointing that out!
You've deleted what I've said here and also not indicated whether you'll do what I asked :)
Please be clearer...
Quoted 18 lines
Quoted 2 lines
Hm but can't this be
pte_protnone()at this stage (or something else)? And thenYeah. The funny thing is, a protnone pte cannot actually get here, IIUC.
``` static inline int pte_protnone(pte_t pte) { return (pte_flags(pte) & (_PAGE_PROTNONE | _PAGE_PRESENT)) == _PAGE_PROTNONE; } static inline int pte_present(pte_t a) { return pte_flags(a) & (_PAGE_PRESENT | _PAGE_PROTNONE); } ```Quoted 2 lines
On x86,
pte_present()returns true for a protnone pte. And I'd assume other archs behave similarly ...This was one example, we may make changes in the future that result in entries that are non-present but also non-swap.
I don't see the point in eliminating this check based on an implicit, open-coded assumption that this can never be the case, this is just asking for trouble.
Quoted 7 lines
Quoted 5 lines
we're just assuming
pte_to_swp_entry()is operating on a swap entry when it in fact might not be?Couldn't we end up with false positives here?
Emm, I think we're good here and the code is doing the right thing.
I mean sorry but just - NO - to doing swap operations based on open-coded checks that you implicitly feel must imply a swap entry.
Quoted 7 lines
This makes the code a lot more confusing, it opens us up to accidentally breaking things in future and has little to no benefit, I don't see why we're doing it.
I don't think every little 'aha X must imply Y so just eliminate Z' idea need be implemented, this feels like a sort of 'mathematical reduction of code ignoring all other factors'.
Not sure I follow. If something is !none && !present it's what we call a swap PTE (that includes actual swap and non-swap PTEs).
We have the exact same code flow for example in copy_pte_range()->copy_nonpresent_pte() and I don't see a problem with it.
If we were to ever change what we call a "swap PTE" (I don't think so?) we'd have to fix stuff all over the place.
Maybe I get the concern here wrong?
Signature · 3 lines
Cheers David / dhildenb
Quoted 109 lines · Lance Yang
Quoted 47 lines · Lorenzo Stoakes
Quoted 46 lines · Lance Yang
Quoted 9 lines · Lorenzo Stoakes
Quoted 8 lines · Lance Yang
mm/khugepaged.c +0
-0@@ -1020,6 +1020,11 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm,if (!is_swap_pte (vmf.orig_pte))continue ;Quoted 6 lines
mm/khugepaged.c static int __collapse_huge_page_swapin(struct mm_struct *mm, · line 1026
Quoted 4 lines
mm/khugepaged.c +4
-0+if (non_swap_entry (pte_to_swp_entry (vmf.orig_pte))) { + result =SCAN_PTE_NON_PRESENT ; + goto out; + }OK seems in line with what we were discussing before...
Yep. That's the idea :)
Quoted 24 lines
Quoted 23 lines
mm/khugepaged.c +14
-1+ vmf.pte = pte; vmf.ptl = ptl; ret =do_swap_page (&vmf);@@ -1281,7 +1286,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,for (addr = start_addr, _pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, addr +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (is_swap_pte (pteval)) { +if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { + ++none_or_zero; +if (!userfaultfd_armed (vma) && + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) { +continue ; + }else { + result =SCAN_EXCEED_NONE_PTE ; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); + goto out_unmap; + } + }else if (!pte_present (pteval)) { +if (non_swap_entry (pte_to_swp_entry (pteval))) {Quoted 6 lines
Quoted 1 line
Thanks for pointing that out!
You've deleted what I've said here and also not indicated whether you'll do what I asked :)
Please be clearer...
Oh, I didn't delete your comment at all ... It's just below ...
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Hm but can't this be
pte_protnone()at this stage (or somethingelse)? And then <-- Here!
Quoted 36 lines
Quoted 16 lines
Yeah. The funny thing is, a protnone pte cannot actually get here, IIUC.
``` static inline int pte_protnone(pte_t pte) { return (pte_flags(pte) & (_PAGE_PROTNONE | _PAGE_PRESENT)) == _PAGE_PROTNONE; } static inline int pte_present(pte_t a) { return pte_flags(a) & (_PAGE_PRESENT | _PAGE_PROTNONE); } ```Quoted 2 lines
On x86,
pte_present()returns true for a protnone pte. And I'd assume other archs behave similarly ...This was one example, we may make changes in the future that result in entries that are non-present but also non-swap.
I don't see the point in eliminating this check based on an implicit, open-coded assumption that this can never be the case, this is just asking for trouble.
Quoted 7 lines
Quoted 5 lines
we're just assuming
pte_to_swp_entry()is operating on a swap entry when it in fact might not be?Couldn't we end up with false positives here?
Emm, I think we're good here and the code is doing the right thing.
I mean sorry but just - NO - to doing swap operations based on open-coded checks that you implicitly feel must imply a swap entry.
Quoted 7 lines
This makes the code a lot more confusing, it opens us up to accidentally breaking things in future and has little to no benefit, I don't see why we're doing it.
I don't think every little 'aha X must imply Y so just eliminate Z' idea need be implemented, this feels like a sort of 'mathematical reduction of code ignoring all other factors'.
Understood. Changing !pte_present() to
is_swap_pte()will resolve all your concerns, right?
Quoted 8 lines
```
if (pte_none(pteval)|| is_zero_pfn(pte_pfn(pteval))) { [...]
} else if (is_swap_pte(pteval)) { <-- Hereif (non_swap_entry(pte_to_swp_entry(pteval))) { [...] }[...]}
Can we please take a step back and make sure we are not starting to do stuff differently than elswehere in the kernel, please?
Signature · 3 lines
Cheers David / dhildenb
Quoted 119 lines · David Hildenbrand
Quoted 109 lines · Lance Yang
Quoted 47 lines · Lorenzo Stoakes
Quoted 46 lines · Lance Yang
Quoted 9 lines · Lorenzo Stoakes
Quoted 8 lines · Lance Yang
mm/khugepaged.c +0
-0@@ -1020,6 +1020,11 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm,if (!is_swap_pte (vmf.orig_pte))continue ;Quoted 6 lines
mm/khugepaged.c static int __collapse_huge_page_swapin(struct mm_struct *mm, · line 1026
Quoted 4 lines
mm/khugepaged.c +4
-0+if (non_swap_entry (pte_to_swp_entry (vmf.orig_pte))) { + result =SCAN_PTE_NON_PRESENT ; + goto out; + }OK seems in line with what we were discussing before...
Yep. That's the idea :)
Quoted 24 lines
Quoted 23 lines
mm/khugepaged.c +14
-1+ vmf.pte = pte; vmf.ptl = ptl; ret =do_swap_page (&vmf);@@ -1281,7 +1286,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,for (addr = start_addr, _pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, addr +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (is_swap_pte (pteval)) { +if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { + ++none_or_zero; +if (!userfaultfd_armed (vma) && + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) { +continue ; + }else { + result =SCAN_EXCEED_NONE_PTE ; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); + goto out_unmap; + } + }else if (!pte_present (pteval)) { +if (non_swap_entry (pte_to_swp_entry (pteval))) {Quoted 6 lines
Quoted 1 line
Thanks for pointing that out!
You've deleted what I've said here and also not indicated whether you'll do what I asked :)
Please be clearer...
Oh, I didn't delete your comment at all ... It's just below ...
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Hm but can't this be
pte_protnone()at this stage (or somethingelse)? And then <-- Here!
Quoted 36 lines
Quoted 16 lines
Yeah. The funny thing is, a protnone pte cannot actually get here, IIUC.
``` static inline int pte_protnone(pte_t pte) { return (pte_flags(pte) & (_PAGE_PROTNONE | _PAGE_PRESENT)) == _PAGE_PROTNONE; } static inline int pte_present(pte_t a) { return pte_flags(a) & (_PAGE_PRESENT | _PAGE_PROTNONE); } ```Quoted 2 lines
On x86,
pte_present()returns true for a protnone pte. And I'd assume other archs behave similarly ...This was one example, we may make changes in the future that result in entries that are non-present but also non-swap.
I don't see the point in eliminating this check based on an implicit, open-coded assumption that this can never be the case, this is just asking for trouble.
Quoted 7 lines
Quoted 5 lines
we're just assuming
pte_to_swp_entry()is operating on a swap entry when it in fact might not be?Couldn't we end up with false positives here?
Emm, I think we're good here and the code is doing the right thing.
I mean sorry but just - NO - to doing swap operations based on open-coded checks that you implicitly feel must imply a swap entry.
Quoted 7 lines
This makes the code a lot more confusing, it opens us up to accidentally breaking things in future and has little to no benefit, I don't see why we're doing it.
I don't think every little 'aha X must imply Y so just eliminate Z' idea need be implemented, this feels like a sort of 'mathematical reduction of code ignoring all other factors'.
Understood. Changing !pte_present() to
is_swap_pte()will resolve all your concerns, right?Quoted 8 lines
```
if (pte_none(pteval)|| is_zero_pfn(pte_pfn(pteval))) { [...]
} else if (is_swap_pte(pteval)) { <-- Hereif (non_swap_entry(pte_to_swp_entry(pteval))) { [...] }[...]}
Quoted 3 lines
Can we please take a step back and make sure we are not starting to do stuff differently than elswehere in the kernel, please?
For the sake of progress, I assume the compiler will optimize out the additional pte_none() stuff.
I absolutely, absolutely hate is_swap_pte(). To me, it makes the code more confusing that talking about something that is !present but also !none: there is something that is not an ordinary page table mapping.
The underlying problem is how we hacked in non-swap into swap (and that's exactly where it gets confusing). Well, which this series is all about.
So, I don't care in the end here.
Signature · 3 lines
Cheers David / dhildenb
Quoted 2 lines · David Hildenbrand
Can we please take a step back and make sure we are not starting to do stuff differently than elswehere in the kernel, please?
The code already had the is_swap_pte() check in it, so this is not changing anything, it's just avoiding confusion.
Even if this pattern exists elsewhere, I don't think propagating 'just have to happen to know X, Y, Z' to understand it is a good idea.
Quoted 6 lines · Lorenzo Stoakes
Quoted 2 lines · David Hildenbrand
Can we please take a step back and make sure we are not starting to do stuff differently than elswehere in the kernel, please?
The code already had the
is_swap_pte()check in it, so this is not changing anything, it's just avoiding confusion.
Well, in hpage_collapse_scan_pmd(), in __collapse_huge_page_isolate() no :)
And then it's called SCAN_PTE_NON_PRESENT and it all starts being a mess TBH.
But as raised offline, I don't have the time+energy for that today, so I don't care enough.
Signature · 3 lines
Cheers David / dhildenb
Quoted 97 lines · David Hildenbrand
Quoted 96 lines · Lorenzo Stoakes
Quoted 46 lines · Lance Yang
Quoted 9 lines · Lorenzo Stoakes
Quoted 8 lines · Lance Yang
mm/khugepaged.c +0
-0@@ -1020,6 +1020,11 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm,if (!is_swap_pte (vmf.orig_pte))continue ;Quoted 6 lines
mm/khugepaged.c static int __collapse_huge_page_swapin(struct mm_struct *mm, · line 1026
Quoted 4 lines
mm/khugepaged.c +4
-0+if (non_swap_entry (pte_to_swp_entry (vmf.orig_pte))) { + result =SCAN_PTE_NON_PRESENT ; + goto out; + }OK seems in line with what we were discussing before...
Yep. That's the idea :)
Quoted 15 lines
Quoted 15 lines
mm/khugepaged.c +6
-1+ vmf.pte = pte; vmf.ptl = ptl; ret =do_swap_page (&vmf);@@ -1281,7 +1286,23 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,for (addr = start_addr, _pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, addr +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (is_swap_pte (pteval)) { +if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { + ++none_or_zero; +if (!userfaultfd_armed (vma) && + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) {Quoted 9 lines
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1301
Quoted 8 lines
mm/khugepaged.c +8
-0+continue ; + }else { + result =SCAN_EXCEED_NONE_PTE ; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); + goto out_unmap; + } + }else if (!pte_present (pteval)) { +if (non_swap_entry (pte_to_swp_entry (pteval))) {Quoted 1 line
Thanks for pointing that out!
You've deleted what I've said here and also not indicated whether you'll do what I asked :)
Please be clearer...
Quoted 18 lines
Quoted 2 lines
Hm but can't this be
pte_protnone()at this stage (or something else)? And thenYeah. The funny thing is, a protnone pte cannot actually get here, IIUC.
``` static inline int pte_protnone(pte_t pte) { return (pte_flags(pte) & (_PAGE_PROTNONE | _PAGE_PRESENT)) == _PAGE_PROTNONE; } static inline int pte_present(pte_t a) { return pte_flags(a) & (_PAGE_PRESENT | _PAGE_PROTNONE); } ```Quoted 2 lines
On x86,
pte_present()returns true for a protnone pte. And I'd assume other archs behave similarly ...This was one example, we may make changes in the future that result in entries that are non-present but also non-swap.
I don't see the point in eliminating this check based on an implicit, open-coded assumption that this can never be the case, this is just asking for trouble.
Quoted 7 lines
Quoted 5 lines
we're just assuming
pte_to_swp_entry()is operating on a swap entry when it in fact might not be?Couldn't we end up with false positives here?
Emm, I think we're good here and the code is doing the right thing.
I mean sorry but just - NO - to doing swap operations based on open-coded checks that you implicitly feel must imply a swap entry.
This makes the code a lot more confusing, it opens us up to accidentally breaking things in future and has little to no benefit, I don't see why we're doing it.
Quoted 6 lines
Quoted 3 lines
I don't think every little 'aha X must imply Y so just eliminate Z' idea need be implemented, this feels like a sort of 'mathematical reduction of code ignoring all other factors'.
Not sure I follow. If something is !none && !present it's what we call a swap PTE (that includes actual swap and non-swap PTEs).
You see this is the issue. You know that. I knew that, then forgot that :)
Obviously you can go read the function:
/* check whether a pte points to a swap entry */
static inline int is_swap_pte(pte_t pte)
{
return !pte_none(pte) && !pte_present(pte);
}And see that this is the definition.
You also have to know that we hack pte_present() so it doesn't actually tell you the whether the PTE is present but whether it's present and _not protnone_ and again, I just forgot :>)
But that's the point! You shouldn't _need_ to 'just know' these things.
We have some operations that _require_ the entry to be a swap entry (including the very poorly named 'non swap entry'), or they'll read back/write garbage.
My key arugment is:
if (is_thing_we_want()) do_stuff_with_thing();
Abstracts all this, and is far better than:
if (condition_a) {
...
} else if (!condition_b) {
do_stuff_with_thing(); // you should just know that !a, !b means we can.
}I do agree that the naming is very bad here.
Quoted 3 lines
We have the exact same code flow for example in copy_pte_range()->copy_nonpresent_pte() and I don't see a problem with it.
Obviously I'd make the same argument above.
Quoted 3 lines
If we were to ever change what we call a "swap PTE" (I don't think so?) we'd have to fix stuff all over the place.
Sorry it's my mistake for raising this point about some imagined new non-present PTE that is also not-none, I'm fresh off vacation and forgetting things clearly :)
So withdraw that point.
Quoted 8 lines
Maybe I get the concern here wrong?
--
CheersDavid / dhildenb
Perhaps the naming is really the issue...
Cheers, Lorenzo
Quoted 5 lines · David Hildenbrand
Quoted 4 lines · Lorenzo Stoakes
Quoted 2 lines · David Hildenbrand
Can we please take a step back and make sure we are not starting to do stuff differently than elswehere in the kernel, please?
Quoted 7 lines
Quoted 2 lines
The code already had the
is_swap_pte()check in it, so this is not changing anything, it's just avoiding confusion.Well, in
hpage_collapse_scan_pmd(), in__collapse_huge_page_isolate()no :)And then it's called
SCAN_PTE_NON_PRESENTand it all starts being a mess TBH.
Yeah ugh. THP code is generally a mess :(
Quoted 3 lines
But as raised offline, I don't have the time+energy for that today, so I don't care enough.
Time, energy? What are those? :P
Quoted 6 lines
--
CheersDavid / dhildenb
Cheers, Lorenzo
Lance Yang
reviewerHi David, Lorenzo,
Thanks to both of you for the deep dive on this!!!
The code we've been discussing was moved to a new helper by another patch series[1], so let's call it a day on this thread and leave it as-is and just review/change the logic at its new home :)
[1] https://lore.kernel.org/linux-mm/[email protected]/
Thanks a lot!
Lance
Quoted 6 lines · David Hildenbrand
For the sake of progress, I assume the compiler will optimize out the additional
pte_none()stuff.I absolutely, absolutely hate
is_swap_pte(). To me, it makes the code more confusing that talking about something that is !present but also !none:
there is something that is not an ordinary page table mapping.
Yeah it's nasty for sure.
I hate non-swap-entry swap entries with a passion.
We clearly need to rework this... :)
Naming is hard however...
Quoted 3 lines
The underlying problem is how we hacked in non-swap into swap (and that's exactly where it gets confusing). Well, which this series is all about.
Yup
Quoted 2 lines
So, I don't care in the end here.
I do still feel a single guard condition works better than having to know implementaiton details so would still prefer this change to be made.
Quoted 6 lines
--
CheersDavid / dhildenb
Cheers, Lorenzo
Quoted 3 lines · Lance Yang
Hi David, Lorenzo,
Thanks to both of you for the deep dive on this!!!
That's ok :)
Quoted 5 lines
The code we've been discussing was moved to a new helper by another patch series[1], so let's call it a day on this thread and leave it as-is and just review/change the logic at its new home :)
Ahh, please do make that clear in future, I reviewed this on the basis it was a dependency of [1] :)
I separately reviewed that on that basis.
Will reply there...
Quoted 6 lines
[1]
https://lore.kernel.org/linux-mm/[email protected]/Thanks a lot!
Lance
Cheers, Lorenzo
Lance Yang
reviewer@Andrew could you please drop this patch from mm-new ;)
Thanks,
Lance
Quoted 28 lines · Lorenzo Stoakes
Quoted 3 lines · Lance Yang
Hi David, Lorenzo,
Thanks to both of you for the deep dive on this!!!
That's ok :)
Quoted 5 lines
The code we've been discussing was moved to a new helper by another patch series[1], so let's call it a day on this thread and leave it as-is and just review/change the logic at its new home :)
Ahh, please do make that clear in future, I reviewed this on the basis it was a dependency of [1] :)
I separately reviewed that on that basis.
Will reply there...
Quoted 6 lines
[1]
https://lore.kernel.org/linux-mm/[email protected]/Thanks a lot!
LanceCheers, Lorenzo