mm/khugepaged: refactor and merge PTE scanning logic
Subsystems (2)
Lance Yang
reviewercover mm/khugepaged: refactor and merge PTE scanning logic
Hi all,
This series cleans up the almost-duplicated PTE scanning logic in the collapse path.
The first one is a preparatory step that refactors both loops to use a single if-else-if-else-if chain for checking disjoint PTEs.
The second one then extracts the common logic into a shared helper.
Thanks,
Lance
Lance Yang (2):
mm/khugepaged: optimize PTE scanning with if-else-if-else-if chain mm/khugepaged: merge PTE scanning logic into a new helper
mm/khugepaged.c | 171 +++++++++++++++++++++++++++++------------------- 1 file changed, 104 insertions(+), 67 deletions(-)
Signature · 1 line
2.49.0
Lance Yang
reviewerPATCH 1/2 mm/khugepaged: optimize PTE scanning with if-else-if-else-if chain
From: Lance Yang <[email protected]>
As pointed out by Dev, the PTE checks for disjoint conditions in the scanning loops can be optimized. is_swap_pte, (pte_none && is_zero_pfn), and pte_uffd_wp are mutually exclusive.
This patch refactors the loops in both __collapse_huge_page_isolate() and hpage_collapse_scan_pmd() to use a continuous if-else-if-else-if chain instead of separate if blocks.
Also, this is a preparatory step to make it easier to merge the almost-duplicated scanning logic in these two functions, as suggested by David.
--- mm/khugepaged.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-)
mm/khugepaged.c +4 -8
@@ -548,8 +548,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,for (_pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, addr +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (pte_none (pteval) || (pte_present (pteval) && -is_zero_pfn (pte_pfn (pteval)))) { +if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { ++none_or_zero; if (!userfaultfd_armed(vma) && (!cc->is_khugepaged ||
@@ -560,12 +559,10 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); goto out; } - } -if (!pte_present (pteval)) { + }else if (!pte_present (pteval)) { result =SCAN_PTE_NON_PRESENT ; goto out; - } -if (pte_uffd_wp (pteval)) { + }else if (pte_uffd_wp (pteval)) { result =SCAN_PTE_UFFD_WP ; goto out; }
@@ -1316,8 +1313,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,count_vm_event (THP_SCAN_EXCEED_SWAP_PTE ); goto out_unmap; } - } -if (pte_uffd_wp (pteval)) { + }else if (pte_uffd_wp (pteval)) {/* * Don't collapse the page if any of the small * PTEs are armed with uffd write protection.
Signature · 1 line
2.49.0
Re: [PATCH mm-new 1/2] mm/khugepaged: optimize PTE scanning with if-else-if-else-if chain
Quoted 10 lines · Lance Yang
From: Lance Yang <[email protected]>
As pointed out by Dev, the PTE checks for disjoint conditions in the scanning loops can be optimized.
is_swap_pte, (pte_none&&is_zero_pfn), andpte_uffd_wpare mutually exclusive.This patch refactors the loops in both
__collapse_huge_page_isolate()andhpage_collapse_scan_pmd()to use a continuous if-else-if-else-if chain instead of separate if blocks.
Quoted 7 lines
Also, this is a preparatory step to make it easier to merge the almost-duplicated scanning logic in these two functions, as suggested by David.
Suggested-by Dev Jain <[email protected]>Suggested-by David Hildenbrand <[email protected]>Signed-off-by Lance Yang <[email protected]>
Signature · 2 lines
Wei Yang Help you, Help me
Re: [PATCH mm-new 1/2] mm/khugepaged: optimize PTE scanning with if-else-if-else-if chain
In reply to Lance YangQuoted 26 lines · Lance Yang
From: Lance Yang <[email protected]>
As pointed out by Dev, the PTE checks for disjoint conditions in the scanning loops can be optimized.
is_swap_pte, (pte_none&&is_zero_pfn), andpte_uffd_wpare mutually exclusive.This patch refactors the loops in both
__collapse_huge_page_isolate()andhpage_collapse_scan_pmd()to use a continuous if-else-if-else-if chain instead of separate if blocks.Also, this is a preparatory step to make it easier to merge the almost-duplicated scanning logic in these two functions, as suggested by David.
Suggested-by Dev Jain <[email protected]>Suggested-by David Hildenbrand <[email protected]>Signed-off-by Lance Yang <[email protected]>--- mm/khugepaged.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-)mm/khugepaged.c +0
-0
mm/khugepaged.c static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · line 551
Quoted 6 lines
mm/khugepaged.c +1
-2@@ -548,8 +548,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,for (_pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, addr +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (pte_none (pteval) || (pte_present (pteval) && -is_zero_pfn (pte_pfn (pteval)))) { +if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) {
Should have mentioned in the description that pte_present() is not required here, so removing it.
Quoted 28 lines
mm/khugepaged.c +3
-6++none_or_zero; if (!userfaultfd_armed(vma) && (!cc->is_khugepaged ||@@ -560,12 +559,10 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); goto out; } - } -if (!pte_present (pteval)) { + }else if (!pte_present (pteval)) { result =SCAN_PTE_NON_PRESENT ; goto out; - } -if (pte_uffd_wp (pteval)) { + }else if (pte_uffd_wp (pteval)) { result =SCAN_PTE_UFFD_WP ; goto out; }@@ -1316,8 +1313,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,count_vm_event (THP_SCAN_EXCEED_SWAP_PTE ); goto out_unmap; } - } -if (pte_uffd_wp (pteval)) { + }else if (pte_uffd_wp (pteval)) {/* * Don't collapse the page if any of the small * PTEs are armed with uffd write protection.
Lance Yang
reviewerRe: [PATCH mm-new 1/2] mm/khugepaged: optimize PTE scanning with if-else-if-else-if chain
Quoted 23 lines · Dev Jain
Quoted 21 lines · Lance Yang
From: Lance Yang <[email protected]>
As pointed out by Dev, the PTE checks for disjoint conditions in the scanning loops can be optimized.
is_swap_pte, (pte_none&&is_zero_pfn), andpte_uffd_wpare mutually exclusive.This patch refactors the loops in both
__collapse_huge_page_isolate()andhpage_collapse_scan_pmd()to use a continuous if-else-if-else-if chain instead of separate if blocks.Also, this is a preparatory step to make it easier to merge the almost-duplicated scanning logic in these two functions, as suggested by David.
Suggested-by Dev Jain <[email protected]>Suggested-by David Hildenbrand <[email protected]>Signed-off-by Lance Yang <[email protected]>---
mm/khugepaged.c| 12 ++++--------1 file changed, 4 insertions(+), 8 deletions(-)
Quoted 15 lines
mm/khugepaged.c static int __collapse_huge_page_isolate(struct · line 552
Quoted 12 lines
mm/khugepaged.c +1
-2@@ -548,8 +548,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, for (_pte = pte; _pte < pte + HPAGE_PMD_NR; _pte++, addr += PAGE_SIZE) { pte_t pteval = ptep_get(_pte); - if (pte_none(pteval) || (pte_present(pteval) && - is_zero_pfn(pte_pfn(pteval)))) { + if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) {Should have mentioned in the description that
pte_present()is not required here, so removing it.
Yep, got it.
Quoted 2 lines
Reviewed-by Dev Jain <[email protected]>
Cheers,
Lance
Quoted 31 lines
Quoted 31 lines
mm/khugepaged.c +0
-0++none_or_zero;if (!userfaultfd_armed(vma) &&
(!cc->is_khugepaged ||mm/khugepaged.c +2
-4@@ -560,12 +559,10 @@ static int __collapse_huge_page_isolate(structvm_area_struct *vma, count_vm_event(THP_SCAN_EXCEED_NONE_PTE); goto out; } - } - if (!pte_present(pteval)) { + } else if (!pte_present(pteval)) { result = SCAN_PTE_NON_PRESENT; goto out; - } - if (pte_uffd_wp(pteval)) { + } else if (pte_uffd_wp(pteval)) { result = SCAN_PTE_UFFD_WP; goto out;}
mm/khugepaged.c +1
-2@@ -1316,8 +1313,7 @@ static int hpage_collapse_scan_pmd(structmm_struct *mm, count_vm_event(THP_SCAN_EXCEED_SWAP_PTE); goto out_unmap; } - } - if (pte_uffd_wp(pteval)) { + } else if (pte_uffd_wp(pteval)) { /** Don't collapse the page if any of the small * PTEs are armed with uffd write protection.
Zi Yan
reviewerRe: [PATCH mm-new 1/2] mm/khugepaged: optimize PTE scanning with if-else-if-else-if chain
In reply to Lance YangQuoted 14 lines · Lance Yang
From: Lance Yang <[email protected]>
As pointed out by Dev, the PTE checks for disjoint conditions in the scanning loops can be optimized.
is_swap_pte, (pte_none&&is_zero_pfn), andpte_uffd_wpare mutually exclusive.This patch refactors the loops in both
__collapse_huge_page_isolate()andhpage_collapse_scan_pmd()to use a continuous if-else-if-else-if chain instead of separate if blocks.Also, this is a preparatory step to make it easier to merge the almost-duplicated scanning logic in these two functions, as suggested by David.
Quoted 7 lines
Suggested-by Dev Jain <[email protected]>Suggested-by David Hildenbrand <[email protected]>Signed-off-by Lance Yang <[email protected]>--- mm/khugepaged.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-)
LGTM. Reviewed-by: Zi Yan <[email protected]>
Best Regards,
Yan, Zi
Lance Yang
reviewerPATCH 2/2 mm/khugepaged: merge PTE scanning logic into a new helper
In reply to Lance YangFrom: Lance Yang <[email protected]>
As David suggested, the PTE scanning logic in hpage_collapse_scan_pmd() and __collapse_huge_page_isolate() was almost duplicated.
This patch cleans things up by moving all the common PTE checking logic into a new shared helper, thp_collapse_check_pte().
--- mm/khugepaged.c | 167 ++++++++++++++++++++++++++++++------------------ 1 file changed, 104 insertions(+), 63 deletions(-)
mm/khugepaged.c +104 -63
@@ -61,6 +61,12 @@ enum scan_result { SCAN_PAGE_FILLED, }; +enum pte_check_result { +PTE_CHECK_SUCCEED , +PTE_CHECK_CONTINUE , +PTE_CHECK_FAIL , +}; +#define CREATE_TRACE_POINTS #include <trace/events/huge_memory.h>
@@ -533,6 +539,87 @@ static void release_pte_pages(pte_t *pte, pte_t *_pte, } } +/* +* thp_collapse_check_pte - Check if a PTE is suitable for THP collapse +* @pte: PTE to check +* @vma: VMA the PTE belongs to +* @cc: Collapse control settings +* @scan_swap_pte: Allow scanning of swap PTEs if true +* @none_or_zero: Counter for none/zero PTEs (must be non-NULL) +* @unmapped: Counter for swap PTEs (must be non-NULL if scan_swap_pte +* is true) +* @scan_result: Used to return the failure reason (SCAN_*) on a +* PTE_CHECK_FAIL return. Must be non-NULL +* +* Returns: +* PTE_CHECK_SUCCEED - Valid PTE, proceed with collapse +* PTE_CHECK_CONTINUE - Skip this none/zero PTE but continue scanning +* PTE_CHECK_FAIL - Abort collapse scan +*/ +static inline int thp_collapse_check_pte (pte_t pte,struct vm_area_struct *vma, +struct collapse_control *cc,bool scan_swap_pte, +int *none_or_zero,int *unmapped,int *scan_result) +{ +VM_BUG_ON (!none_or_zero || !scan_result); +VM_BUG_ON (scan_swap_pte && !unmapped); + +if (pte_none (pte) ||is_zero_pfn (pte_pfn (pte))) { + (*none_or_zero)++; +if (!userfaultfd_armed (vma) && + (!cc->is_khugepaged || + *none_or_zero <= khugepaged_max_ptes_none)) { +return PTE_CHECK_CONTINUE ; + }else { + *scan_result =SCAN_EXCEED_NONE_PTE ; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); +return PTE_CHECK_FAIL ; + } + }else if (!pte_present (pte)) { +if (!scan_swap_pte) { + *scan_result =SCAN_PTE_NON_PRESENT ; +return PTE_CHECK_FAIL ; + } + +if (non_swap_entry (pte_to_swp_entry (pte))) { + *scan_result =SCAN_PTE_NON_PRESENT ; +return PTE_CHECK_FAIL ; + } + + (*unmapped)++; +if (!cc->is_khugepaged || + *unmapped <= khugepaged_max_ptes_swap) { +/* +* Always be strict with uffd-wp +* enabled swap entries. Please see +* comment below for pte_uffd_wp(). +*/ +if (pte_swp_uffd_wp (pte)) { + *scan_result =SCAN_PTE_UFFD_WP ; +return PTE_CHECK_FAIL ; + } +return PTE_CHECK_CONTINUE ; + }else { + *scan_result =SCAN_EXCEED_SWAP_PTE ; +count_vm_event (THP_SCAN_EXCEED_SWAP_PTE ); +return PTE_CHECK_FAIL ; + } + }else if (pte_uffd_wp (pte)) { +/* +* Don't collapse the page if any of the small +* PTEs are armed with uffd write protection. +* Here we can also mark the new huge pmd as +* write protected if any of the small ones is +* marked but that could bring unknown +* userfault messages that falls outside of +* the registered range. So, just be simple. +*/ + *scan_result =SCAN_PTE_UFFD_WP ; +return PTE_CHECK_FAIL ; + } + +return PTE_CHECK_SUCCEED ; +} + static int __collapse_huge_page_isolate(struct vm_area_struct *vma, unsigned long start_addr, pte_t *pte,
@@ -544,28 +631,20 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,unsigned long addr = start_addr;pte_t *_pte;int none_or_zero =0 , shared =0 , result =SCAN_FAIL , referenced =0 ; +int pte_check_res;for (_pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, addr +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { - ++none_or_zero; -if (!userfaultfd_armed (vma) && - (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { -continue ; - }else { - result =SCAN_EXCEED_NONE_PTE ; -count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); - goto out; - } - }else if (!pte_present (pteval)) { - result =SCAN_PTE_NON_PRESENT ; - goto out; - }else if (pte_uffd_wp (pteval)) { - result =SCAN_PTE_UFFD_WP ; + pte_check_res =thp_collapse_check_pte ( + pteval, vma, cc, false,/* scan_swap_pte = false */ + &none_or_zero,NULL , &result); + +if (pte_check_res ==PTE_CHECK_CONTINUE ) +continue ; +else if (pte_check_res ==PTE_CHECK_FAIL ) goto out; - } + page =vm_normal_page (vma, addr, pteval);if (unlikely (!page) ||unlikely (is_zone_device_page (page))) { result =SCAN_PAGE_NULL ;
@@ -1260,6 +1339,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,unsigned long addr;spinlock_t *ptl;int node =NUMA_NO_NODE , unmapped =0 ; +int pte_check_res;VM_BUG_ON (start_addr & ~HPAGE_PMD_MASK );
@@ -1278,54 +1358,15 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,for (addr = start_addr, _pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, addr +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); -if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { - ++none_or_zero; -if (!userfaultfd_armed (vma) && - (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { -continue ; - }else { - result =SCAN_EXCEED_NONE_PTE ; -count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); - goto out_unmap; - } - }else if (!pte_present (pteval)) { -if (non_swap_entry (pte_to_swp_entry (pteval))) { - result =SCAN_PTE_NON_PRESENT ; - goto out_unmap; - } - ++unmapped; -if (!cc->is_khugepaged || - unmapped <= khugepaged_max_ptes_swap) { -/* -* Always be strict with uffd-wp -* enabled swap entries. Please see -* comment below for pte_uffd_wp(). -*/ -if (pte_swp_uffd_wp (pteval)) { - result =SCAN_PTE_UFFD_WP ; - goto out_unmap; - } -continue ; - }else { - result =SCAN_EXCEED_SWAP_PTE ; -count_vm_event (THP_SCAN_EXCEED_SWAP_PTE ); - goto out_unmap; - } - }else if (pte_uffd_wp (pteval)) { -/* -* Don't collapse the page if any of the small -* PTEs are armed with uffd write protection. -* Here we can also mark the new huge pmd as -* write protected if any of the small ones is -* marked but that could bring unknown -* userfault messages that falls outside of -* the registered range. So, just be simple. -*/ - result =SCAN_PTE_UFFD_WP ; + pte_check_res =thp_collapse_check_pte ( + pteval, vma, cc, true,/* scan_swap_pte = true */ + &none_or_zero, &unmapped, &result); + +if (pte_check_res ==PTE_CHECK_CONTINUE ) +continue ; +else if (pte_check_res ==PTE_CHECK_FAIL ) goto out_unmap; - } page =vm_normal_page (vma, addr, pteval);if (unlikely (!page) ||unlikely (is_zone_device_page (page))) {
Signature · 1 line
2.49.0
Dev Jain
reviewerRe: [PATCH mm-new 2/2] mm/khugepaged: merge PTE scanning logic into a new helper
Quoted 5 lines · Lance Yang
From: Lance Yang <[email protected]>
As David suggested, the PTE scanning logic in
hpage_collapse_scan_pmd()and__collapse_huge_page_isolate()was almost duplicated.
Quoted 7 lines
This patch cleans things up by moving all the common PTE checking logic into a new shared helper,
thp_collapse_check_pte().Suggested-by David Hildenbrand <[email protected]>Signed-off-by Lance Yang <[email protected]>---
In hpage_collapse_scan_pmd(), we enter with mmap lock held, so for an anonymous vma, is it even possible to hit if (!folio_test_anon(folio))? In which case we can replace this with VM_BUG_ON_FOLIO and abstract away till the folio_maybe_mapped_shared() block?
Lance Yang
reviewerRe: [PATCH mm-new 2/2] mm/khugepaged: merge PTE scanning logic into a new helper
Quoted 10 lines · Dev Jain
Quoted 8 lines · Lance Yang
From: Lance Yang <[email protected]>
As David suggested, the PTE scanning logic in
hpage_collapse_scan_pmd()and__collapse_huge_page_isolate()was almost duplicated.This patch cleans things up by moving all the common PTE checking logic into a new shared helper,
thp_collapse_check_pte().
Quoted 6 lines
Quoted 3 lines
Suggested-by David Hildenbrand <[email protected]>Signed-off-by Lance Yang <[email protected]>---
In
hpage_collapse_scan_pmd(), we enter with mmap lock held, so for an anonymous vma, is it even possible to hit if (!folio_test_anon(folio))?
Ah, indeed :)
Quoted 2 lines
In which case we can replace this with
VM_BUG_ON_FOLIOand abstract away till thefolio_maybe_mapped_shared()block?
So you meant something like this:
mm/khugepaged.c +1 -5
@@ -1374,11 +1374,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, goto out_unmap; } folio =page_folio (page); - -if (!folio_test_anon (folio)) { - result =SCAN_PAGE_ANON ; - goto out_unmap; - } +VM_BUG_ON_FOLIO (!folio_test_anon (folio), folio);/* * We treat a single page as shared if any part of the THP
Dev Jain
reviewerRe: [PATCH mm-new 2/2] mm/khugepaged: merge PTE scanning logic into a new helper
Quoted 32 lines · Lance Yang
Quoted 10 lines · Dev Jain
Quoted 8 lines · Lance Yang
From: Lance Yang <[email protected]>
As David suggested, the PTE scanning logic in
hpage_collapse_scan_pmd()and__collapse_huge_page_isolate()was almost duplicated.This patch cleans things up by moving all the common PTE checking logic into a new shared helper,
thp_collapse_check_pte().Quoted 6 lines
Quoted 3 lines
Suggested-by David Hildenbrand <[email protected]>Signed-off-by Lance Yang <[email protected]>---
In
hpage_collapse_scan_pmd(), we enter with mmap lock held, so for an anonymous vma, is it even possible to hit if (!folio_test_anon(folio))?Ah, indeed :)
Quoted 2 lines
In which case we can replace this with
VM_BUG_ON_FOLIOand abstract away till thefolio_maybe_mapped_shared()block?So you meant something like this:
mm/khugepaged.c +0
-0
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct · line 1378
Quoted 10 lines
mm/khugepaged.c +1
-5@@ -1374,11 +1374,7 @@ static int hpage_collapse_scan_pmd(structmm_struct *mm, goto out_unmap; } folio = page_folio(page); - - if (!folio_test_anon(folio)) { - result = SCAN_PAGE_ANON; - goto out_unmap; - } + VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio);
Yes, I would suggest a different patch making this change, then in the
last patch you can abstract away till the shared statistics line since that
much code will become common between scan_pmd and _isolate.
Quoted 5 lines
mm/khugepaged.c +0
-0/** We treat a single page as shared if any part of the THP
Lance Yang
reviewerRe: [PATCH mm-new 2/2] mm/khugepaged: merge PTE scanning logic into a new helper
Quoted 47 lines · Dev Jain
Quoted 32 lines · Lance Yang
Quoted 10 lines · Dev Jain
Quoted 8 lines · Lance Yang
From: Lance Yang <[email protected]>
As David suggested, the PTE scanning logic in
hpage_collapse_scan_pmd()and__collapse_huge_page_isolate()was almost duplicated.This patch cleans things up by moving all the common PTE checking logic into a new shared helper,
thp_collapse_check_pte().Quoted 6 lines
Quoted 3 lines
Suggested-by David Hildenbrand <[email protected]>Signed-off-by Lance Yang <[email protected]>---
In
hpage_collapse_scan_pmd(), we enter with mmap lock held, so for an anonymous vma, is it even possible to hit if (!folio_test_anon(folio))?Ah, indeed :)
Quoted 2 lines
In which case we can replace this with
VM_BUG_ON_FOLIOand abstract away till thefolio_maybe_mapped_shared()block?So you meant something like this:
mm/khugepaged.c +0
-0mm/khugepaged.c static int hpage_collapse_scan_pmd(struct · line 1378
Quoted 10 lines
mm/khugepaged.c +1
-5@@ -1374,11 +1374,7 @@ static int hpage_collapse_scan_pmd(structmm_struct *mm, goto out_unmap; } folio = page_folio(page); - - if (!folio_test_anon(folio)) { - result = SCAN_PAGE_ANON; - goto out_unmap; - } + VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio);Yes, I would suggest a different patch making this change, then in the
Quoted 3 lines
last patch you can abstract away till the shared statistics line since that
much code will become common between
scan_pmdand_isolate.
Nice. Makes sense to me!
I'll rework the series into three patches then:
1) The if-else-if-else-if optimization
2) Convert the !folio_test_anon() check to a VM_BUG_ON_FOLIO()
3) The final merge of the scanning logic into a new helper
Thanks,
Lance
Quoted 7 lines
Quoted 7 lines
mm/khugepaged.c +0
-0/** We treat a single page as shared if any part of the THP
Re: [PATCH mm-new 2/2] mm/khugepaged: merge PTE scanning logic into a new helper
In reply to Dev JainQuoted 7 lines · Dev Jain
Quoted 5 lines · Lance Yang
From: Lance Yang <[email protected]>
As David suggested, the PTE scanning logic in
hpage_collapse_scan_pmd()and__collapse_huge_page_isolate()was almost duplicated.
Quoted 8 lines
Quoted 6 lines
This patch cleans things up by moving all the common PTE checking logic into a new shared helper,
thp_collapse_check_pte().Suggested-by David Hildenbrand <[email protected]>Signed-off-by Lance Yang <[email protected]>---
In
hpage_collapse_scan_pmd(), we enter with mmap lock held, so for
This is true for the first loop, but we will unlock/lock mmap and revalidate vma before isolation.
Quoted 3 lines
an anonymous vma, is it even possible to hit if (!folio_test_anon(folio))? In which case we can replace this with
VM_BUG_ON_FOLIOand abstract away till thefolio_maybe_mapped_shared()block?
But it looks still valid, since hugepage_vma_revalidate() will check the vma is still anonymous vma after grab the mmap lock again.
My concern is would VM_BUG_ON_FOLIO() be too heavy? How about warn on and return?
Signature · 2 lines
Wei Yang Help you, Help me
Dev Jain
reviewerRe: [PATCH mm-new 2/2] mm/khugepaged: merge PTE scanning logic into a new helper
Quoted 17 lines · Wei Yang
Quoted 6 lines · Dev Jain
Quoted 5 lines · Lance Yang
From: Lance Yang <[email protected]>
As David suggested, the PTE scanning logic in
hpage_collapse_scan_pmd()and__collapse_huge_page_isolate()was almost duplicated.Quoted 7 lines
Quoted 6 lines
This patch cleans things up by moving all the common PTE checking logic into a new shared helper,
thp_collapse_check_pte().Suggested-by David Hildenbrand <[email protected]>Signed-off-by Lance Yang <[email protected]>---
In
hpage_collapse_scan_pmd(), we enter with mmap lock held, so forThis is true for the first loop, but we will unlock/lock mmap and revalidate vma before isolation.
Quoted 8 lines
Quoted 3 lines
an anonymous vma, is it even possible to hit if (!folio_test_anon(folio))? In which case we can replace this with
VM_BUG_ON_FOLIOand abstract away till thefolio_maybe_mapped_shared()block?But it looks still valid, since
hugepage_vma_revalidate()will check the vma is still anonymous vma after grab the mmap lock again.My concern is would
VM_BUG_ON_FOLIO()be too heavy? How about warn on and return?
Frankly I do not have much opinion on the BUG_ON/WARN_ON debate since I haven't properly understood that, but this BUG_ON is under CONFIG_DEBUG_VM anways. But if you want to change this to WARN then you can do it at both places.
Quoted 1 line
Lance Yang
reviewerRe: [PATCH mm-new 2/2] mm/khugepaged: merge PTE scanning logic into a new helper
Quoted 22 lines · Dev Jain
Quoted 20 lines · Wei Yang
Quoted 6 lines · Dev Jain
Quoted 5 lines · Lance Yang
From: Lance Yang <[email protected]>
As David suggested, the PTE scanning logic in
hpage_collapse_scan_pmd()and__collapse_huge_page_isolate()was almost duplicated.Quoted 7 lines
Quoted 6 lines
This patch cleans things up by moving all the common PTE checking logic into a new shared helper,
thp_collapse_check_pte().Suggested-by David Hildenbrand <[email protected]>Signed-off-by Lance Yang <[email protected]>---
In
hpage_collapse_scan_pmd(), we enter with mmap lock held, so forThis is true for the first loop, but we will unlock/lock mmap and revalidate vma before isolation.
Quoted 3 lines
an anonymous vma, is it even possible to hit if (!
folio_test_anon(folio))? In which case we can replace this withVM_BUG_ON_FOLIOand abstract away till thefolio_maybe_mapped_shared()block?
Quoted 8 lines
Quoted 5 lines
But it looks still valid, since
hugepage_vma_revalidate()will check the vma is still anonymous vma after grab the mmap lock again.My concern is would
VM_BUG_ON_FOLIO()be too heavy? How about warn on and return?Frankly I do not have much opinion on the BUG_ON/WARN_ON debate since I haven't properly understood that, but this
BUG_ONis underCONFIG_DEBUG_VManways. But
Yeah, VM_BUG_ON_FOLIO() is under CONFIG_DEBUG_VM, so it won't affect production kernels.
Quoted 1 line
if you want to change this to WARN then you can do it at both places.
It should flag such an impossible condition there during development. So, I'd prefer to stick with VM_BUG_ON_FOLIO().
@Wei please let me know if you feel strongly otherwise :)
Zi Yan
reviewerRe: [PATCH mm-new 2/2] mm/khugepaged: merge PTE scanning logic into a new helper
Quoted 26 lines · Lance Yang
Quoted 25 lines · Dev Jain
Quoted 23 lines · Wei Yang
Quoted 6 lines · Dev Jain
Quoted 5 lines · Lance Yang
From: Lance Yang <[email protected]>
As David suggested, the PTE scanning logic in
hpage_collapse_scan_pmd()and__collapse_huge_page_isolate()was almost duplicated.Quoted 7 lines
Quoted 6 lines
This patch cleans things up by moving all the common PTE checking logic into a new shared helper,
thp_collapse_check_pte().Suggested-by David Hildenbrand <[email protected]>Signed-off-by Lance Yang <[email protected]>---
In
hpage_collapse_scan_pmd(), we enter with mmap lock held, so forThis is true for the first loop, but we will unlock/lock mmap and revalidate vma before isolation.
Quoted 3 lines
an anonymous vma, is it even possible to hit if (!
folio_test_anon(folio))? In which case we can replace this withVM_BUG_ON_FOLIOand abstract away till thefolio_maybe_mapped_shared()block?But it looks still valid, since
hugepage_vma_revalidate()will check the vma is still anonymous vma after grab the mmap lock again.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
My concern is would
VM_BUG_ON_FOLIO()be too heavy? How about warn on and return?Frankly I do not have much opinion on the BUG_ON/WARN_ON debate since I haven't properly understood that, but this
BUG_ONis underCONFIG_DEBUG_VManways. ButYeah,
VM_BUG_ON_FOLIO()is underCONFIG_DEBUG_VM, so it won't affect production kernels.
Many distros enable it by default. For mm, we are moving away from using BUG_ON or VM_BUG_ON. No need to crash the system if it is possible to handle it gracefully.
Quoted 7 lines
Quoted 2 lines
if you want to change this to WARN then you can do it at both places.
It should flag such an impossible condition there during development. So, I'd prefer to stick with
VM_BUG_ON_FOLIO().@Wei please let me know if you feel strongly otherwise :)
--
Best Regards,
Yan, Zi
Lance Yang
reviewerRe: [PATCH mm-new 2/2] mm/khugepaged: merge PTE scanning logic into a new helper
Quoted 34 lines · Zi Yan
Quoted 32 lines · Lance Yang
Quoted 30 lines · Dev Jain
Quoted 17 lines · Wei Yang
Quoted 6 lines · Dev Jain
Quoted 5 lines · Lance Yang
From: Lance Yang <[email protected]>
As David suggested, the PTE scanning logic in
hpage_collapse_scan_pmd()and__collapse_huge_page_isolate()was almost duplicated.Quoted 7 lines
Quoted 6 lines
This patch cleans things up by moving all the common PTE checking logic into a new shared helper,
thp_collapse_check_pte().Suggested-by David Hildenbrand <[email protected]>Signed-off-by Lance Yang <[email protected]>---
In
hpage_collapse_scan_pmd(), we enter with mmap lock held, so forThis is true for the first loop, but we will unlock/lock mmap and revalidate vma before isolation.
Quoted 8 lines
Quoted 3 lines
an anonymous vma, is it even possible to hit if (!
folio_test_anon(folio))? In which case we can replace this withVM_BUG_ON_FOLIOand abstract away till thefolio_maybe_mapped_shared()block?But it looks still valid, since
hugepage_vma_revalidate()will check the vma is still anonymous vma after grab the mmap lock again.My concern is would
VM_BUG_ON_FOLIO()be too heavy? How about warn on and return?Frankly I do not have much opinion on the BUG_ON/WARN_ON debate since I haven't properly understood that, but this
BUG_ONis underCONFIG_DEBUG_VManways. But
Quoted 6 lines
Quoted 2 lines
Yeah,
VM_BUG_ON_FOLIO()is underCONFIG_DEBUG_VM, so it won't affect production kernels.Many distros enable it by default. For mm, we are moving away from using
BUG_ONorVM_BUG_ON. No need to crash the system if it is possible to handle it gracefully.
Ah, good to know that, thanks!
Quoted 13 lines
Quoted 8 lines
Quoted 3 lines
if you want to change this to WARN then you can do it at both places.
It should flag such an impossible condition there during development. So, I'd prefer to stick with
VM_BUG_ON_FOLIO().@Wei please let me know if you feel strongly otherwise :)
--
Best Regards,
Yan, Zi
Zi Yan
reviewerRe: [PATCH mm-new 2/2] mm/khugepaged: merge PTE scanning logic into a new helper
In reply to Lance YangQuoted 48 lines · Lance Yang
From: Lance Yang <[email protected]>
As David suggested, the PTE scanning logic in
hpage_collapse_scan_pmd()and__collapse_huge_page_isolate()was almost duplicated.This patch cleans things up by moving all the common PTE checking logic into a new shared helper,
thp_collapse_check_pte().Suggested-by David Hildenbrand <[email protected]>Signed-off-by Lance Yang <[email protected]>--- mm/khugepaged.c | 167 ++++++++++++++++++++++++++++++------------------ 1 file changed, 104 insertions(+), 63 deletions(-)mm/khugepaged.c +19
-0@@ -61,6 +61,12 @@ enum scan_result { SCAN_PAGE_FILLED, }; +enum pte_check_result { +PTE_CHECK_SUCCEED , +PTE_CHECK_CONTINUE , +PTE_CHECK_FAIL , +}; +#define CREATE_TRACE_POINTS #include <trace/events/huge_memory.h> @@ -533,6 +539,87 @@ static void release_pte_pages(pte_t *pte, pte_t *_pte, } } +/* +* thp_collapse_check_pte - Check if a PTE is suitable for THP collapse +* @pte: PTE to check +* @vma: VMA the PTE belongs to +* @cc: Collapse control settings +* @scan_swap_pte: Allow scanning of swap PTEs if true +* @none_or_zero: Counter for none/zero PTEs (must be non-NULL) +* @unmapped: Counter for swap PTEs (must be non-NULL if scan_swap_pte +* is true) +* @scan_result: Used to return the failure reason (SCAN_*) on a +* PTE_CHECK_FAIL return. Must be non-NULL +* +* Returns:
mm/khugepaged.c static void release_pte_pages(pte_t *pte, pte_t *_pte, · line 564
Quoted 10 lines
mm/khugepaged.c +10
-0+* PTE_CHECK_SUCCEED - Valid PTE, proceed with collapse +* PTE_CHECK_CONTINUE - Skip this none/zero PTE but continue scanning +* PTE_CHECK_FAIL - Abort collapse scan +*/ +static inline int thp_collapse_check_pte (pte_t pte,struct vm_area_struct *vma, +struct collapse_control *cc,bool scan_swap_pte, +int *none_or_zero,int *unmapped,int *scan_result) +{ +VM_BUG_ON (!none_or_zero || !scan_result); +VM_BUG_ON (scan_swap_pte && !unmapped);
Please use VM_WARN_ONCE_ON and just return. Or put all stats in a struct, pass and check the struct pointer instead.
Best Regards,
Yan, Zi
Lance Yang
reviewerRe: [PATCH mm-new 2/2] mm/khugepaged: merge PTE scanning logic into a new helper
Quoted 55 lines · Zi Yan
Quoted 53 lines · Lance Yang
From: Lance Yang <[email protected]>
As David suggested, the PTE scanning logic in
hpage_collapse_scan_pmd()and__collapse_huge_page_isolate()was almost duplicated.This patch cleans things up by moving all the common PTE checking logic into a new shared helper,
thp_collapse_check_pte().Suggested-by David Hildenbrand <[email protected]>Signed-off-by Lance Yang <[email protected]>--- mm/khugepaged.c | 167 ++++++++++++++++++++++++++++++------------------ 1 file changed, 104 insertions(+), 63 deletions(-)mm/khugepaged.c +24
-0@@ -61,6 +61,12 @@ enum scan_result { SCAN_PAGE_FILLED, }; +enum pte_check_result { +PTE_CHECK_SUCCEED , +PTE_CHECK_CONTINUE , +PTE_CHECK_FAIL , +}; +#define CREATE_TRACE_POINTS #include <trace/events/huge_memory.h> @@ -533,6 +539,87 @@ static void release_pte_pages(pte_t *pte, pte_t *_pte, } } +/* +* thp_collapse_check_pte - Check if a PTE is suitable for THP collapse +* @pte: PTE to check +* @vma: VMA the PTE belongs to +* @cc: Collapse control settings +* @scan_swap_pte: Allow scanning of swap PTEs if true +* @none_or_zero: Counter for none/zero PTEs (must be non-NULL) +* @unmapped: Counter for swap PTEs (must be non-NULL if scan_swap_pte +* is true) +* @scan_result: Used to return the failure reason (SCAN_*) on a +* PTE_CHECK_FAIL return. Must be non-NULL +* +* Returns: +* PTE_CHECK_SUCCEED - Valid PTE, proceed with collapse +* PTE_CHECK_CONTINUE - Skip this none/zero PTE but continue scanning +* PTE_CHECK_FAIL - Abort collapse scan +*/ +static inline int thp_collapse_check_pte(pte_t pte, struct vm_area_struct *vma,
Quoted 8 lines
mm/khugepaged.c static void release_pte_pages(pte_t *pte, pte_t *_pte, · line 564
Quoted 5 lines
mm/khugepaged.c +5
-0+ struct collapse_control *cc, bool scan_swap_pte, + int *none_or_zero, int *unmapped, int *scan_result) +{ + VM_BUG_ON(!none_or_zero || !scan_result); +VM_BUG_ON (scan_swap_pte && !unmapped);Please use
VM_WARN_ONCE_ONand just return. Or put all stats in a struct, pass and check the struct pointer instead.
Yes, will use VM_WARN_ONCE_ON instead of VM_BUG_ON, and just return PTE_CHECK_FAIL.
Thanks,
Lance
Quoted 4 lines
Best Regards,
Yan, Zi