mm/khugepaged: optimize collapse candidate detection
Subsystems (5)
Lance Yang
reviewercover mm/khugepaged: optimize collapse candidate detection
Hi all,
This series contains a couple of small optimizations for the scanner. The idea is to detect unsuitable collapse candidates, like mlocked VMAs or guard PTEs, earlier in the scan and bail out to avoid wasted work ;)
Thanks,
Lance
Lance Yang (3):
mm/khugepaged: skip unsuitable VMAs earlier in
khugepaged_scan_mm_slot()
mm: clean up and expose is_guard_pte_marker() mm/khugepaged: abort collapse scan on guard PTEs
include/linux/mm.h | 6 +++++- include/linux/swapops.h | 6 ++++++ mm/huge_memory.c | 2 +- mm/khugepaged.c | 26 +++++++++++++++++++++++++- mm/madvise.c | 6 ------ 5 files changed, 37 insertions(+), 9 deletions(-)
Signature · 1 line
2.49.0
Lance Yang
reviewerPATCH 1/3 mm/khugepaged: skip unsuitable VMAs earlier in khugepaged_scan_mm_slot()
From: Lance Yang <[email protected]>
Let's skip unsuitable VMAs early in the khugepaged scan; specifically, mlocked VMAs should not be touched.
Note that the only other user of the VM_NO_KHUGEPAGED mask is
__thp_vma_allowable_orders(), which is also used by the MADV_COLLAPSE
path. Since MADV_COLLAPSE has different rules (e.g., for mlocked VMAs), we cannot simply make the shared mask stricter as that would break it.
So, we also introduce a new VM_NO_THP_COLLAPSE mask for that helper, leaving the stricter checks to be applied only within the khugepaged path itself.
--- include/linux/mm.h | 6 +++++- mm/huge_memory.c | 2 +- mm/khugepaged.c | 14 +++++++++++++- 3 files changed, 19 insertions(+), 3 deletions(-)
include/linux/mm.h +5 -1
@@ -505,7 +505,11 @@ extern unsigned int kobjsize(const void *objp);#define VM_REMAP_FLAGS (VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP)/* This mask prevents VMA from being scanned with khugepaged */ -#define VM_NO_KHUGEPAGED (VM_SPECIAL | VM_HUGETLB) +#define VM_NO_KHUGEPAGED \ + (VM_SPECIAL | VM_HUGETLB | VM_LOCKED_MASK | VM_NOHUGEPAGE) + +/* This mask prevents VMA from being collapsed by any THP path */ +#define VM_NO_THP_COLLAPSE (VM_SPECIAL | VM_HUGETLB)/* This mask defines which mm->def_flags a process can inherit its parent */ #define VM_INIT_DEF_MASK VM_NOHUGEPAGE
mm/huge_memory.c +1 -1
@@ -134,7 +134,7 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,* Must be checked after dax since some dax mappings may have * VM_MIXEDMAP set. */ -if (!in_pf && !smaps && (vm_flags &VM_NO_KHUGEPAGED )) +if (!in_pf && !smaps && (vm_flags &VM_NO_THP_COLLAPSE ))return 0 ;/*
mm/khugepaged.c +13 -1
@@ -345,6 +345,17 @@ struct attribute_group khugepaged_attr_group = { };#endif /* CONFIG_SYSFS */ +/** +* khugepaged_should_scan_vma - check if a VMA is a candidate for collapse +* @vm_flags: The flags of the VMA to check. +* +* Returns: true if the VMA should be scanned by khugepaged, false otherwise. +*/ +static inline bool khugepaged_should_scan_vma (vm_flags_t vm_flags) +{ +return !(vm_flags &VM_NO_KHUGEPAGED ); +} +int hugepage_madvise (struct vm_area_struct *vma,vm_flags_t *vm_flags,int advice) {
@@ -2443,7 +2454,8 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, progress++;break ; } -if (!thp_vma_allowable_order (vma, vma->vm_flags,TVA_KHUGEPAGED ,PMD_ORDER )) { +if (!khugepaged_should_scan_vma (vma->vm_flags) || + !thp_vma_allowable_order (vma, vma->vm_flags,TVA_KHUGEPAGED ,PMD_ORDER )) { skip: progress++;continue ;
Signature · 1 line
2.49.0
Lance Yang
reviewerPATCH 2/3 mm: clean up and expose is_guard_pte_marker()
In reply to Lance YangFrom: Lance Yang <[email protected]>
is_guard_pte_marker() performs a redundant check because it calls both is_pte_marker() and is_guard_swp_entry(), both of which internally check for a PTE marker.
is_guard_pte_marker()
|- is_pte_marker()
| `- is_pte_marker_entry() // First check
`- is_guard_swp_entry()
`- is_pte_marker_entry() // Second, redundant check
While a modern compiler could likely optimize this away, let's have clean code and not rely on it ;)
Also, make it available for hugepage collapsing code.
--- include/linux/swapops.h | 6 ++++++ mm/madvise.c | 6 ------ 2 files changed, 6 insertions(+), 6 deletions(-)
include/linux/swapops.h +6 -0
@@ -469,6 +469,12 @@ static inline int is_guard_swp_entry(swp_entry_t entry) (pte_marker_get (entry) &PTE_MARKER_GUARD ); } +static inline bool is_guard_pte_marker (pte_t ptent) +{ +return is_swap_pte (ptent) && +is_guard_swp_entry (pte_to_swp_entry (ptent)); +} +/* * This is a special version to check pte_none() just to cover the case when * the pte is a pte marker. It existed because in many cases the pte marker
mm/madvise.c +0 -6
@@ -1069,12 +1069,6 @@ static bool is_valid_guard_vma(struct vm_area_struct *vma, bool allow_locked)return !(vma->vm_flags & disallowed); } -static bool is_guard_pte_marker (pte_t ptent) -{ -return is_pte_marker (ptent) && -is_guard_swp_entry (pte_to_swp_entry (ptent)); -} -static int guard_install_pud_entry (pud_t *pud,unsigned long addr,unsigned long next,struct mm_walk *walk) {
Signature · 1 line
2.49.0
Lance Yang
reviewerPATCH 3/3 mm/khugepaged: abort collapse scan on guard PTEs
In reply to Lance YangFrom: Lance Yang <[email protected]>
Guard PTE markers are installed via MADV_GUARD_INSTALL to create lightweight guard regions.
Currently, any collapse path (khugepaged or MADV_COLLAPSE) will fail when encountering such a range.
MADV_COLLAPSE fails deep inside the collapse logic when trying to swap-in the special marker in __collapse_huge_page_swapin().
hpage_collapse_scan_pmd()
`- collapse_huge_page()
`- __collapse_huge_page_swapin() -> fails!
khugepaged's behavior is slightly different due to its max_ptes_swap limit (default 64). It won't fail as deep, but it will still needlessly scan up to 64 swap entries before bailing out.
IMHO, we can and should detect this much earlier ;)
This patch adds a check directly inside the PTE scan loop. If a guard marker is found, the scan is aborted immediately with a new SCAN_PTE_GUARD status, avoiding wasted work.
--- mm/khugepaged.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
mm/khugepaged.c +12 -0
@@ -59,6 +59,7 @@ enum scan_result {SCAN_STORE_FAILED ,SCAN_COPY_MC ,SCAN_PAGE_FILLED , + SCAN_PTE_GUARD, };#define CREATE_TRACE_POINTS
@@ -1317,6 +1318,16 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, result =SCAN_PTE_UFFD_WP ; goto out_unmap; } +/* +* Guard PTE markers are installed by +* MADV_GUARD_INSTALL. Any collapse path must +* not touch them, so abort the scan immediately +* if one is found. +*/ +if (is_guard_pte_marker (pteval)) { + result =SCAN_PTE_GUARD ; + goto out_unmap; + }continue ; }else { result =SCAN_EXCEED_SWAP_PTE ;
@@ -2860,6 +2871,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,case SCAN_PAGE_COMPOUND :case SCAN_PAGE_LRU :case SCAN_DEL_PAGE_LRU : +case SCAN_PTE_GUARD : last_fail = result;break ;default :
Signature · 1 line
2.49.0
Dev Jain
reviewerRe: [PATCH mm-new 1/3] mm/khugepaged: skip unsuitable VMAs earlier in khugepaged_scan_mm_slot()
In reply to Lance YangQuoted 26 lines · Lance Yang
From: Lance Yang <[email protected]>
Let's skip unsuitable VMAs early in the khugepaged scan; specifically, mlocked VMAs should not be touched.
Note that the only other user of the
VM_NO_KHUGEPAGEDmask is
__thp_vma_allowable_orders(), which is also used by theMADV_COLLAPSE
path. SinceMADV_COLLAPSEhas different rules (e.g., for mlocked VMAs), we cannot simply make the shared mask stricter as that would break it.So, we also introduce a new
VM_NO_THP_COLLAPSEmask for that helper, leaving the stricter checks to be applied only within the khugepaged path itself.Signed-off-by Lance Yang <[email protected]>--- include/linux/mm.h | 6 +++++- mm/huge_memory.c | 2 +- mm/khugepaged.c | 14 +++++++++++++- 3 files changed, 19 insertions(+), 3 deletions(-)include/linux/mm.h +0
-0
include/linux/mm.h extern unsigned int kobjsize(const void *objp); · line 512
Quoted 9 lines
include/linux/mm.h +5
-1@@ -505,7 +505,11 @@ extern unsigned int kobjsize(const void *objp);#define VM_REMAP_FLAGS (VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP)/* This mask prevents VMA from being scanned with khugepaged */ -#define VM_NO_KHUGEPAGED (VM_SPECIAL | VM_HUGETLB) +#define VM_NO_KHUGEPAGED \ + (VM_SPECIAL | VM_HUGETLB | VM_LOCKED_MASK | VM_NOHUGEPAGE) + +/* This mask prevents VMA from being collapsed by any THP path */ +#define VM_NO_THP_COLLAPSE (VM_SPECIAL | VM_HUGETLB)
VM_NO_KHUGEPAGED should then be defined as VM_NO_THP_COLLAPSE | VM_LOCKED_MASK | VM_NOHUGEPAGE. But...
I believe that the eligibility checking for khugepaged collapse is the business of thp_vma_allowable_order(). This functionality should be put there, we literally have a TVA_KHUGEPAGED flag :)
Quoted 48 lines
include/linux/mm.h +0
-0/* This mask defines which mm->def_flags a process can inherit its parent */ #define VM_INIT_DEF_MASK VM_NOHUGEPAGEmm/huge_memory.c +1
-1@@ -134,7 +134,7 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,* Must be checked after dax since some dax mappings may have * VM_MIXEDMAP set. */ -if (!in_pf && !smaps && (vm_flags &VM_NO_KHUGEPAGED )) +if (!in_pf && !smaps && (vm_flags &VM_NO_THP_COLLAPSE ))return 0 ;/* mm/khugepaged.c +13
-1@@ -345,6 +345,17 @@ struct attribute_group khugepaged_attr_group = { };#endif /* CONFIG_SYSFS */ +/** +* khugepaged_should_scan_vma - check if a VMA is a candidate for collapse +* @vm_flags: The flags of the VMA to check. +* +* Returns: true if the VMA should be scanned by khugepaged, false otherwise. +*/ +static inline bool khugepaged_should_scan_vma (vm_flags_t vm_flags) +{ +return !(vm_flags &VM_NO_KHUGEPAGED ); +} +int hugepage_madvise (struct vm_area_struct *vma,vm_flags_t *vm_flags,int advice) {@@ -2443,7 +2454,8 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, progress++;break ; } -if (!thp_vma_allowable_order (vma, vma->vm_flags,TVA_KHUGEPAGED ,PMD_ORDER )) { +if (!khugepaged_should_scan_vma (vma->vm_flags) || + !thp_vma_allowable_order (vma, vma->vm_flags,TVA_KHUGEPAGED ,PMD_ORDER )) { skip: progress++;continue ;
Dev Jain
reviewerRe: [PATCH mm-new 2/3] mm: clean up and expose is_guard_pte_marker()
In reply to Lance YangQuoted 12 lines · Lance Yang
From: Lance Yang <[email protected]>
is_guard_pte_marker()performs a redundant check because it calls bothis_pte_marker()andis_guard_swp_entry(), both of which internally check for a PTE marker.
is_guard_pte_marker()
|-is_pte_marker()
| `-is_pte_marker_entry()// First check
`-is_guard_swp_entry()
`-is_pte_marker_entry()// Second, redundant check
Quoted 4 lines
While a modern compiler could likely optimize this away, let's have clean code and not rely on it ;)
Also, make it available for hugepage collapsing code.
The movement of the code should be merged with the next patch.
Quoted 42 lines
Cc Kairui Song <[email protected]>Signed-off-by Lance Yang <[email protected]>--- include/linux/swapops.h | 6 ++++++ mm/madvise.c | 6 ------ 2 files changed, 6 insertions(+), 6 deletions(-)include/linux/swapops.h +6
-0@@ -469,6 +469,12 @@ static inline int is_guard_swp_entry(swp_entry_t entry) (pte_marker_get (entry) &PTE_MARKER_GUARD ); } +static inline bool is_guard_pte_marker (pte_t ptent) +{ +return is_swap_pte (ptent) && +is_guard_swp_entry (pte_to_swp_entry (ptent)); +} +/* * This is a special version to check pte_none() just to cover the case when * the pte is a pte marker. It existed because in many cases the pte marker mm/madvise.c +0
-6@@ -1069,12 +1069,6 @@ static bool is_valid_guard_vma(struct vm_area_struct *vma, bool allow_locked)return !(vma->vm_flags & disallowed); } -static bool is_guard_pte_marker (pte_t ptent) -{ -return is_pte_marker (ptent) && -is_guard_swp_entry (pte_to_swp_entry (ptent)); -} -static int guard_install_pud_entry (pud_t *pud,unsigned long addr,unsigned long next,struct mm_walk *walk) {
Dev Jain
reviewerRe: [PATCH mm-new 3/3] mm/khugepaged: abort collapse scan on guard PTEs
In reply to Lance YangQuoted 47 lines · Lance Yang
From: Lance Yang <[email protected]>
Guard PTE markers are installed via
MADV_GUARD_INSTALLto create lightweight guard regions.Currently, any collapse path (khugepaged or
MADV_COLLAPSE) will fail when encountering such a range.
MADV_COLLAPSEfails deep inside the collapse logic when trying to swap-in the special marker in__collapse_huge_page_swapin().
hpage_collapse_scan_pmd()
`-collapse_huge_page()`- __collapse_huge_page_swapin() -> fails!khugepaged's behavior is slightly different due to its
max_ptes_swaplimit (default 64). It won't fail as deep, but it will still needlessly scan up to 64 swap entries before bailing out.IMHO, we can and should detect this much earlier ;)
This patch adds a check directly inside the PTE scan loop. If a guard marker is found, the scan is aborted immediately with a new
SCAN_PTE_GUARDstatus, avoiding wasted work.Signed-off-by Lance Yang <[email protected]>--- mm/khugepaged.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)mm/khugepaged.c +2
-0@@ -59,6 +59,7 @@ enum scan_result {SCAN_STORE_FAILED ,SCAN_COPY_MC ,SCAN_PAGE_FILLED , + SCAN_PTE_GUARD, };#define CREATE_TRACE_POINTS @@ -1317,6 +1318,16 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, result =SCAN_PTE_UFFD_WP ; goto out_unmap; } +/*
mm/khugepaged.c static int hpage_collapse_scan_pmd(struct mm_struct *mm, · line 1331
Quoted 10 lines
mm/khugepaged.c +9
-0+* Guard PTE markers are installed by +* MADV_GUARD_INSTALL. Any collapse path must +* not touch them, so abort the scan immediately +* if one is found. +*/ +if (is_guard_pte_marker (pteval)) { + result =SCAN_PTE_GUARD ; + goto out_unmap; + }continue ;
This looks good, but see below.
mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2875
Quoted 8 lines
mm/khugepaged.c +1
-0}else { result =SCAN_EXCEED_SWAP_PTE ;@@ -2860,6 +2871,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,case SCAN_PAGE_COMPOUND :case SCAN_PAGE_LRU :case SCAN_DEL_PAGE_LRU : +case SCAN_PTE_GUARD : last_fail = result;
Should we not do this, and just send this case over to the default case. That would mean immediate exit with -EINVAL, instead of iterating over the complete range, potentially collapsing a non-guard range, and returning -EINVAL. I do not think we should spend a significant time in the kernel when the user is literally invoking madvise(MADV_GUARD_INSTALL) and madvise(MADV_COLLAPSE) on overlapping regions.
Quoted 2 lines
mm/khugepaged.c +0
-0break ;default :
Lance Yang
reviewerRe: [PATCH mm-new 1/3] mm/khugepaged: skip unsuitable VMAs earlier in khugepaged_scan_mm_slot()
In reply to Dev JainHey Dev,
Thanks for taking time to review!
Quoted 23 lines · Dev Jain
Quoted 21 lines · Lance Yang
From: Lance Yang <[email protected]>
Let's skip unsuitable VMAs early in the khugepaged scan; specifically, mlocked VMAs should not be touched.
Note that the only other user of the
VM_NO_KHUGEPAGEDmask is__thp_vma_allowable_orders(), which is also used by theMADV_COLLAPSEpath. SinceMADV_COLLAPSEhas different rules (e.g., for mlocked VMAs), we cannot simply make the shared mask stricter as that would break it.So, we also introduce a new
VM_NO_THP_COLLAPSEmask for that helper, leaving the stricter checks to be applied only within the khugepaged path itself.Signed-off-by Lance Yang <[email protected]>---
include/linux/mm.h| 6 +++++-
mm/huge_memory.c | 2 +-
mm/khugepaged.c | 14 +++++++++++++-3 files changed, 19 insertions(+), 3 deletions(-)
Quoted 15 lines
include/linux/mm.h extern unsigned int kobjsize(const void *objp); · line 511
Quoted 13 lines
include/linux/mm.h +5
-1@@ -505,7 +505,11 @@ extern unsigned int kobjsize(const void *objp); #define VM_REMAP_FLAGS (VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP) /* This mask prevents VMA from being scanned with khugepaged */ -#define VM_NO_KHUGEPAGED (VM_SPECIAL | VM_HUGETLB) +#define VM_NO_KHUGEPAGED \ + (VM_SPECIAL | VM_HUGETLB | VM_LOCKED_MASK | VM_NOHUGEPAGE) + +/* This mask prevents VMA from being collapsed by any THP path */ +#define VM_NO_THP_COLLAPSE (VM_SPECIAL | VM_HUGETLB)
VM_NO_KHUGEPAGEDshould then be defined asVM_NO_THP_COLLAPSE|VM_LOCKED_MASK|VM_NOHUGEPAGE.
Yep, it's a good cleanup ;)
Quoted 5 lines
But...
I believe that the eligibility checking for khugepaged collapse is the business of
thp_vma_allowable_order(). This functionality should be put there, we literally have aTVA_KHUGEPAGEDflag :)
Good spot. That's a much better apporach!
My initial thinking was to keep thp_vma_allowable_order() as generic as possible, avoiding specific checks for individual callers ;)
BUT you are right, the TVA_KHUGEPAGED flag is only passed from the khugepaged path, so the compiler will optimize out the branch for other callers, leaving no runtime overhead.
Will rework this patch for v2 as your suggestion!
Thanks,
Lance
Quoted 52 lines
Quoted 52 lines
include/linux/mm.h +0
-0/* This mask defines which mm->def_flags a process can inherit its parent */ #define VM_INIT_DEF_MASK VM_NOHUGEPAGEmm/huge_memory.c +1
-1@@ -134,7 +134,7 @@ unsigned long __thp_vma_allowable_orders(structvm_area_struct *vma, * Must be checked after dax since some dax mappings may have * VM_MIXEDMAP set. */ - if (!in_pf && !smaps && (vm_flags & VM_NO_KHUGEPAGED)) + if (!in_pf && !smaps && (vm_flags & VM_NO_THP_COLLAPSE)) return 0; /*mm/khugepaged.c +2
-0@@ -345,6 +345,17 @@ struct attribute_group khugepaged_attr_group = { }; #endif /* CONFIG_SYSFS */ +/** +* khugepaged_should_scan_vma - check if a VMA is a candidate for collapse
+ * @vm_flags: The flags of the VMA to check.
+ *
+ * Returns: true if the VMA should be scanned by khugepaged, false otherwise.
+ */mm/khugepaged.c +5
-0+static inline bool khugepaged_should_scan_vma (vm_flags_t vm_flags) +{ + return !(vm_flags & VM_NO_KHUGEPAGED); +} +int
hugepage_madvise(struct vm_area_struct *vma, vm_flags_t *vm_flags, int advice){mm/khugepaged.c +2
-1@@ -2443,7 +2454,8 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, progress++; break; } - if (!thp_vma_allowable_order(vma, vma->vm_flags, TVA_KHUGEPAGED, PMD_ORDER)) { + if (!khugepaged_should_scan_vma(vma->vm_flags) || + !thp_vma_allowable_order(vma, vma->vm_flags,TVA_KHUGEPAGED ,PMD_ORDER )) {skip:
progress++;
continue;
Lance Yang
reviewerRe: [PATCH mm-new 3/3] mm/khugepaged: abort collapse scan on guard PTEs
In reply to Dev JainQuoted 62 lines · Dev Jain
Quoted 47 lines · Lance Yang
From: Lance Yang <[email protected]>
Guard PTE markers are installed via
MADV_GUARD_INSTALLto create lightweight guard regions.Currently, any collapse path (khugepaged or
MADV_COLLAPSE) will fail when encountering such a range.
MADV_COLLAPSEfails deep inside the collapse logic when trying to swap-in the special marker in__collapse_huge_page_swapin().
hpage_collapse_scan_pmd()
`-collapse_huge_page()
`-__collapse_huge_page_swapin()-> fails!khugepaged's behavior is slightly different due to its
max_ptes_swaplimit (default 64). It won't fail as deep, but it will still needlessly scan up to 64 swap entries before bailing out.IMHO, we can and should detect this much earlier ;)
This patch adds a check directly inside the PTE scan loop. If a guard marker is found, the scan is aborted immediately with a new
SCAN_PTE_GUARDstatus, avoiding wasted work.Signed-off-by Lance Yang <[email protected]>---
mm/khugepaged.c| 12 ++++++++++++1 file changed, 12 insertions(+)mm/khugepaged.c +2
-0@@ -59,6 +59,7 @@ enum scan_result { SCAN_STORE_FAILED, SCAN_COPY_MC, SCAN_PAGE_FILLED, + SCAN_PTE_GUARD, }; #define CREATE_TRACE_POINTS@@ -1317,6 +1318,16 @@ static int hpage_collapse_scan_pmd(structmm_struct *mm, result = SCAN_PTE_UFFD_WP; goto out_unmap; } + /*mm/khugepaged.c static int hpage_collapse_scan_pmd(struct · line 1332
Quoted 10 lines
mm/khugepaged.c +9
-0+* Guard PTE markers are installed by +* MADV_GUARD_INSTALL. Any collapse path must +* not touch them, so abort the scan immediately +* if one is found. +*/ + if (is_guard_pte_marker(pteval)) { + result = SCAN_PTE_GUARD; + goto out_unmap; + } continue;This looks good, but see below.
Quoted 13 lines
Quoted 9 lines
mm/khugepaged.c +0
-0} else {result =
SCAN_EXCEED_SWAP_PTE;mm/khugepaged.c +0
-0@@ -2860,6 +2871,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,case
SCAN_PAGE_COMPOUND:
caseSCAN_PAGE_LRU:
caseSCAN_DEL_PAGE_LRU:
+ caseSCAN_PTE_GUARD:
last_fail= result;Should we not do this, and just send this case over to the default case. That would mean immediate exit with
-EINVAL, instead of iterating over the complete range, potentially collapsing a non-guard range, and returning-EINVAL.
That makes sense to me ;)
Quoted 3 lines
I do not
think we should spend a significant time in the kernel when the user is literally invokingmadvise(MADV_GUARD_INSTALL)andmadvise(MADV_COLLAPSE)on overlapping regions.
I'm just a bit unsure because the MADV_COLLAPSE man page[1] describes it as a "best-effort" collapse. This patch follows that idea, collapsing what it can.
MADV_COLLAPSE (since Linux 6.1)
Perform a best-effort synchronous collapse of the native pages mapped by the memory range into Transparent Huge Pages (THPs). MADV_COLLAPSE operates on the current state of memory of the calling process and makes no persistent changes or guarantees on how pages will be mapped, constructed, or faulted in the future.
A hard-fail on a guard PTE marker might go against that.
Well, I'm open to either approach. What do other folks think?
[1] https://man7.org/linux/man-pages/man2/madvise.2.html
Cheers,
Lance
Quoted 3 lines
Quoted 3 lines
break;
default:
Lance Yang
reviewerRe: [PATCH mm-new 2/3] mm: clean up and expose is_guard_pte_marker()
In reply to Dev JainQuoted 14 lines · Dev Jain
Quoted 12 lines · Lance Yang
From: Lance Yang <[email protected]>
is_guard_pte_marker()performs a redundant check because it calls bothis_pte_marker()andis_guard_swp_entry(), both of which internally check for a PTE marker.
is_guard_pte_marker()
|-is_pte_marker()
| `- is_pte_marker_entry() // First check
`-is_guard_swp_entry()
`- is_pte_marker_entry() // Second, redundant check
Quoted 6 lines
Quoted 4 lines
While a modern compiler could likely optimize this away, let's have clean code and not rely on it ;)
Also, make it available for hugepage collapsing code.
The movement of the code should be merged with the next patch.
Thanks for the suggestion ;)
I'd prefer to keep them as separate patches to make them easier to review, if that's okay.
Cheers,
Lance
Quoted 43 lines
Quoted 43 lines
Cc Kairui Song <[email protected]>Signed-off-by Lance Yang <[email protected]>---
include/linux/swapops.h| 6 ++++++
mm/madvise.c | 6 ------2 files changed, 6 insertions(+), 6 deletions(-)include/linux/swapops.h +6
-0@@ -469,6 +469,12 @@ static inline int is_guard_swp_entry(swp_entry_t entry) (pte_marker_get(entry) & PTE_MARKER_GUARD); } +static inline bool is_guard_pte_marker (pte_t ptent) +{ + return is_swap_pte(ptent) && + is_guard_swp_entry(pte_to_swp_entry(ptent)); +} + /** This is a special version to check pte_none() just to cover the case when * the pte is a pte marker. It existed because in many cases the pte marker mm/madvise.c +0
-6@@ -1069,12 +1069,6 @@ static bool is_valid_guard_vma(struct vm_area_struct *vma, bool allow_locked) return !(vma->vm_flags & disallowed); } -static bool is_guard_pte_marker (pte_t ptent) -{ - return is_pte_marker(ptent) && - is_guard_swp_entry(pte_to_swp_entry(ptent)); -} - static int guard_install_pud_entry(pud_t *pud, unsigned long addr, unsigned long next, struct mm_walk *walk) {
Re: [PATCH mm-new 2/3] mm: clean up and expose is_guard_pte_marker()
In reply to Lance YangQuoted 6 lines · Lance Yang
From: Lance Yang <[email protected]>
is_guard_pte_marker()performs a redundant check because it calls bothis_pte_marker()andis_guard_swp_entry(), both of which internally check for a PTE marker.
Quoted 6 lines
is_guard_pte_marker()
|-is_pte_marker()
| `-is_pte_marker_entry()// First check
`-is_guard_swp_entry()
`-is_pte_marker_entry()// Second, redundant check
I mean, it expands to:
is_swap_pte(pte) && is_pte_marker_entry(pte_to_swp_entry(pte)) && is_pte_marker_entry(pte_to_swp_entry(pte))
So I don't think it's really unreasonable to expect compiler magic here...
But you're right that I should have just used is_swap_pte() really, it's a bit silly not to, so this is fine :)
Quoted 2 lines
While a modern compiler could likely optimize this away, let's have clean code and not rely on it ;)
Please don't put smileys in commit messages :) as cute as they are, this is going on the permanent kernel record and while we all love them, it's probably not the best place to put them :P
Quoted 2 lines
Also, make it available for hugepage collapsing code.
Nit but put a newline after this.
I think probably if I'm really really nitty I'd say that you should put this bit first, as it's the primary motivation for the change, and put the refactoring stuff after.
Quoted 2 lines
Cc Kairui Song <[email protected]>Signed-off-by Lance Yang <[email protected]>
This seems fine to me, so:
Quoted 42 lines
--- include/linux/swapops.h | 6 ++++++ mm/madvise.c | 6 ------ 2 files changed, 6 insertions(+), 6 deletions(-)include/linux/swapops.h +6
-0@@ -469,6 +469,12 @@ static inline int is_guard_swp_entry(swp_entry_t entry) (pte_marker_get (entry) &PTE_MARKER_GUARD ); } +static inline bool is_guard_pte_marker (pte_t ptent) +{ +return is_swap_pte (ptent) && +is_guard_swp_entry (pte_to_swp_entry (ptent)); +} +/* * This is a special version to check pte_none() just to cover the case when * the pte is a pte marker. It existed because in many cases the pte marker mm/madvise.c +0
-7@@ -1069,12 +1069,6 @@ static bool is_valid_guard_vma(struct vm_area_struct *vma, bool allow_locked)return !(vma->vm_flags & disallowed); } -static bool is_guard_pte_marker (pte_t ptent) -{ -return is_pte_marker (ptent) && -is_guard_swp_entry (pte_to_swp_entry (ptent)); -} -static int guard_install_pud_entry (pud_t *pud,unsigned long addr,unsigned long next,struct mm_walk *walk) { --2.49.0
Re: [PATCH mm-new 3/3] mm/khugepaged: abort collapse scan on guard PTEs
In reply to Lance YangQuoted 13 lines · Lance Yang
From: Lance Yang <[email protected]>
Guard PTE markers are installed via
MADV_GUARD_INSTALLto create lightweight guard regions.Currently, any collapse path (khugepaged or
MADV_COLLAPSE) will fail when encountering such a range.
MADV_COLLAPSEfails deep inside the collapse logic when trying to swap-in the special marker in__collapse_huge_page_swapin().
hpage_collapse_scan_pmd()
`-collapse_huge_page()
Quoted 7 lines
`- __collapse_huge_page_swapin() -> fails!khugepaged's behavior is slightly different due to its
max_ptes_swaplimit (default 64). It won't fail as deep, but it will still needlessly scan up to 64 swap entries before bailing out.IMHO, we can and should detect this much earlier ;)
No smileys in commit messages please... :)
Quoted 15 lines
This patch adds a check directly inside the PTE scan loop. If a guard marker is found, the scan is aborted immediately with a new
SCAN_PTE_GUARDstatus, avoiding wasted work.Signed-off-by Lance Yang <[email protected]>--- mm/khugepaged.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)mm/khugepaged.c +0
-0
mm/khugepaged.c enum scan_result { · line 62
Quoted 4 lines
mm/khugepaged.c +1
-0@@ -59,6 +59,7 @@ enum scan_result {SCAN_STORE_FAILED ,SCAN_COPY_MC ,SCAN_PAGE_FILLED , + SCAN_PTE_GUARD,
I wonder if we really need to have it literally called out though, can we just use:
SCAN_PTE_NON_PRESENT
Instead?
As it is, indeed, non-present :)
Quoted 29 lines
mm/khugepaged.c +11
-1};#define CREATE_TRACE_POINTS @@ -1317,6 +1318,16 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, result =SCAN_PTE_UFFD_WP ; goto out_unmap; } +/* +* Guard PTE markers are installed by +* MADV_GUARD_INSTALL. Any collapse path must +* not touch them, so abort the scan immediately +* if one is found. +*/ +if (is_guard_pte_marker (pteval)) { + result =SCAN_PTE_GUARD ; + goto out_unmap; + }continue ; }else { result =SCAN_EXCEED_SWAP_PTE ;@@ -2860,6 +2871,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,case SCAN_PAGE_COMPOUND :case SCAN_PAGE_LRU :case SCAN_DEL_PAGE_LRU : +case SCAN_PTE_GUARD : last_fail = result;break ;default : --
Quoted 2 lines
2.49.0
Cheers, Lorenzo
Lance Yang
reviewerRe: [PATCH mm-new 2/3] mm: clean up and expose is_guard_pte_marker()
In reply to Lorenzo StoakesHi Lorenzo,
Thanks for taking time to review!
Quoted 14 lines · Lorenzo Stoakes
Quoted 6 lines · Lance Yang
From: Lance Yang <[email protected]>
is_guard_pte_marker()performs a redundant check because it calls bothis_pte_marker()andis_guard_swp_entry(), both of which internally check for a PTE marker.Quoted 6 lines
is_guard_pte_marker()
|-is_pte_marker()
| `-is_pte_marker_entry()// First check
`-is_guard_swp_entry()
`-is_pte_marker_entry()// Second, redundant check
Quoted 4 lines
I mean, it expands to:
is_swap_pte(pte)&& is_pte_marker_entry(pte_to_swp_entry(pte)) && is_pte_marker_entry(pte_to_swp_entry(pte))
Yes, that's a much clearer way to lay it out ;)
Quoted 5 lines
So I don't think it's really unreasonable to expect compiler magic here...
But you're right that I should have just used
is_swap_pte()really, it's a bit silly not to, so this is fine :)
Exactly. Glad we're on the same page!
Quoted 4 lines
Quoted 3 lines
While a modern compiler could likely optimize this away, let's have clean code and not rely on it ;)
Quoted 8 lines
Please don't put smileys in commit messages :) as cute as they are, this is going on the permanent kernel record and while we all love them, it's probably not the best place to put them :P
Quoted 2 lines
Also, make it available for hugepage collapsing code.
Nit but put a newline after this.
Got it. Will fix up all nits in v2.
Quoted 4 lines
I think probably if I'm really really nitty I'd say that you should put this bit first, as it's the primary motivation for the change, and put the refactoring stuff after.
Ah, right. The motivation for exposing the helper should come first. I'll reorder this changelog in v2.
Quoted 7 lines
Quoted 3 lines
Cc Kairui Song <[email protected]>Signed-off-by Lance Yang <[email protected]>This seems fine to me, so:
Reviewed-by Lorenzo Stoakes <[email protected]>
Thanks,
Lance
Quoted 43 lines
Quoted 43 lines
--- include/linux/swapops.h | 6 ++++++ mm/madvise.c | 6 ------ 2 files changed, 6 insertions(+), 6 deletions(-)include/linux/swapops.h +6
-0@@ -469,6 +469,12 @@ static inline int is_guard_swp_entry(swp_entry_t entry) (pte_marker_get (entry) &PTE_MARKER_GUARD ); } +static inline bool is_guard_pte_marker (pte_t ptent) +{ +return is_swap_pte (ptent) && +is_guard_swp_entry (pte_to_swp_entry (ptent)); +} +/* * This is a special version to check pte_none() just to cover the case when * the pte is a pte marker. It existed because in many cases the pte marker mm/madvise.c +0
-7@@ -1069,12 +1069,6 @@ static bool is_valid_guard_vma(struct vm_area_struct *vma, bool allow_locked)return !(vma->vm_flags & disallowed); } -static bool is_guard_pte_marker (pte_t ptent) -{ -return is_pte_marker (ptent) && -is_guard_swp_entry (pte_to_swp_entry (ptent)); -} -static int guard_install_pud_entry (pud_t *pud,unsigned long addr,unsigned long next,struct mm_walk *walk) { --2.49.0
Lance Yang
reviewerRe: [PATCH mm-new 3/3] mm/khugepaged: abort collapse scan on guard PTEs
In reply to Lorenzo StoakesQuoted 16 lines · Lorenzo Stoakes
Quoted 15 lines · Lance Yang
From: Lance Yang <[email protected]>
Guard PTE markers are installed via
MADV_GUARD_INSTALLto create lightweight guard regions.Currently, any collapse path (khugepaged or
MADV_COLLAPSE) will fail when encountering such a range.
MADV_COLLAPSEfails deep inside the collapse logic when trying to swap-in the special marker in__collapse_huge_page_swapin().
hpage_collapse_scan_pmd()
`-collapse_huge_page()`- __collapse_huge_page_swapin() -> fails!
Quoted 7 lines
Quoted 5 lines
khugepaged's behavior is slightly different due to its
max_ptes_swaplimit (default 64). It won't fail as deep, but it will still needlessly scan up to 64 swap entries before bailing out.IMHO, we can and should detect this much earlier ;)
No smileys in commit messages please... :)
Got it. Apparently, I'm a bit too fond of them ... Will remove it in v2.
Quoted 24 lines
Quoted 16 lines
This patch adds a check directly inside the PTE scan loop. If a guard marker is found, the scan is aborted immediately with a new
SCAN_PTE_GUARDstatus, avoiding wasted work.Signed-off-by Lance Yang <[email protected]>--- mm/khugepaged.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)mm/khugepaged.c +0
-0mm/khugepaged.c enum scan_result { · line 62
Quoted 4 lines
mm/khugepaged.c +1
-0@@ -59,6 +59,7 @@ enum scan_result {SCAN_STORE_FAILED ,SCAN_COPY_MC ,SCAN_PAGE_FILLED , + SCAN_PTE_GUARD,I wonder if we really need to have it literally called out though, can we just use:
Quoted 5 lines
SCAN_PTE_NON_PRESENTInstead?
As it is, indeed, non-present :)
Makes sense to me. A guard PTE is indeed a special non-present case.
So, let's reuse SCAN_PTE_NON_PRESENT for that case ;)
Cheers,
Lance
Quoted 34 lines
Quoted 30 lines
mm/khugepaged.c +11
-1};#define CREATE_TRACE_POINTS @@ -1317,6 +1318,16 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, result =SCAN_PTE_UFFD_WP ; goto out_unmap; } +/* +* Guard PTE markers are installed by +* MADV_GUARD_INSTALL. Any collapse path must +* not touch them, so abort the scan immediately +* if one is found. +*/ +if (is_guard_pte_marker (pteval)) { + result =SCAN_PTE_GUARD ; + goto out_unmap; + }continue ; }else { result =SCAN_EXCEED_SWAP_PTE ;@@ -2860,6 +2871,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,case SCAN_PAGE_COMPOUND :case SCAN_PAGE_LRU :case SCAN_DEL_PAGE_LRU : +case SCAN_PTE_GUARD : last_fail = result;break ;default : --Quoted 2 lines
2.49.0
Cheers, Lorenzo
Re: [PATCH mm-new 1/3] mm/khugepaged: skip unsuitable VMAs earlier in khugepaged_scan_mm_slot()
In reply to Lance YangQuoted 4 lines · Lance Yang
From: Lance Yang <[email protected]>
Let's skip unsuitable VMAs early in the khugepaged scan; specifically, mlocked VMAs should not be touched.
Why? That's a change in behaviour, isn't it?
I'm aware that hugepage collapse on an mlocked VMA can insert a fault latency, not universally welcome; but I've not seen discussion, let alone agreement, that current behaviour should be changed. Somewhere in yet-to-be-read mail? Please give us a link.
Hugh
Lance Yang
reviewerRe: [PATCH mm-new 1/3] mm/khugepaged: skip unsuitable VMAs earlier in khugepaged_scan_mm_slot()
Hi Hugh,
Thanks for taking a look and for raising this important point!
Quoted 7 lines · Hugh Dickins
Quoted 4 lines · Lance Yang
From: Lance Yang <[email protected]>
Let's skip unsuitable VMAs early in the khugepaged scan; specifically, mlocked VMAs should not be touched.
Quoted 8 lines
Why? That's a change in behaviour, isn't it?
I'm aware that hugepage collapse on an mlocked VMA can insert a fault latency, not universally welcome; but I've not seen discussion, let alone agreement, that current behaviour should be changed. Somewhere in yet-to-be-read mail? Please give us a link.
Hugh
You're right, this is indeed a change in behaviour. But it's specifically for khugepaged.
Users of mlock() expect low and predictable latency. THP collapse is a heavy operation that introduces exactly the kind of unpredictable delays they want to avoid. It has to unmap PTEs, copy data from the small folios to a new THP, and then remap the THP back to the PMD ;)
IMO, that change is acceptable because THP is generally transparent to users, and khugepaged does not guarantee when THP collapse or split will happen.
Well, we don't have a discussion on that, just something I noticed.
Thanks,
Lance
Re: [PATCH mm-new 1/3] mm/khugepaged: skip unsuitable VMAs earlier in khugepaged_scan_mm_slot()
Quoted 24 lines · Lance Yang
Hi Hugh,
Thanks for taking a look and for raising this important point!
Quoted 7 lines · Hugh Dickins
Quoted 4 lines · Lance Yang
From: Lance Yang <[email protected]>
Let's skip unsuitable VMAs early in the khugepaged scan; specifically, mlocked VMAs should not be touched.
Quoted 8 lines
Why? That's a change in behaviour, isn't it?
I'm aware that hugepage collapse on an mlocked VMA can insert a fault latency, not universally welcome; but I've not seen discussion, let alone agreement, that current behaviour should be changed. Somewhere in yet-to-be-read mail? Please give us a link.
Hugh
You're right, this is indeed a change in behaviour. But it's specifically for khugepaged.
Quoted 8 lines
Users of
mlock()expect low and predictable latency. THP collapse is a heavy operation that introduces exactly the kind of unpredictable delays they want to avoid. It has to unmap PTEs, copy data from the small folios to a new THP, and then remap the THP back to the PMD ;)IMO, that change is acceptable because THP is generally transparent to users, and khugepaged does not guarantee when THP collapse or split will happen.
I disagree. Many of those who have khugepaged enabled would prefer it to give them hugepages, even or especially on mlocked areas.
If you make that change, it must be guarded by a sysfs or sysctl tuning.
Perhaps it could share the sysctl_compact_unevictable_allowed tuning (I'm not sure whether that's a good or bad idea: opinions will differ).
Hugh
Quoted 5 lines
Well, we don't have a discussion on that, just something I noticed.
Thanks,
Lance
Lance Yang
reviewerRe: [PATCH mm-new 1/3] mm/khugepaged: skip unsuitable VMAs earlier in khugepaged_scan_mm_slot()
Quoted 31 lines · Hugh Dickins
Quoted 29 lines · Lance Yang
Hi Hugh,
Thanks for taking a look and for raising this important point!
Quoted 7 lines · Hugh Dickins
Quoted 4 lines · Lance Yang
From: Lance Yang <[email protected]>
Let's skip unsuitable VMAs early in the khugepaged scan; specifically, mlocked VMAs should not be touched.
Quoted 8 lines
Why? That's a change in behaviour, isn't it?
I'm aware that hugepage collapse on an mlocked VMA can insert a fault latency, not universally welcome; but I've not seen discussion, let alone agreement, that current behaviour should be changed. Somewhere in yet-to-be-read mail? Please give us a link.
Hugh
You're right, this is indeed a change in behaviour. But it's specifically for khugepaged.
Users of
mlock()expect low and predictable latency. THP collapse is a heavy operation that introduces exactly the kind of unpredictable delays they want to avoid. It has to unmap PTEs, copy data from the small folios to a new THP, and then remap the THP back to the PMD ;)
Quoted 8 lines
Quoted 3 lines
IMO, that change is acceptable because THP is generally transparent to users, and khugepaged does not guarantee when THP collapse or split will happen.
I disagree. Many of those who have khugepaged enabled would prefer it to give them hugepages, even or especially on mlocked areas.
If you make that change, it must be guarded by a sysfs or sysctl tuning.
Thanks for the feedback!
Well, seems like we're not on the same page. Let's gather more opinions from other folks ;)
Quoted 3 lines
Perhaps it could share the
sysctl_compact_unevictable_allowedtuning (I'm not sure whether that's a good or bad idea: opinions will differ).
Thanks,
Lance
Quoted 8 lines
Hugh
Quoted 5 lines
Well, we don't have a discussion on that, just something I noticed.
Thanks,
Lance
Re: [PATCH mm-new 1/3] mm/khugepaged: skip unsuitable VMAs earlier in khugepaged_scan_mm_slot()
In reply to Lance YangQuoted 4 lines · Lance Yang
Users of
mlock()expect low and predictable latency. THP collapse is a heavy operation that introduces exactly the kind of unpredictable delays they want to avoid. It has to unmap PTEs, copy data from the small folios to a new THP, and then remap the THP back to the PMD ;)
Generally, we allow minor page faults into mlocked VMAs and avoid major. This is minor page fault territory in my view.
Also it is very similar to what compaction does and we allow compaction of mlocked VMA by default, unless sysctl vm.compact_unevictable_allowed is set to zero.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Re: [PATCH mm-new 1/3] mm/khugepaged: skip unsuitable VMAs earlier in khugepaged_scan_mm_slot()
Quoted 8 lines · Kiryl Shutsemau
Quoted 4 lines · Lance Yang
Users of
mlock()expect low and predictable latency. THP collapse is a heavy operation that introduces exactly the kind of unpredictable delays they want to avoid. It has to unmap PTEs, copy data from the small folios to a new THP, and then remap the THP back to the PMD ;)Generally, we allow minor page faults into mlocked VMAs and avoid major. This is minor page fault territory in my view.
Hm, but we won't be causing minor faults via reclaim right, since they're not on any LRU?
Quoted 4 lines
Also it is very similar to what compaction does and we allow compaction of mlocked VMA by default, unless sysctl vm.compact_unevictable_allowed is set to zero.
This is a much stronger point.
I think we are sometimes too vague as to what mlock() means in totality. But given that we default allow compaction it seems sensible to keep this behaviour the same.
Unless you have a specific situation where this is problematic Lance?
Quoted 3 lines
--
Kiryl Shutsemau / Kirill A. Shutemov
Cheers, Lorenzo
Re: [PATCH mm-new 1/3] mm/khugepaged: skip unsuitable VMAs earlier in khugepaged_scan_mm_slot()
Quoted 7 lines · Lorenzo Stoakes
Quoted 6 lines · Kiryl Shutsemau
Quoted 4 lines · Lance Yang
Users of
mlock()expect low and predictable latency. THP collapse is a heavy operation that introduces exactly the kind of unpredictable delays they want to avoid. It has to unmap PTEs, copy data from the small folios to a new THP, and then remap the THP back to the PMD ;)
Quoted 5 lines
Quoted 2 lines
Generally, we allow minor page faults into mlocked VMAs and avoid major. This is minor page fault territory in my view.
Hm, but we won't be causing minor faults via reclaim right, since they're not on any LRU?
PTEs are still present when we do THP allocation. No reclaim while the access is blocked. We only block the access on copy and PTEs->PMD collapse.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Re: [PATCH mm-new 1/3] mm/khugepaged: skip unsuitable VMAs earlier in khugepaged_scan_mm_slot()
Quoted 11 lines · Kiryl Shutsemau
Quoted 10 lines · Lorenzo Stoakes
Quoted 8 lines · Kiryl Shutsemau
Quoted 4 lines · Lance Yang
Users of
mlock()expect low and predictable latency. THP collapse is a heavy operation that introduces exactly the kind of unpredictable delays they want to avoid. It has to unmap PTEs, copy data from the small folios to a new THP, and then remap the THP back to the PMD ;)Generally, we allow minor page faults into mlocked VMAs and avoid major. This is minor page fault territory in my view.
Quoted 6 lines
Quoted 2 lines
Hm, but we won't be causing minor faults via reclaim right, since they're not on any LRU?
PTEs are still present when we do THP allocation. No reclaim while the access is blocked. We only block the access on copy and
PTEs->PMDcollapse.
Right indeed, esp. with compaction being allowed for mlock, I agree with you that this patch should be dropped :)
Quoted 3 lines
--
Kiryl Shutsemau / Kirill A. Shutemov
Cheers, Lorenzo
Lance Yang
reviewerRe: [PATCH mm-new 1/3] mm/khugepaged: skip unsuitable VMAs earlier in khugepaged_scan_mm_slot()
In reply to Lorenzo StoakesQuoted 9 lines · Lorenzo Stoakes
Quoted 8 lines · Kiryl Shutsemau
Quoted 4 lines · Lance Yang
Users of
mlock()expect low and predictable latency. THP collapse is a heavy operation that introduces exactly the kind of unpredictable delays they want to avoid. It has to unmap PTEs, copy data from the small folios to a new THP, and then remap the THP back to the PMD ;)Generally, we allow minor page faults into mlocked VMAs and avoid major. This is minor page fault territory in my view.
Makes sense to me!
Quoted 10 lines
Hm, but we won't be causing minor faults via reclaim right, since they're not on any LRU?
Quoted 4 lines
Also it is very similar to what compaction does and we allow compaction of mlocked VMA by default, unless sysctl vm.compact_unevictable_allowed is set to zero.
This is a much stronger point.
Ah, indeed, the compaction analogy is quite strong here, thanks!
Quoted 2 lines
I think we are sometimes too vague as to what
mlock()means in
Totally agree on too vague ;)
Quoted 4 lines
totality. But given that we default allow compaction it seems sensible to keep this behaviour the same.
Unless you have a specific situation where this is problematic Lance?
Not a specific situation right now that would clearly make this problematic.
Anyway, I will drop this patch from the series.
Thanks again for all the feedback everyone!
Lance
Lance Yang
reviewerRe: [PATCH mm-new 1/3] mm/khugepaged: skip unsuitable VMAs earlier in khugepaged_scan_mm_slot()
In reply to Lorenzo StoakesQuoted 15 lines · Lorenzo Stoakes
Quoted 14 lines · Kiryl Shutsemau
Quoted 12 lines · Lorenzo Stoakes
Quoted 8 lines · Kiryl Shutsemau
Quoted 4 lines · Lance Yang
Users of
mlock()expect low and predictable latency. THP collapse is a heavy operation that introduces exactly the kind of unpredictable delays they want to avoid. It has to unmap PTEs, copy data from the small folios to a new THP, and then remap the THP back to the PMD ;)Generally, we allow minor page faults into mlocked VMAs and avoid major. This is minor page fault territory in my view.
Hm, but we won't be causing minor faults via reclaim right, since they're not on any LRU?
Quoted 6 lines
Quoted 3 lines
PTEs are still present when we do THP allocation. No reclaim while the access is blocked. We only block the access on copy and
PTEs->PMDcollapse.Right indeed, esp. with compaction being allowed for mlock, I agree with you that this patch should be dropped :)
Got it. Will do ;)
Thanks,
Lance
Re: [PATCH mm-new 2/3] mm: clean up and expose is_guard_pte_marker()
In reply to Lance YangQuoted 12 lines · Lance Yang
From: Lance Yang <[email protected]>
is_guard_pte_marker()performs a redundant check because it calls bothis_pte_marker()andis_guard_swp_entry(), both of which internally check for a PTE marker.
is_guard_pte_marker()
|-is_pte_marker()
| `-is_pte_marker_entry()// First check
`-is_guard_swp_entry()
`-is_pte_marker_entry()// Second, redundant check
Quoted 8 lines
While a modern compiler could likely optimize this away, let's have clean code and not rely on it ;)
Also, make it available for hugepage collapsing code.
Cc Kairui Song <[email protected]>Signed-off-by Lance Yang <[email protected]>---
Signature · 3 lines
Cheers David / dhildenb