khugepaged: Reduce race probability between migration and khugepaged
Subsystems (2)
Dev Jain
reviewerPATCH 1/1 khugepaged: Reduce race probability between migration and khugepaged
Suppose a folio is under migration, and khugepaged is also trying to collapse it. collapse_pte_mapped_thp() will retrieve the folio from the page cache via filemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in__folio_migrate_mapping -> folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.
This fixes the migration-shared-anon-thp testcase failure on Apple M3.
Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".
---
This patch was part of
https://lore.kernel.org/all/[email protected]/
but I have sent it separately on suggestion of Lorenzo, and also because I plan to send the first two patches after David Hildenbrand's folio_pte_batch series gets merged.
mm/khugepaged.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
mm/khugepaged.c +10 -2
@@ -31,6 +31,7 @@ enum scan_result {SCAN_FAIL ,SCAN_SUCCEED ,SCAN_PMD_NULL , +SCAN_PMD_MIGRATION ,SCAN_PMD_NONE ,SCAN_PMD_MAPPED , SCAN_EXCEED_NONE_PTE,
@@ -941,6 +942,8 @@ static inline int check_pmd_state(pmd_t *pmd)if (pmd_none (pmde))return SCAN_PMD_NONE ; +if (is_pmd_migration_entry (pmde)) +return SCAN_PMD_MIGRATION ;if (!pmd_present (pmde))return SCAN_PMD_NULL ;if (pmd_trans_huge (pmde))
@@ -1502,9 +1505,12 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, !range_in_vma(vma, haddr, haddr + HPAGE_PMD_SIZE))return SCAN_VMA_CHECK ; -/* Fast check before locking page if already PMD-mapped */ +/* +* Fast check before locking folio if already PMD-mapped, or if the +* folio is under migration +*/ result =find_pmd_or_thp_or_none (mm, haddr, &pmd); -if (result ==SCAN_PMD_MAPPED ) +if (result ==SCAN_PMD_MAPPED || result ==SCAN_PMD_MIGRATION )return result;/*
@@ -2716,6 +2722,7 @@ static int madvise_collapse_errno(enum scan_result r)case SCAN_PAGE_LRU :case SCAN_DEL_PAGE_LRU :case SCAN_PAGE_FILLED : +case SCAN_PMD_MIGRATION :return -EAGAIN ;/* * Other: Trying again likely not to succeed / error intrinsic to
@@ -2802,6 +2809,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, goto handle_result;/* Whitelisted set of results where continuing OK */ case SCAN_PMD_NULL : +case SCAN_PMD_MIGRATION :case SCAN_PTE_NON_PRESENT :case SCAN_PTE_UFFD_WP :case SCAN_PAGE_RO :
Signature · 1 line
2.30.2
Quoted 9 lines · Dev Jain
Suppose a folio is under migration, and khugepaged is also trying to collapse it.
collapse_pte_mapped_thp()will retrieve the folio from the page cache viafilemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in__folio_migrate_mapping->folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.
Quoted 8 lines
This fixes the migration-shared-anon-thp testcase failure on Apple M3.
Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".
Signed-off-by Dev Jain <[email protected]>---
Looks reasonable to me.
Quoted 8 lines · Dev Jain
Suppose a folio is under migration, and khugepaged is also trying to collapse it.
collapse_pte_mapped_thp()will retrieve the folio from the page cache viafilemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in__folio_migrate_mapping->folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.
Could the migration be re-attempted after such failure ? Seems like the migration failure here is traded for a scan failure instead.
Quoted 2 lines
This fixes the migration-shared-anon-thp testcase failure on Apple M3.
Could you please provide some more context why this test case was failing earlier and how does this change here fixes the problem ?
Quoted 39 lines
Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".
Quoted 1 line
Signed-off-by Dev Jain <[email protected]>---
This patch was part of
https://lore.kernel.org/all/[email protected]/
but I have sent it separately on suggestion of Lorenzo, and also because I plan to send the first two patches after David Hildenbrand'sfolio_pte_batchseries gets merged.mm/khugepaged.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)mm/khugepaged.c +3
-0@@ -31,6 +31,7 @@ enum scan_result {SCAN_FAIL ,SCAN_SUCCEED ,SCAN_PMD_NULL , +SCAN_PMD_MIGRATION ,SCAN_PMD_NONE ,SCAN_PMD_MAPPED , SCAN_EXCEED_NONE_PTE,@@ -941,6 +942,8 @@ static inline int check_pmd_state(pmd_t *pmd)if (pmd_none (pmde))return SCAN_PMD_NONE ; +if (is_pmd_migration_entry (pmde)) +return SCAN_PMD_MIGRATION ;if (!pmd_present (pmde))return SCAN_PMD_NULL ;if (pmd_trans_huge (pmde))@@ -1502,9 +1505,12 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, !range_in_vma(vma, haddr, haddr + HPAGE_PMD_SIZE))
mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1513
Quoted 10 lines
mm/khugepaged.c +5
-2return SCAN_VMA_CHECK ; -/* Fast check before locking page if already PMD-mapped */ +/* +* Fast check before locking folio if already PMD-mapped, or if the +* folio is under migration +*/ result =find_pmd_or_thp_or_none (mm, haddr, &pmd); -if (result ==SCAN_PMD_MAPPED ) +if (result ==SCAN_PMD_MAPPED || result ==SCAN_PMD_MIGRATION )
Should mapped PMD and migrating PMD be treated equally while scanning ?
Quoted 19 lines
mm/khugepaged.c +2
-0return result;/* @@ -2716,6 +2722,7 @@ static int madvise_collapse_errno(enum scan_result r)case SCAN_PAGE_LRU :case SCAN_DEL_PAGE_LRU :case SCAN_PAGE_FILLED : +case SCAN_PMD_MIGRATION :return -EAGAIN ;/* * Other: Trying again likely not to succeed / error intrinsic to @@ -2802,6 +2809,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, goto handle_result;/* Whitelisted set of results where continuing OK */ case SCAN_PMD_NULL : +case SCAN_PMD_MIGRATION :case SCAN_PTE_NON_PRESENT :case SCAN_PTE_UFFD_WP :case SCAN_PAGE_RO :
Quoted 60 lines · Anshuman Khandual
Quoted 8 lines · Dev Jain
Suppose a folio is under migration, and khugepaged is also trying to collapse it.
collapse_pte_mapped_thp()will retrieve the folio from the page cache viafilemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in__folio_migrate_mapping->folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.Could the migration be re-attempted after such failure ? Seems like the migration failure here is traded for a scan failure instead.
Quoted 2 lines
This fixes the migration-shared-anon-thp testcase failure on Apple M3.
Could you please provide some more context why this test case was failing earlier and how does this change here fixes the problem ?
Quoted 41 lines
Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".
Quoted 1 line
Signed-off-by Dev Jain <[email protected]>---
This patch was part of
https://lore.kernel.org/all/[email protected]/
but I have sent it separately on suggestion of Lorenzo, and also because I plan to send the first two patches after David Hildenbrand'sfolio_pte_batchseries gets merged.mm/khugepaged.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)mm/khugepaged.c +3
-0@@ -31,6 +31,7 @@ enum scan_result {SCAN_FAIL ,SCAN_SUCCEED ,SCAN_PMD_NULL , +SCAN_PMD_MIGRATION ,SCAN_PMD_NONE ,SCAN_PMD_MAPPED , SCAN_EXCEED_NONE_PTE,@@ -941,6 +942,8 @@ static inline int check_pmd_state(pmd_t *pmd)if (pmd_none (pmde))return SCAN_PMD_NONE ; +if (is_pmd_migration_entry (pmde)) +return SCAN_PMD_MIGRATION ;if (!pmd_present (pmde))return SCAN_PMD_NULL ;if (pmd_trans_huge (pmde))@@ -1502,9 +1505,12 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, !range_in_vma(vma, haddr, haddr + HPAGE_PMD_SIZE))return SCAN_VMA_CHECK ;
Quoted 9 lines
mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1513
Quoted 8 lines
mm/khugepaged.c +5
-2-/* Fast check before locking page if already PMD-mapped */ +/* +* Fast check before locking folio if already PMD-mapped, or if the +* folio is under migration +*/ result =find_pmd_or_thp_or_none (mm, haddr, &pmd); -if (result ==SCAN_PMD_MAPPED ) +if (result ==SCAN_PMD_MAPPED || result ==SCAN_PMD_MIGRATION )Should mapped PMD and migrating PMD be treated equally while scanning ?
Wanted to ask the same thing I think: why not simply use SCAN_PMD_MAPPED? After all, the folio is already pmd-mapped, just not using a present entry but (temporarily) using a migration entry.
Signature · 3 lines
Cheers, David / dhildenb
Dev Jain
reviewerQuoted 11 lines · Anshuman Khandual
Quoted 8 lines · Dev Jain
Suppose a folio is under migration, and khugepaged is also trying to collapse it.
collapse_pte_mapped_thp()will retrieve the folio from the page cache viafilemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in__folio_migrate_mapping->folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.Could the migration be re-attempted after such failure ? Seems like the migration failure here is traded for a scan failure instead.
We already re-attempt migration. See NR_MAX_MIGRATE_PAGES_RETRY and NR_MAX_MIGRATE_ASYNC_RETRY. Also just before freezing the refcount, we do a suitable refcount check in folio_migrate_mapping(). So the race happens after this and folio_ref_freeze() in __folio_migrate_mapping(), therefore the window for the race is already very small in the migration path, but large in the khugepaged path.
Quoted 4 lines
Quoted 2 lines
This fixes the migration-shared-anon-thp testcase failure on Apple M3.
Could you please provide some more context why this test case was failing earlier and how does this change here fixes the problem ?
IMHO the explanation I have given in the patch description is clear and succinct: the testcase is failing due to the race. This patch shortens the race window, and the test on this particular hardware does not hit the race window again.
Quoted 41 lines
Quoted 41 lines
Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".
Quoted 1 line
Signed-off-by Dev Jain <[email protected]>---
This patch was part of
https://lore.kernel.org/all/[email protected]/
but I have sent it separately on suggestion of Lorenzo, and also because I plan to send the first two patches after David Hildenbrand'sfolio_pte_batchseries gets merged.mm/khugepaged.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)mm/khugepaged.c +3
-0@@ -31,6 +31,7 @@ enum scan_result {SCAN_FAIL ,SCAN_SUCCEED ,SCAN_PMD_NULL , +SCAN_PMD_MIGRATION ,SCAN_PMD_NONE ,SCAN_PMD_MAPPED , SCAN_EXCEED_NONE_PTE,@@ -941,6 +942,8 @@ static inline int check_pmd_state(pmd_t *pmd)if (pmd_none (pmde))return SCAN_PMD_NONE ; +if (is_pmd_migration_entry (pmde)) +return SCAN_PMD_MIGRATION ;if (!pmd_present (pmde))return SCAN_PMD_NULL ;if (pmd_trans_huge (pmde))@@ -1502,9 +1505,12 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, !range_in_vma(vma, haddr, haddr + HPAGE_PMD_SIZE))return SCAN_VMA_CHECK ;
Quoted 9 lines
mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1513
Quoted 8 lines
mm/khugepaged.c +5
-2-/* Fast check before locking page if already PMD-mapped */ +/* +* Fast check before locking folio if already PMD-mapped, or if the +* folio is under migration +*/ result =find_pmd_or_thp_or_none (mm, haddr, &pmd); -if (result ==SCAN_PMD_MAPPED ) +if (result ==SCAN_PMD_MAPPED || result ==SCAN_PMD_MIGRATION )Should mapped PMD and migrating PMD be treated equally while scanning ?
SCAN_PMD_MAPPED is used as an indicator to change result to SCAN_SUCCEED in khugepaged_scan_mm_slot: after the call to collapse_pte_mapped_thp. And, it is also used in madvise_collapse() to do ++thps which is used to set the return value of madvise_collapse. So I think this approach will be wrong.
Quoted 20 lines
Quoted 20 lines
mm/khugepaged.c +2
-0return result;/* @@ -2716,6 +2722,7 @@ static int madvise_collapse_errno(enum scan_result r)case SCAN_PAGE_LRU :case SCAN_DEL_PAGE_LRU :case SCAN_PAGE_FILLED : +case SCAN_PMD_MIGRATION :return -EAGAIN ;/* * Other: Trying again likely not to succeed / error intrinsic to @@ -2802,6 +2809,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, goto handle_result;/* Whitelisted set of results where continuing OK */ case SCAN_PMD_NULL : +case SCAN_PMD_MIGRATION :case SCAN_PTE_NON_PRESENT :case SCAN_PTE_UFFD_WP :case SCAN_PAGE_RO :
Quoted 83 lines · Dev Jain
Quoted 11 lines · Anshuman Khandual
Quoted 8 lines · Dev Jain
Suppose a folio is under migration, and khugepaged is also trying to collapse it.
collapse_pte_mapped_thp()will retrieve the folio from the page cache viafilemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in__folio_migrate_mapping->folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.Could the migration be re-attempted after such failure ? Seems like the migration failure here is traded for a scan failure instead.
We already re-attempt migration. See
NR_MAX_MIGRATE_PAGES_RETRYandNR_MAX_MIGRATE_ASYNC_RETRY. Also just before freezing the refcount, we do a suitable refcount check infolio_migrate_mapping(). So the race happens after this andfolio_ref_freeze()in__folio_migrate_mapping(), therefore the window for the race is already very small in the migration path, but large in the khugepaged path.Quoted 4 lines
Quoted 2 lines
This fixes the migration-shared-anon-thp testcase failure on Apple M3.
Could you please provide some more context why this test case was failing earlier and how does this change here fixes the problem ?
IMHO the explanation I have given in the patch description is clear and succinct: the testcase is failing due to the race. This patch shortens the race window, and the test on this particular hardware does not hit the race window again.
Quoted 41 lines
Quoted 41 lines
Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".
Quoted 1 line
Signed-off-by Dev Jain <[email protected]>---
This patch was part of
https://lore.kernel.org/all/[email protected]/
but I have sent it separately on suggestion of Lorenzo, and also because I plan to send the first two patches after David Hildenbrand'sfolio_pte_batchseries gets merged.mm/khugepaged.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)mm/khugepaged.c +3
-0@@ -31,6 +31,7 @@ enum scan_result {SCAN_FAIL ,SCAN_SUCCEED ,SCAN_PMD_NULL , +SCAN_PMD_MIGRATION ,SCAN_PMD_NONE ,SCAN_PMD_MAPPED , SCAN_EXCEED_NONE_PTE,@@ -941,6 +942,8 @@ static inline int check_pmd_state(pmd_t *pmd)if (pmd_none (pmde))return SCAN_PMD_NONE ; +if (is_pmd_migration_entry (pmde)) +return SCAN_PMD_MIGRATION ;if (!pmd_present (pmde))return SCAN_PMD_NULL ;if (pmd_trans_huge (pmde))@@ -1502,9 +1505,12 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, !range_in_vma(vma, haddr, haddr + HPAGE_PMD_SIZE))return SCAN_VMA_CHECK ;Quoted 9 lines
mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1513
Quoted 8 lines
mm/khugepaged.c +5
-2-/* Fast check before locking page if already PMD-mapped */ +/* +* Fast check before locking folio if already PMD-mapped, or if the +* folio is under migration +*/ result =find_pmd_or_thp_or_none (mm, haddr, &pmd); -if (result ==SCAN_PMD_MAPPED ) +if (result ==SCAN_PMD_MAPPED || result ==SCAN_PMD_MIGRATION )Should mapped PMD and migrating PMD be treated equally while scanning ?
Quoted 4 lines
SCAN_PMD_MAPPEDis used as an indicator to change result toSCAN_SUCCEEDinkhugepaged_scan_mm_slot: after the call tocollapse_pte_mapped_thp. And, it is also used inmadvise_collapse()to do ++thps which is used to set the return value ofmadvise_collapse. So I think this approach will be wrong.
But if it already is PMD mapped (just temporarily through a migration entry), isn't this exactly what we want?
Signature · 3 lines
Cheers, David / dhildenb
Dev Jain
reviewerQuoted 83 lines · David Hildenbrand
Quoted 82 lines · Dev Jain
Quoted 11 lines · Anshuman Khandual
Quoted 8 lines · Dev Jain
Suppose a folio is under migration, and khugepaged is also trying to collapse it.
collapse_pte_mapped_thp()will retrieve the folio from the page cache viafilemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in__folio_migrate_mapping->folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.Could the migration be re-attempted after such failure ? Seems like the migration failure here is traded for a scan failure instead.
We already re-attempt migration. See
NR_MAX_MIGRATE_PAGES_RETRYandNR_MAX_MIGRATE_ASYNC_RETRY. Also just before freezing the refcount, we do a suitable refcount check infolio_migrate_mapping(). So the race happens after this andfolio_ref_freeze()in__folio_migrate_mapping(), therefore the window for the race is already very small in the migration path, but large in the khugepaged path.Quoted 4 lines
Quoted 2 lines
This fixes the migration-shared-anon-thp testcase failure on Apple M3.
Could you please provide some more context why this test case was failing earlier and how does this change here fixes the problem ?
IMHO the explanation I have given in the patch description is clear and succinct: the testcase is failing due to the race. This patch shortens the race window, and the test on this particular hardware does not hit the race window again.
Quoted 50 lines
Quoted 41 lines
Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".
Quoted 1 line
Signed-off-by Dev Jain <[email protected]>---
This patch was part of
https://lore.kernel.org/all/[email protected]/
but I have sent it separately on suggestion of Lorenzo, and also because I plan to send the first two patches after David Hildenbrand'sfolio_pte_batchseries gets merged.
mm/khugepaged.c| 12 ++++++++++--1 file changed, 10 insertions(+), 2 deletions(-)mm/khugepaged.c +3
-0@@ -31,6 +31,7 @@ enum scan_result { SCAN_FAIL, SCAN_SUCCEED, SCAN_PMD_NULL, + SCAN_PMD_MIGRATION, SCAN_PMD_NONE, SCAN_PMD_MAPPED, SCAN_EXCEED_NONE_PTE,@@ -941,6 +942,8 @@ static inline int check_pmd_state(pmd_t *pmd) if (pmd_none(pmde)) return SCAN_PMD_NONE; + if (is_pmd_migration_entry(pmde)) + return SCAN_PMD_MIGRATION; if (!pmd_present(pmde)) return SCAN_PMD_NULL; if (pmd_trans_huge(pmde))@@ -1502,9 +1505,12 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, !range_in_vma(vma, haddr, haddr + HPAGE_PMD_SIZE)) return SCAN_VMA_CHECK;- /* Fast check before locking page if already PMD-mapped */
Quoted 8 lines
+ /*
+ * Fast check before locking folio if already PMD-mapped, or if the
+ * folio is under migration
+ */
result =find_pmd_or_thp_or_none(mm, haddr, &pmd); - if (result ==SCAN_PMD_MAPPED)
+ if (result ==SCAN_PMD_MAPPED|| result ==SCAN_PMD_MIGRATION)Should mapped PMD and migrating PMD be treated equally while scanning ?
Quoted 6 lines
Quoted 4 lines
SCAN_PMD_MAPPEDis used as an indicator to change result toSCAN_SUCCEEDinkhugepaged_scan_mm_slot: after the call tocollapse_pte_mapped_thp. And, it is also used inmadvise_collapse()to do ++thps which is used to set the return value ofmadvise_collapse. So I think this approach will be wrong.But if it already is PMD mapped (just temporarily through a migration entry), isn't this exactly what we want?
Good point. I was about to say that what about PMD-folio splitting during migration, but then
during unmapping the folios, if we cannot migrate the PMD-folios, they will be splitted via
unmap_folio() along with the PMD, therefore we can be sure that if we encounter a PMD
migration entry, then eventually it will be converted to a PMD leaf entry on migration success
or failure.
I'll merge this into SCAN_PMD_MAPPED, thanks.
Quoted 8 lines · Dev Jain
Suppose a folio is under migration, and khugepaged is also trying to collapse it.
collapse_pte_mapped_thp()will retrieve the folio from the page cache viafilemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in__folio_migrate_mapping->folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.
This is a nice find!
Quoted 6 lines
This fixes the migration-shared-anon-thp testcase failure on Apple M3.
Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".
Thanks for separating this out, appreciated!
Quoted 26 lines
Signed-off-by Dev Jain <[email protected]>---
This patch was part of
https://lore.kernel.org/all/[email protected]/
but I have sent it separately on suggestion of Lorenzo, and also because I plan to send the first two patches after David Hildenbrand'sfolio_pte_batchseries gets merged.mm/khugepaged.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)mm/khugepaged.c +1
-0@@ -31,6 +31,7 @@ enum scan_result {SCAN_FAIL ,SCAN_SUCCEED ,SCAN_PMD_NULL , +SCAN_PMD_MIGRATION ,SCAN_PMD_NONE ,SCAN_PMD_MAPPED , SCAN_EXCEED_NONE_PTE,
mm/khugepaged.c static inline int check_pmd_state(pmd_t *pmd) · line 946
Quoted 5 lines
mm/khugepaged.c +2
-0@@ -941,6 +942,8 @@ static inline int check_pmd_state(pmd_t *pmd)if (pmd_none (pmde))return SCAN_PMD_NONE ; +if (is_pmd_migration_entry (pmde)) +return SCAN_PMD_MIGRATION ;
With David's suggestions I guess this boils down to simply adding this line.
Could we add a quick comment to explain why here?
Thanks!
Quoted 37 lines
mm/khugepaged.c +7
-3if (!pmd_present (pmde))return SCAN_PMD_NULL ;if (pmd_trans_huge (pmde))@@ -1502,9 +1505,12 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, !range_in_vma(vma, haddr, haddr + HPAGE_PMD_SIZE))return SCAN_VMA_CHECK ; -/* Fast check before locking page if already PMD-mapped */ +/* +* Fast check before locking folio if already PMD-mapped, or if the +* folio is under migration +*/ result =find_pmd_or_thp_or_none (mm, haddr, &pmd); -if (result ==SCAN_PMD_MAPPED ) +if (result ==SCAN_PMD_MAPPED || result ==SCAN_PMD_MIGRATION )return result;/* @@ -2716,6 +2722,7 @@ static int madvise_collapse_errno(enum scan_result r)case SCAN_PAGE_LRU :case SCAN_DEL_PAGE_LRU :case SCAN_PAGE_FILLED : +case SCAN_PMD_MIGRATION :return -EAGAIN ;/* * Other: Trying again likely not to succeed / error intrinsic to @@ -2802,6 +2809,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, goto handle_result;/* Whitelisted set of results where continuing OK */ case SCAN_PMD_NULL : +case SCAN_PMD_MIGRATION :case SCAN_PTE_NON_PRESENT :case SCAN_PTE_UFFD_WP :case SCAN_PAGE_RO : --2.30.2
Dev Jain
reviewerQuoted 44 lines · Lorenzo Stoakes
Quoted 8 lines · Dev Jain
Suppose a folio is under migration, and khugepaged is also trying to collapse it.
collapse_pte_mapped_thp()will retrieve the folio from the page cache viafilemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in__folio_migrate_mapping->folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.This is a nice find!
Quoted 5 lines
This fixes the migration-shared-anon-thp testcase failure on Apple M3.
Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".
Thanks for separating this out, appreciated!
Quoted 26 lines
Signed-off-by Dev Jain <[email protected]>---
This patch was part of
https://lore.kernel.org/all/[email protected]/
but I have sent it separately on suggestion of Lorenzo, and also because I plan to send the first two patches after David Hildenbrand'sfolio_pte_batchseries gets merged.mm/khugepaged.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)mm/khugepaged.c +1
-0@@ -31,6 +31,7 @@ enum scan_result {SCAN_FAIL ,SCAN_SUCCEED ,SCAN_PMD_NULL , +SCAN_PMD_MIGRATION ,SCAN_PMD_NONE ,SCAN_PMD_MAPPED , SCAN_EXCEED_NONE_PTE,@@ -941,6 +942,8 @@ static inline int check_pmd_state(pmd_t *pmd)
Quoted 5 lines
mm/khugepaged.c static inline int check_pmd_state(pmd_t *pmd) · line 946
Quoted 4 lines
mm/khugepaged.c +2
-0if (pmd_none (pmde))return SCAN_PMD_NONE ; +if (is_pmd_migration_entry (pmde)) +return SCAN_PMD_MIGRATION ;With David's suggestions I guess this boils down to simply adding this line.
I think it should be
mm/khugepaged.c +2 -2
@@ -941,10 +941,10 @@ static inline int check_pmd_state(pmd_t *pmd)if (pmd_none (pmde))return SCAN_PMD_NONE ; +if (is_pmd_migration_entry (pmde) ||pmd_trans_huge (pmde)) +return SCAN_PMD_MAPPED ;if (!pmd_present (pmde))return SCAN_PMD_NULL ; -if (pmd_trans_huge (pmde)) -return SCAN_PMD_MAPPED ;if (pmd_bad (pmde))return SCAN_PMD_NULL ;return SCAN_SUCCEED ;
Moving this line above since we don't want to exit prematurely due to !pmd_present(pmde).
Quoted 2 lines
Could we add a quick comment to explain why here?
Sure.
Quoted 40 lines
Thanks!
Quoted 37 lines
mm/khugepaged.c +7
-3if (!pmd_present (pmde))return SCAN_PMD_NULL ;if (pmd_trans_huge (pmde))@@ -1502,9 +1505,12 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, !range_in_vma(vma, haddr, haddr + HPAGE_PMD_SIZE))return SCAN_VMA_CHECK ; -/* Fast check before locking page if already PMD-mapped */ +/* +* Fast check before locking folio if already PMD-mapped, or if the +* folio is under migration +*/ result =find_pmd_or_thp_or_none (mm, haddr, &pmd); -if (result ==SCAN_PMD_MAPPED ) +if (result ==SCAN_PMD_MAPPED || result ==SCAN_PMD_MIGRATION )return result;/* @@ -2716,6 +2722,7 @@ static int madvise_collapse_errno(enum scan_result r)case SCAN_PAGE_LRU :case SCAN_DEL_PAGE_LRU :case SCAN_PAGE_FILLED : +case SCAN_PMD_MIGRATION :return -EAGAIN ;/* * Other: Trying again likely not to succeed / error intrinsic to @@ -2802,6 +2809,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, goto handle_result;/* Whitelisted set of results where continuing OK */ case SCAN_PMD_NULL : +case SCAN_PMD_MIGRATION :case SCAN_PTE_NON_PRESENT :case SCAN_PTE_UFFD_WP :case SCAN_PAGE_RO : --2.30.2
Quoted 72 lines · Dev Jain
Quoted 44 lines · Lorenzo Stoakes
Quoted 8 lines · Dev Jain
Suppose a folio is under migration, and khugepaged is also trying to collapse it.
collapse_pte_mapped_thp()will retrieve the folio from the page cache viafilemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in__folio_migrate_mapping->folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.This is a nice find!
Quoted 5 lines
This fixes the migration-shared-anon-thp testcase failure on Apple M3.
Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".
Thanks for separating this out, appreciated!
Quoted 26 lines
Signed-off-by Dev Jain <[email protected]>---
This patch was part of
https://lore.kernel.org/all/[email protected]/
but I have sent it separately on suggestion of Lorenzo, and also because I plan to send the first two patches after David Hildenbrand'sfolio_pte_batchseries gets merged.
mm/khugepaged.c| 12 ++++++++++--1 file changed, 10 insertions(+), 2 deletions(-)mm/khugepaged.c +1
-0@@ -31,6 +31,7 @@ enum scan_result { SCAN_FAIL, SCAN_SUCCEED, SCAN_PMD_NULL, + SCAN_PMD_MIGRATION, SCAN_PMD_NONE, SCAN_PMD_MAPPED, SCAN_EXCEED_NONE_PTE,@@ -941,6 +942,8 @@ static inline int check_pmd_state(pmd_t *pmd)Quoted 5 lines
mm/khugepaged.c static inline int check_pmd_state(pmd_t *pmd)
Quoted 4 lines
mm/khugepaged.c +2
-0if (pmd_none(pmde)) return SCAN_PMD_NONE; + if (is_pmd_migration_entry(pmde)) + return SCAN_PMD_MIGRATION;With David's suggestions I guess this boils down to simply adding this line.
I think it should be
mm/khugepaged.c +0
-0@@ -941,10 +941,10 @@ static inline int check_pmd_state(pmd_t *pmd)
if (pmd_none(pmde))
returnSCAN_PMD_NONE;
+ if (is_pmd_migration_entry(pmde)||pmd_trans_huge(pmde)) + returnSCAN_PMD_MAPPED;
if (!pmd_present(pmde))
returnSCAN_PMD_NULL;
- if (pmd_trans_huge(pmde))
- returnSCAN_PMD_MAPPED;
if (pmd_bad(pmde))
returnSCAN_PMD_NULL;
returnSCAN_SUCCEED;
Quoted 2 lines
Moving this line above since we don't want to exit prematurely due to !pmd_present(pmde).
Might be cleaner to just add the migration test separately before the pmd_present() and without modifying existing pmd_trans_huge().
if (is_pmd_migration_entry(pmde)) return SCAN_PMD_MAPPED;
Quoted 48 lines
Quoted 4 lines
Could we add a quick comment to explain why here?
Sure.
Quoted 40 lines
Thanks!
Quoted 37 lines
mm/khugepaged.c +5
-2if (!pmd_present(pmde)) return SCAN_PMD_NULL; if (pmd_trans_huge(pmde))@@ -1502,9 +1505,12 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, !range_in_vma(vma, haddr, haddr + HPAGE_PMD_SIZE)) return SCAN_VMA_CHECK; - /* Fast check before locking page if already PMD-mapped */ + /* + * Fast check before locking folio if already PMD-mapped, or if the + * folio is under migration + */ result = find_pmd_or_thp_or_none(mm, haddr, &pmd); - if (result == SCAN_PMD_MAPPED) + if (result == SCAN_PMD_MAPPED || result == SCAN_PMD_MIGRATION) return result; /*@@ -2716,6 +2722,7 @@ static int madvise_collapse_errno(enum scan_result r)case
SCAN_PAGE_LRU:
caseSCAN_DEL_PAGE_LRU:
caseSCAN_PAGE_FILLED:
+ caseSCAN_PMD_MIGRATION:
return-EAGAIN;
/*
* Other: Trying again likely not to succeed / error intrinsic tomm/khugepaged.c +0
-0@@ -2802,6 +2809,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, goto handle_result; /* Whitelisted set of results where continuing OK */case
SCAN_PMD_NULL:
+ caseSCAN_PMD_MIGRATION:
caseSCAN_PTE_NON_PRESENT:
caseSCAN_PTE_UFFD_WP:
caseSCAN_PAGE_RO:Signature · 1 line
2.30.2
Dev Jain
reviewerQuoted 72 lines · Anshuman Khandual
Quoted 70 lines · Dev Jain
Quoted 44 lines · Lorenzo Stoakes
Quoted 8 lines · Dev Jain
Suppose a folio is under migration, and khugepaged is also trying to collapse it.
collapse_pte_mapped_thp()will retrieve the folio from the page cache viafilemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in__folio_migrate_mapping->folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.This is a nice find!
Quoted 5 lines
This fixes the migration-shared-anon-thp testcase failure on Apple M3.
Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".
Thanks for separating this out, appreciated!
Quoted 26 lines
Signed-off-by Dev Jain <[email protected]>---
This patch was part of
https://lore.kernel.org/all/[email protected]/
but I have sent it separately on suggestion of Lorenzo, and also because I plan to send the first two patches after David Hildenbrand'sfolio_pte_batchseries gets merged.mm/khugepaged.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)mm/khugepaged.c +1
-0@@ -31,6 +31,7 @@ enum scan_result { SCAN_FAIL, SCAN_SUCCEED, SCAN_PMD_NULL, + SCAN_PMD_MIGRATION, SCAN_PMD_NONE, SCAN_PMD_MAPPED, SCAN_EXCEED_NONE_PTE,@@ -941,6 +942,8 @@ static inline int check_pmd_state(pmd_t *pmd)Quoted 5 lines
mm/khugepaged.c static inline int check_pmd_state(pmd_t *pmd)
Quoted 4 lines
mm/khugepaged.c +2
-0if (pmd_none(pmde)) return SCAN_PMD_NONE; + if (is_pmd_migration_entry(pmde)) + return SCAN_PMD_MIGRATION;With David's suggestions I guess this boils down to simply adding this line.
I think it should be
mm/khugepaged.c +2
-2@@ -941,10 +941,10 @@ static inline int check_pmd_state(pmd_t *pmd) if (pmd_none(pmde)) return SCAN_PMD_NONE; + if (is_pmd_migration_entry(pmde) || pmd_trans_huge(pmde)) + return SCAN_PMD_MAPPED; if (!pmd_present(pmde)) return SCAN_PMD_NULL; - if (pmd_trans_huge(pmde)) - return SCAN_PMD_MAPPED; if (pmd_bad(pmde)) return SCAN_PMD_NULL; return SCAN_SUCCEED;
Quoted 7 lines
Quoted 2 lines
Moving this line above since we don't want to exit prematurely due to !pmd_present(pmde).
Might be cleaner to just add the migration test separately before the
pmd_present()and without modifying existingpmd_trans_huge().if (is_pmd_migration_entry(pmde)) return SCAN_PMD_MAPPED;
Sounds good.
Quoted 43 lines
Quoted 43 lines
Quoted 2 lines
Could we add a quick comment to explain why here?
Sure.
Quoted 39 lines
Thanks!
Quoted 37 lines
mm/khugepaged.c +7
-2if (!pmd_present(pmde)) return SCAN_PMD_NULL; if (pmd_trans_huge(pmde))@@ -1502,9 +1505,12 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, !range_in_vma(vma, haddr, haddr + HPAGE_PMD_SIZE)) return SCAN_VMA_CHECK; - /* Fast check before locking page if already PMD-mapped */ + /* + * Fast check before locking folio if already PMD-mapped, or if the + * folio is under migration + */ result = find_pmd_or_thp_or_none(mm, haddr, &pmd); - if (result == SCAN_PMD_MAPPED) + if (result == SCAN_PMD_MAPPED || result == SCAN_PMD_MIGRATION) return result; /*@@ -2716,6 +2722,7 @@ static int madvise_collapse_errno(enum scan_result r) case SCAN_PAGE_LRU: case SCAN_DEL_PAGE_LRU: case SCAN_PAGE_FILLED: + case SCAN_PMD_MIGRATION: return -EAGAIN; /** Other: Trying again likely not to succeed / error intrinsic to @@ -2802,6 +2809,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, goto handle_result; /* Whitelisted set of results where continuing OK */ case SCAN_PMD_NULL: + case SCAN_PMD_MIGRATION: case SCAN_PTE_NON_PRESENT: case SCAN_PTE_UFFD_WP: case SCAN_PAGE_RO:Signature · 1 line
2.30.2