Skip to contentSkip to conversation

khugepaged: Reduce race probability between migration and khugepaged

PATCH Superseded Dev Jain 5 participants 11 messages Jun 30, 2025 – Jul 1, 2025

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Dev Jain

author reviewer
#0 lore raw

PATCH 1/1 khugepaged: Reduce race probability between migration and khugepaged

Suppose a folio is under migration, and khugepaged is also trying to collapse it. collapse_pte_mapped_thp() will retrieve the folio from the page cache via filemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in
__folio_migrate_mapping -> folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.

This fixes the migration-shared-anon-thp testcase failure on Apple M3.

Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".

---

This patch was part of
https://lore.kernel.org/all/[email protected]/
but I have sent it separately on suggestion of Lorenzo, and also because I plan to send the first two patches after David Hildenbrand's folio_pte_batch series gets merged.

 mm/khugepaged.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
mm/khugepaged.c +10 -2
index 1aa7ca67c756..99977bb9bf6a 100644
@@ -31,6 +31,7 @@ enum scan_result {
 	SCAN_FAIL,
 	SCAN_SUCCEED,
 	SCAN_PMD_NULL,
+	SCAN_PMD_MIGRATION,
 	SCAN_PMD_NONE,
 	SCAN_PMD_MAPPED,
 	SCAN_EXCEED_NONE_PTE,
@@ -941,6 +942,8 @@ static inline int check_pmd_state(pmd_t *pmd)
 
 	if (pmd_none(pmde))
 		return SCAN_PMD_NONE;
+	if (is_pmd_migration_entry(pmde))
+		return SCAN_PMD_MIGRATION;
 	if (!pmd_present(pmde))
 		return SCAN_PMD_NULL;
 	if (pmd_trans_huge(pmde))
@@ -1502,9 +1505,12 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
 	    !range_in_vma(vma, haddr, haddr + HPAGE_PMD_SIZE))
 		return SCAN_VMA_CHECK;
 
-	/* Fast check before locking page if already PMD-mapped */
+	/*
+	 * Fast check before locking folio if already PMD-mapped, or if the
+	 * folio is under migration
+	 */
 	result = find_pmd_or_thp_or_none(mm, haddr, &pmd);
-	if (result == SCAN_PMD_MAPPED)
+	if (result == SCAN_PMD_MAPPED || result == SCAN_PMD_MIGRATION)
 		return result;
 
 	/*
@@ -2716,6 +2722,7 @@ static int madvise_collapse_errno(enum scan_result r)
 	case SCAN_PAGE_LRU:
 	case SCAN_DEL_PAGE_LRU:
 	case SCAN_PAGE_FILLED:
+	case SCAN_PMD_MIGRATION:
 		return -EAGAIN;
 	/*
 	 * Other: Trying again likely not to succeed / error intrinsic to
@@ -2802,6 +2809,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
 			goto handle_result;
 		/* Whitelisted set of results where continuing OK */
 		case SCAN_PMD_NULL:
+		case SCAN_PMD_MIGRATION:
 		case SCAN_PTE_NON_PRESENT:
 		case SCAN_PTE_UFFD_WP:
 		case SCAN_PAGE_RO:
Signature · 1 line
2.30.2
reviewer Reviewed-by on v1 1/1
#1 lore raw
Quoted 9 lines · Dev Jain

Suppose a folio is under migration, and khugepaged is also trying to collapse it. collapse_pte_mapped_thp() will retrieve the folio from the page cache via filemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in
__folio_migrate_mapping -> folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.

Quoted 8 lines

This fixes the migration-shared-anon-thp testcase failure on Apple M3.

Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".

---

Looks reasonable to me.

#2 lore raw
In reply to Dev Jain
Quoted 8 lines · Dev Jain

Suppose a folio is under migration, and khugepaged is also trying to collapse it. collapse_pte_mapped_thp() will retrieve the folio from the page cache via filemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in
__folio_migrate_mapping -> folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.

Could the migration be re-attempted after such failure ? Seems like the migration failure here is traded for a scan failure instead.

Quoted 2 lines

This fixes the migration-shared-anon-thp testcase failure on Apple M3.

Could you please provide some more context why this test case was failing earlier and how does this change here fixes the problem ?

Quoted 39 lines

Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".

Quoted 1 line

---

This patch was part of
https://lore.kernel.org/all/[email protected]/
but I have sent it separately on suggestion of Lorenzo, and also because I plan to send the first two patches after David Hildenbrand's folio_pte_batch series gets merged.

 mm/khugepaged.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
mm/khugepaged.c +3 -0
index 1aa7ca67c756..99977bb9bf6a 100644
@@ -31,6 +31,7 @@ enum scan_result {
 	SCAN_FAIL,
 	SCAN_SUCCEED,
 	SCAN_PMD_NULL,
+	SCAN_PMD_MIGRATION,
 	SCAN_PMD_NONE,
 	SCAN_PMD_MAPPED,
 	SCAN_EXCEED_NONE_PTE,
@@ -941,6 +942,8 @@ static inline int check_pmd_state(pmd_t *pmd)
 
 	if (pmd_none(pmde))
 		return SCAN_PMD_NONE;
+	if (is_pmd_migration_entry(pmde))
+		return SCAN_PMD_MIGRATION;
 	if (!pmd_present(pmde))
 		return SCAN_PMD_NULL;
 	if (pmd_trans_huge(pmde))
@@ -1502,9 +1505,12 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
 	    !range_in_vma(vma, haddr, haddr + HPAGE_PMD_SIZE))

mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1513

Quoted 10 lines
mm/khugepaged.c +5 -2
 		return SCAN_VMA_CHECK;
 
-	/* Fast check before locking page if already PMD-mapped */
+	/*
+	 * Fast check before locking folio if already PMD-mapped, or if the
+	 * folio is under migration
+	 */
 	result = find_pmd_or_thp_or_none(mm, haddr, &pmd);
-	if (result == SCAN_PMD_MAPPED)
+	if (result == SCAN_PMD_MAPPED || result == SCAN_PMD_MIGRATION)

Should mapped PMD and migrating PMD be treated equally while scanning ?

Quoted 19 lines
mm/khugepaged.c +2 -0
 		return result;
 
 	/*
@@ -2716,6 +2722,7 @@ static int madvise_collapse_errno(enum scan_result r)
 	case SCAN_PAGE_LRU:
 	case SCAN_DEL_PAGE_LRU:
 	case SCAN_PAGE_FILLED:
+	case SCAN_PMD_MIGRATION:
 		return -EAGAIN;
 	/*
 	 * Other: Trying again likely not to succeed / error intrinsic to
@@ -2802,6 +2809,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
 			goto handle_result;
 		/* Whitelisted set of results where continuing OK */
 		case SCAN_PMD_NULL:
+		case SCAN_PMD_MIGRATION:
 		case SCAN_PTE_NON_PRESENT:
 		case SCAN_PTE_UFFD_WP:
 		case SCAN_PAGE_RO:
#3 lore raw
Quoted 60 lines · Anshuman Khandual
Quoted 8 lines · Dev Jain

Suppose a folio is under migration, and khugepaged is also trying to collapse it. collapse_pte_mapped_thp() will retrieve the folio from the page cache via filemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in
__folio_migrate_mapping -> folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.

Could the migration be re-attempted after such failure ? Seems like the migration failure here is traded for a scan failure instead.

Quoted 2 lines

This fixes the migration-shared-anon-thp testcase failure on Apple M3.

Could you please provide some more context why this test case was failing earlier and how does this change here fixes the problem ?

Quoted 41 lines

Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".

Quoted 1 line

---

This patch was part of
https://lore.kernel.org/all/[email protected]/
but I have sent it separately on suggestion of Lorenzo, and also because I plan to send the first two patches after David Hildenbrand's folio_pte_batch series gets merged.

  mm/khugepaged.c | 12 ++++++++++--
  1 file changed, 10 insertions(+), 2 deletions(-)
mm/khugepaged.c +3 -0
index 1aa7ca67c756..99977bb9bf6a 100644
@@ -31,6 +31,7 @@ enum scan_result {
  	SCAN_FAIL,
  	SCAN_SUCCEED,
  	SCAN_PMD_NULL,
+	SCAN_PMD_MIGRATION,
  	SCAN_PMD_NONE,
  	SCAN_PMD_MAPPED,
  	SCAN_EXCEED_NONE_PTE,
@@ -941,6 +942,8 @@ static inline int check_pmd_state(pmd_t *pmd)
 
  	if (pmd_none(pmde))
  		return SCAN_PMD_NONE;
+	if (is_pmd_migration_entry(pmde))
+		return SCAN_PMD_MIGRATION;
  	if (!pmd_present(pmde))
  		return SCAN_PMD_NULL;
  	if (pmd_trans_huge(pmde))
@@ -1502,9 +1505,12 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
  	    !range_in_vma(vma, haddr, haddr + HPAGE_PMD_SIZE))
  		return SCAN_VMA_CHECK;
 
Quoted 9 lines

mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1513

Quoted 8 lines
mm/khugepaged.c +5 -2
-	/* Fast check before locking page if already PMD-mapped */
+	/*
+	 * Fast check before locking folio if already PMD-mapped, or if the
+	 * folio is under migration
+	 */
  	result = find_pmd_or_thp_or_none(mm, haddr, &pmd);
-	if (result == SCAN_PMD_MAPPED)
+	if (result == SCAN_PMD_MAPPED || result == SCAN_PMD_MIGRATION)

Should mapped PMD and migrating PMD be treated equally while scanning ?

Wanted to ask the same thing I think: why not simply use SCAN_PMD_MAPPED? After all, the folio is already pmd-mapped, just not using a present entry but (temporarily) using a migration entry.

Signature · 3 lines
Cheers,

David / dhildenb
author reviewer
#4 lore raw
In reply to Anshuman Khandual
Quoted 11 lines · Anshuman Khandual
Quoted 8 lines · Dev Jain

Suppose a folio is under migration, and khugepaged is also trying to collapse it. collapse_pte_mapped_thp() will retrieve the folio from the page cache via filemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in
__folio_migrate_mapping -> folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.

Could the migration be re-attempted after such failure ? Seems like the migration failure here is traded for a scan failure instead.

We already re-attempt migration. See NR_MAX_MIGRATE_PAGES_RETRY and NR_MAX_MIGRATE_ASYNC_RETRY. Also just before freezing the refcount, we do a suitable refcount check in folio_migrate_mapping(). So the race happens after this and folio_ref_freeze() in __folio_migrate_mapping(), therefore the window for the race is already very small in the migration path, but large in the khugepaged path.

Quoted 4 lines
Quoted 2 lines

This fixes the migration-shared-anon-thp testcase failure on Apple M3.

Could you please provide some more context why this test case was failing earlier and how does this change here fixes the problem ?

IMHO the explanation I have given in the patch description is clear and succinct: the testcase is failing due to the race. This patch shortens the race window, and the test on this particular hardware does not hit the race window again.

Quoted 41 lines
Quoted 41 lines

Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".

Quoted 1 line

---

This patch was part of
https://lore.kernel.org/all/[email protected]/
but I have sent it separately on suggestion of Lorenzo, and also because I plan to send the first two patches after David Hildenbrand's folio_pte_batch series gets merged.

  mm/khugepaged.c | 12 ++++++++++--
  1 file changed, 10 insertions(+), 2 deletions(-)
mm/khugepaged.c +3 -0
index 1aa7ca67c756..99977bb9bf6a 100644
@@ -31,6 +31,7 @@ enum scan_result {
  	SCAN_FAIL,
  	SCAN_SUCCEED,
  	SCAN_PMD_NULL,
+	SCAN_PMD_MIGRATION,
  	SCAN_PMD_NONE,
  	SCAN_PMD_MAPPED,
  	SCAN_EXCEED_NONE_PTE,
@@ -941,6 +942,8 @@ static inline int check_pmd_state(pmd_t *pmd)
 
  	if (pmd_none(pmde))
  		return SCAN_PMD_NONE;
+	if (is_pmd_migration_entry(pmde))
+		return SCAN_PMD_MIGRATION;
  	if (!pmd_present(pmde))
  		return SCAN_PMD_NULL;
  	if (pmd_trans_huge(pmde))
@@ -1502,9 +1505,12 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
  	    !range_in_vma(vma, haddr, haddr + HPAGE_PMD_SIZE))
  		return SCAN_VMA_CHECK;
 
Quoted 9 lines

mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1513

Quoted 8 lines
mm/khugepaged.c +5 -2
-	/* Fast check before locking page if already PMD-mapped */
+	/*
+	 * Fast check before locking folio if already PMD-mapped, or if the
+	 * folio is under migration
+	 */
  	result = find_pmd_or_thp_or_none(mm, haddr, &pmd);
-	if (result == SCAN_PMD_MAPPED)
+	if (result == SCAN_PMD_MAPPED || result == SCAN_PMD_MIGRATION)

Should mapped PMD and migrating PMD be treated equally while scanning ?

SCAN_PMD_MAPPED is used as an indicator to change result to SCAN_SUCCEED in khugepaged_scan_mm_slot: after the call to collapse_pte_mapped_thp. And, it is also used in madvise_collapse() to do ++thps which is used to set the return value of madvise_collapse. So I think this approach will be wrong.

Quoted 20 lines
Quoted 20 lines
mm/khugepaged.c +2 -0
  		return result;
 
  	/*
@@ -2716,6 +2722,7 @@ static int madvise_collapse_errno(enum scan_result r)
  	case SCAN_PAGE_LRU:
  	case SCAN_DEL_PAGE_LRU:
  	case SCAN_PAGE_FILLED:
+	case SCAN_PMD_MIGRATION:
  		return -EAGAIN;
  	/*
  	 * Other: Trying again likely not to succeed / error intrinsic to
@@ -2802,6 +2809,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
  			goto handle_result;
  		/* Whitelisted set of results where continuing OK */
  		case SCAN_PMD_NULL:
+		case SCAN_PMD_MIGRATION:
  		case SCAN_PTE_NON_PRESENT:
  		case SCAN_PTE_UFFD_WP:
  		case SCAN_PAGE_RO:
#5 lore raw
Quoted 83 lines · Dev Jain
Quoted 11 lines · Anshuman Khandual
Quoted 8 lines · Dev Jain

Suppose a folio is under migration, and khugepaged is also trying to collapse it. collapse_pte_mapped_thp() will retrieve the folio from the page cache via filemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in
__folio_migrate_mapping -> folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.

Could the migration be re-attempted after such failure ? Seems like the migration failure here is traded for a scan failure instead.

We already re-attempt migration. See NR_MAX_MIGRATE_PAGES_RETRY and NR_MAX_MIGRATE_ASYNC_RETRY. Also just before freezing the refcount, we do a suitable refcount check in folio_migrate_mapping(). So the race happens after this and folio_ref_freeze() in __folio_migrate_mapping(), therefore the window for the race is already very small in the migration path, but large in the khugepaged path.

Quoted 4 lines
Quoted 2 lines

This fixes the migration-shared-anon-thp testcase failure on Apple M3.

Could you please provide some more context why this test case was failing earlier and how does this change here fixes the problem ?

IMHO the explanation I have given in the patch description is clear and succinct: the testcase is failing due to the race. This patch shortens the race window, and the test on this particular hardware does not hit the race window again.

Quoted 41 lines
Quoted 41 lines

Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".

Quoted 1 line

---

This patch was part of
https://lore.kernel.org/all/[email protected]/
but I have sent it separately on suggestion of Lorenzo, and also because I plan to send the first two patches after David Hildenbrand's folio_pte_batch series gets merged.

   mm/khugepaged.c | 12 ++++++++++--
   1 file changed, 10 insertions(+), 2 deletions(-)
mm/khugepaged.c +3 -0
index 1aa7ca67c756..99977bb9bf6a 100644
@@ -31,6 +31,7 @@ enum scan_result {
   	SCAN_FAIL,
   	SCAN_SUCCEED,
   	SCAN_PMD_NULL,
+	SCAN_PMD_MIGRATION,
   	SCAN_PMD_NONE,
   	SCAN_PMD_MAPPED,
   	SCAN_EXCEED_NONE_PTE,
@@ -941,6 +942,8 @@ static inline int check_pmd_state(pmd_t *pmd)
 
   	if (pmd_none(pmde))
   		return SCAN_PMD_NONE;
+	if (is_pmd_migration_entry(pmde))
+		return SCAN_PMD_MIGRATION;
   	if (!pmd_present(pmde))
   		return SCAN_PMD_NULL;
   	if (pmd_trans_huge(pmde))
@@ -1502,9 +1505,12 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
   	    !range_in_vma(vma, haddr, haddr + HPAGE_PMD_SIZE))
   		return SCAN_VMA_CHECK;
 
Quoted 9 lines

mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1513

Quoted 8 lines
mm/khugepaged.c +5 -2
-	/* Fast check before locking page if already PMD-mapped */
+	/*
+	 * Fast check before locking folio if already PMD-mapped, or if the
+	 * folio is under migration
+	 */
   	result = find_pmd_or_thp_or_none(mm, haddr, &pmd);
-	if (result == SCAN_PMD_MAPPED)
+	if (result == SCAN_PMD_MAPPED || result == SCAN_PMD_MIGRATION)

Should mapped PMD and migrating PMD be treated equally while scanning ?

Quoted 4 lines

SCAN_PMD_MAPPED is used as an indicator to change result to SCAN_SUCCEED in khugepaged_scan_mm_slot: after the call to collapse_pte_mapped_thp. And, it is also used in madvise_collapse() to do ++thps which is used to set the return value of madvise_collapse. So I think this approach will be wrong.

But if it already is PMD mapped (just temporarily through a migration entry), isn't this exactly what we want?

Signature · 3 lines
Cheers,

David / dhildenb
author reviewer
#6 lore raw
Quoted 83 lines · David Hildenbrand
Quoted 82 lines · Dev Jain
Quoted 11 lines · Anshuman Khandual
Quoted 8 lines · Dev Jain

Suppose a folio is under migration, and khugepaged is also trying to collapse it. collapse_pte_mapped_thp() will retrieve the folio from the page cache via filemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in
__folio_migrate_mapping -> folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.

Could the migration be re-attempted after such failure ? Seems like the migration failure here is traded for a scan failure instead.

We already re-attempt migration. See NR_MAX_MIGRATE_PAGES_RETRY and NR_MAX_MIGRATE_ASYNC_RETRY. Also just before freezing the refcount, we do a suitable refcount check in folio_migrate_mapping(). So the race happens after this and folio_ref_freeze() in __folio_migrate_mapping(), therefore the window for the race is already very small in the migration path, but large in the khugepaged path.

Quoted 4 lines
Quoted 2 lines

This fixes the migration-shared-anon-thp testcase failure on Apple M3.

Could you please provide some more context why this test case was failing earlier and how does this change here fixes the problem ?

IMHO the explanation I have given in the patch description is clear and succinct: the testcase is failing due to the race. This patch shortens the race window, and the test on this particular hardware does not hit the race window again.

Quoted 50 lines
Quoted 41 lines

Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".

Quoted 1 line

---

This patch was part of
https://lore.kernel.org/all/[email protected]/
but I have sent it separately on suggestion of Lorenzo, and also because I plan to send the first two patches after David Hildenbrand's folio_pte_batch series gets merged.

   mm/khugepaged.c | 12 ++++++++++--

   1 file changed, 10 insertions(+), 2 deletions(-)
mm/khugepaged.c +3 -0
index 1aa7ca67c756..99977bb9bf6a 100644
@@ -31,6 +31,7 @@ enum scan_result {
        SCAN_FAIL,
        SCAN_SUCCEED,
        SCAN_PMD_NULL,
+    SCAN_PMD_MIGRATION,
        SCAN_PMD_NONE,
        SCAN_PMD_MAPPED,
        SCAN_EXCEED_NONE_PTE,
@@ -941,6 +942,8 @@ static inline int check_pmd_state(pmd_t *pmd)
           if (pmd_none(pmde))
            return SCAN_PMD_NONE;
+    if (is_pmd_migration_entry(pmde))
+        return SCAN_PMD_MIGRATION;
        if (!pmd_present(pmde))
            return SCAN_PMD_NULL;
        if (pmd_trans_huge(pmde))
@@ -1502,9 +1505,12 @@ int collapse_pte_mapped_thp(struct mm_struct 
 *mm, unsigned long addr,
            !range_in_vma(vma, haddr, haddr + HPAGE_PMD_SIZE))
            return SCAN_VMA_CHECK;

   -    /* Fast check before locking page if already PMD-mapped */

Quoted 8 lines

+    /*
+     * Fast check before locking folio if already PMD-mapped, or if the
+     * folio is under migration
+     */
       result = find_pmd_or_thp_or_none(mm, haddr, &pmd); -    if (result == SCAN_PMD_MAPPED)
+    if (result == SCAN_PMD_MAPPED || result == SCAN_PMD_MIGRATION)

Should mapped PMD and migrating PMD be treated equally while scanning ?

Quoted 6 lines
Quoted 4 lines

SCAN_PMD_MAPPED is used as an indicator to change result to SCAN_SUCCEED in khugepaged_scan_mm_slot: after the call to collapse_pte_mapped_thp. And, it is also used in madvise_collapse() to do ++thps which is used to set the return value of madvise_collapse. So I think this approach will be wrong.

But if it already is PMD mapped (just temporarily through a migration entry), isn't this exactly what we want?

Good point. I was about to say that what about PMD-folio splitting during migration, but then

during unmapping the folios, if we cannot migrate the PMD-folios, they will be splitted via

unmap_folio() along with the PMD, therefore we can be sure that if we encounter a PMD

migration entry, then eventually it will be converted to a PMD leaf entry on migration success

or failure.

I'll merge this into SCAN_PMD_MAPPED, thanks.

#7 lore raw
In reply to Dev Jain
Quoted 8 lines · Dev Jain

Suppose a folio is under migration, and khugepaged is also trying to collapse it. collapse_pte_mapped_thp() will retrieve the folio from the page cache via filemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in
__folio_migrate_mapping -> folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.

This is a nice find!

Quoted 6 lines

This fixes the migration-shared-anon-thp testcase failure on Apple M3.

Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".

Thanks for separating this out, appreciated!

Quoted 26 lines

---

This patch was part of
https://lore.kernel.org/all/[email protected]/
but I have sent it separately on suggestion of Lorenzo, and also because I plan to send the first two patches after David Hildenbrand's folio_pte_batch series gets merged.

 mm/khugepaged.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
mm/khugepaged.c +1 -0
index 1aa7ca67c756..99977bb9bf6a 100644
@@ -31,6 +31,7 @@ enum scan_result {
 	SCAN_FAIL,
 	SCAN_SUCCEED,
 	SCAN_PMD_NULL,
+	SCAN_PMD_MIGRATION,
 	SCAN_PMD_NONE,
 	SCAN_PMD_MAPPED,
 	SCAN_EXCEED_NONE_PTE,

mm/khugepaged.c static inline int check_pmd_state(pmd_t *pmd) · line 946

Quoted 5 lines
mm/khugepaged.c +2 -0
@@ -941,6 +942,8 @@ static inline int check_pmd_state(pmd_t *pmd)
 
 	if (pmd_none(pmde))
 		return SCAN_PMD_NONE;
+	if (is_pmd_migration_entry(pmde))
+		return SCAN_PMD_MIGRATION;

With David's suggestions I guess this boils down to simply adding this line.

Could we add a quick comment to explain why here?

Thanks!

Quoted 37 lines
mm/khugepaged.c +7 -3
 	if (!pmd_present(pmde))
 		return SCAN_PMD_NULL;
 	if (pmd_trans_huge(pmde))
@@ -1502,9 +1505,12 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
 	    !range_in_vma(vma, haddr, haddr + HPAGE_PMD_SIZE))
 		return SCAN_VMA_CHECK;
 
-	/* Fast check before locking page if already PMD-mapped */
+	/*
+	 * Fast check before locking folio if already PMD-mapped, or if the
+	 * folio is under migration
+	 */
 	result = find_pmd_or_thp_or_none(mm, haddr, &pmd);
-	if (result == SCAN_PMD_MAPPED)
+	if (result == SCAN_PMD_MAPPED || result == SCAN_PMD_MIGRATION)
 		return result;
 
 	/*
@@ -2716,6 +2722,7 @@ static int madvise_collapse_errno(enum scan_result r)
 	case SCAN_PAGE_LRU:
 	case SCAN_DEL_PAGE_LRU:
 	case SCAN_PAGE_FILLED:
+	case SCAN_PMD_MIGRATION:
 		return -EAGAIN;
 	/*
 	 * Other: Trying again likely not to succeed / error intrinsic to
@@ -2802,6 +2809,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
 			goto handle_result;
 		/* Whitelisted set of results where continuing OK */
 		case SCAN_PMD_NULL:
+		case SCAN_PMD_MIGRATION:
 		case SCAN_PTE_NON_PRESENT:
 		case SCAN_PTE_UFFD_WP:
 		case SCAN_PAGE_RO:
--

2.30.2

author reviewer
#8 lore raw
Quoted 44 lines · Lorenzo Stoakes
Quoted 8 lines · Dev Jain

Suppose a folio is under migration, and khugepaged is also trying to collapse it. collapse_pte_mapped_thp() will retrieve the folio from the page cache via filemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in
__folio_migrate_mapping -> folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.

This is a nice find!

Quoted 5 lines

This fixes the migration-shared-anon-thp testcase failure on Apple M3.

Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".

Thanks for separating this out, appreciated!

Quoted 26 lines

---

This patch was part of
https://lore.kernel.org/all/[email protected]/
but I have sent it separately on suggestion of Lorenzo, and also because I plan to send the first two patches after David Hildenbrand's folio_pte_batch series gets merged.

  mm/khugepaged.c | 12 ++++++++++--
  1 file changed, 10 insertions(+), 2 deletions(-)
mm/khugepaged.c +1 -0
index 1aa7ca67c756..99977bb9bf6a 100644
@@ -31,6 +31,7 @@ enum scan_result {
  	SCAN_FAIL,
  	SCAN_SUCCEED,
  	SCAN_PMD_NULL,
+	SCAN_PMD_MIGRATION,
  	SCAN_PMD_NONE,
  	SCAN_PMD_MAPPED,
  	SCAN_EXCEED_NONE_PTE,
@@ -941,6 +942,8 @@ static inline int check_pmd_state(pmd_t *pmd)
 
Quoted 5 lines

mm/khugepaged.c static inline int check_pmd_state(pmd_t *pmd) · line 946

Quoted 4 lines
mm/khugepaged.c +2 -0
  	if (pmd_none(pmde))
  		return SCAN_PMD_NONE;
+	if (is_pmd_migration_entry(pmde))
+		return SCAN_PMD_MIGRATION;

With David's suggestions I guess this boils down to simply adding this line.

I think it should be

mm/khugepaged.c +2 -2
index 1aa7ca67c756..8a6ba5c8ba4d 100644
@@ -941,10 +941,10 @@ static inline int check_pmd_state(pmd_t *pmd)
 
 	if (pmd_none(pmde))
 		return SCAN_PMD_NONE;
+	if (is_pmd_migration_entry(pmde) || pmd_trans_huge(pmde))
+		return SCAN_PMD_MAPPED;
 	if (!pmd_present(pmde))
 		return SCAN_PMD_NULL;
-	if (pmd_trans_huge(pmde))
-		return SCAN_PMD_MAPPED;
 	if (pmd_bad(pmde))
 		return SCAN_PMD_NULL;
 	return SCAN_SUCCEED;

Moving this line above since we don't want to exit prematurely due to !pmd_present(pmde).

Quoted 2 lines

Could we add a quick comment to explain why here?

Sure.

Quoted 40 lines

Thanks!

Quoted 37 lines
mm/khugepaged.c +7 -3
  	if (!pmd_present(pmde))
  		return SCAN_PMD_NULL;
  	if (pmd_trans_huge(pmde))
@@ -1502,9 +1505,12 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
  	    !range_in_vma(vma, haddr, haddr + HPAGE_PMD_SIZE))
  		return SCAN_VMA_CHECK;
 
-	/* Fast check before locking page if already PMD-mapped */
+	/*
+	 * Fast check before locking folio if already PMD-mapped, or if the
+	 * folio is under migration
+	 */
  	result = find_pmd_or_thp_or_none(mm, haddr, &pmd);
-	if (result == SCAN_PMD_MAPPED)
+	if (result == SCAN_PMD_MAPPED || result == SCAN_PMD_MIGRATION)
  		return result;
 
  	/*
@@ -2716,6 +2722,7 @@ static int madvise_collapse_errno(enum scan_result r)
  	case SCAN_PAGE_LRU:
  	case SCAN_DEL_PAGE_LRU:
  	case SCAN_PAGE_FILLED:
+	case SCAN_PMD_MIGRATION:
  		return -EAGAIN;
  	/*
  	 * Other: Trying again likely not to succeed / error intrinsic to
@@ -2802,6 +2809,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
  			goto handle_result;
  		/* Whitelisted set of results where continuing OK */
  		case SCAN_PMD_NULL:
+		case SCAN_PMD_MIGRATION:
  		case SCAN_PTE_NON_PRESENT:
  		case SCAN_PTE_UFFD_WP:
  		case SCAN_PAGE_RO:
--

2.30.2

#9 lore raw
Quoted 72 lines · Dev Jain
Quoted 44 lines · Lorenzo Stoakes
Quoted 8 lines · Dev Jain

Suppose a folio is under migration, and khugepaged is also trying to collapse it. collapse_pte_mapped_thp() will retrieve the folio from the page cache via filemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in
__folio_migrate_mapping -> folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.

This is a nice find!

Quoted 5 lines

This fixes the migration-shared-anon-thp testcase failure on Apple M3.

Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".

Thanks for separating this out, appreciated!

Quoted 26 lines

---

This patch was part of
https://lore.kernel.org/all/[email protected]/
but I have sent it separately on suggestion of Lorenzo, and also because I plan to send the first two patches after David Hildenbrand's folio_pte_batch series gets merged.

  mm/khugepaged.c | 12 ++++++++++--

  1 file changed, 10 insertions(+), 2 deletions(-)
mm/khugepaged.c +1 -0
index 1aa7ca67c756..99977bb9bf6a 100644
@@ -31,6 +31,7 @@ enum scan_result {
       SCAN_FAIL,
       SCAN_SUCCEED,
       SCAN_PMD_NULL,
+    SCAN_PMD_MIGRATION,
       SCAN_PMD_NONE,
       SCAN_PMD_MAPPED,
       SCAN_EXCEED_NONE_PTE,
@@ -941,6 +942,8 @@ static inline int check_pmd_state(pmd_t *pmd)
 
Quoted 5 lines

mm/khugepaged.c static inline int check_pmd_state(pmd_t *pmd)

Quoted 4 lines
mm/khugepaged.c +2 -0
       if (pmd_none(pmde))
           return SCAN_PMD_NONE;
+    if (is_pmd_migration_entry(pmde))
+        return SCAN_PMD_MIGRATION;

With David's suggestions I guess this boils down to simply adding this line.

I think it should be

mm/khugepaged.c +0 -0
index 1aa7ca67c756..8a6ba5c8ba4d 100644
@@ -941,10 +941,10 @@ static inline int check_pmd_state(pmd_t *pmd)


     if (pmd_none(pmde))
         return SCAN_PMD_NONE;
+    if (is_pmd_migration_entry(pmde) || pmd_trans_huge(pmde)) +        return SCAN_PMD_MAPPED;
     if (!pmd_present(pmde))
         return SCAN_PMD_NULL;
-    if (pmd_trans_huge(pmde))
-        return SCAN_PMD_MAPPED;
     if (pmd_bad(pmde))
         return SCAN_PMD_NULL;
     return SCAN_SUCCEED;

Quoted 2 lines

Moving this line above since we don't want to exit prematurely due to !pmd_present(pmde).

Might be cleaner to just add the migration test separately before the pmd_present() and without modifying existing pmd_trans_huge().

	if (is_pmd_migration_entry(pmde))
		return SCAN_PMD_MAPPED;
Quoted 48 lines
Quoted 4 lines

Could we add a quick comment to explain why here?

Sure.

Quoted 40 lines

Thanks!

Quoted 37 lines
mm/khugepaged.c +5 -2
       if (!pmd_present(pmde))
           return SCAN_PMD_NULL;
       if (pmd_trans_huge(pmde))
@@ -1502,9 +1505,12 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
           !range_in_vma(vma, haddr, haddr + HPAGE_PMD_SIZE))
           return SCAN_VMA_CHECK;
 
-    /* Fast check before locking page if already PMD-mapped */
+    /*
+     * Fast check before locking folio if already PMD-mapped, or if the
+     * folio is under migration
+     */
       result = find_pmd_or_thp_or_none(mm, haddr, &pmd);
-    if (result == SCAN_PMD_MAPPED)
+    if (result == SCAN_PMD_MAPPED || result == SCAN_PMD_MIGRATION)
           return result;
 
       /*
@@ -2716,6 +2722,7 @@ static int madvise_collapse_errno(enum scan_result r)

      case SCAN_PAGE_LRU:
      case SCAN_DEL_PAGE_LRU:
      case SCAN_PAGE_FILLED:
+    case SCAN_PMD_MIGRATION:
          return -EAGAIN;
      /*
       * Other: Trying again likely not to succeed / error intrinsic to

mm/khugepaged.c +0 -0
@@ -2802,6 +2809,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
               goto handle_result;
           /* Whitelisted set of results where continuing OK */

          case SCAN_PMD_NULL:
+        case SCAN_PMD_MIGRATION:
          case SCAN_PTE_NON_PRESENT:
          case SCAN_PTE_UFFD_WP:
          case SCAN_PAGE_RO:

Signature · 1 line
2.30.2
author reviewer
#10 lore raw
Quoted 72 lines · Anshuman Khandual
Quoted 70 lines · Dev Jain
Quoted 44 lines · Lorenzo Stoakes
Quoted 8 lines · Dev Jain

Suppose a folio is under migration, and khugepaged is also trying to collapse it. collapse_pte_mapped_thp() will retrieve the folio from the page cache via filemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in
__folio_migrate_mapping -> folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.

This is a nice find!

Quoted 5 lines

This fixes the migration-shared-anon-thp testcase failure on Apple M3.

Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".

Thanks for separating this out, appreciated!

Quoted 26 lines

---

This patch was part of
https://lore.kernel.org/all/[email protected]/
but I have sent it separately on suggestion of Lorenzo, and also because I plan to send the first two patches after David Hildenbrand's folio_pte_batch series gets merged.

   mm/khugepaged.c | 12 ++++++++++--
   1 file changed, 10 insertions(+), 2 deletions(-)
mm/khugepaged.c +1 -0
index 1aa7ca67c756..99977bb9bf6a 100644
@@ -31,6 +31,7 @@ enum scan_result {
       SCAN_FAIL,
       SCAN_SUCCEED,
       SCAN_PMD_NULL,
+    SCAN_PMD_MIGRATION,
       SCAN_PMD_NONE,
       SCAN_PMD_MAPPED,
       SCAN_EXCEED_NONE_PTE,
@@ -941,6 +942,8 @@ static inline int check_pmd_state(pmd_t *pmd)
 
Quoted 5 lines

mm/khugepaged.c static inline int check_pmd_state(pmd_t *pmd)

Quoted 4 lines
mm/khugepaged.c +2 -0
       if (pmd_none(pmde))
           return SCAN_PMD_NONE;
+    if (is_pmd_migration_entry(pmde))
+        return SCAN_PMD_MIGRATION;

With David's suggestions I guess this boils down to simply adding this line.

I think it should be

mm/khugepaged.c +2 -2
index 1aa7ca67c756..8a6ba5c8ba4d 100644
@@ -941,10 +941,10 @@ static inline int check_pmd_state(pmd_t *pmd)
 
      if (pmd_none(pmde))
          return SCAN_PMD_NONE;
+    if (is_pmd_migration_entry(pmde) || pmd_trans_huge(pmde))
+        return SCAN_PMD_MAPPED;
      if (!pmd_present(pmde))
          return SCAN_PMD_NULL;
-    if (pmd_trans_huge(pmde))
-        return SCAN_PMD_MAPPED;
      if (pmd_bad(pmde))
          return SCAN_PMD_NULL;
      return SCAN_SUCCEED;
Quoted 7 lines
Quoted 2 lines

Moving this line above since we don't want to exit prematurely due to !pmd_present(pmde).

Might be cleaner to just add the migration test separately before the pmd_present() and without modifying existing pmd_trans_huge().

	if (is_pmd_migration_entry(pmde))
		return SCAN_PMD_MAPPED;

Sounds good.

Quoted 43 lines
Quoted 43 lines
Quoted 2 lines

Could we add a quick comment to explain why here?

Sure.

Quoted 39 lines

Thanks!

Quoted 37 lines
mm/khugepaged.c +7 -2
       if (!pmd_present(pmde))
           return SCAN_PMD_NULL;
       if (pmd_trans_huge(pmde))
@@ -1502,9 +1505,12 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
           !range_in_vma(vma, haddr, haddr + HPAGE_PMD_SIZE))
           return SCAN_VMA_CHECK;
 
-    /* Fast check before locking page if already PMD-mapped */
+    /*
+     * Fast check before locking folio if already PMD-mapped, or if the
+     * folio is under migration
+     */
       result = find_pmd_or_thp_or_none(mm, haddr, &pmd);
-    if (result == SCAN_PMD_MAPPED)
+    if (result == SCAN_PMD_MAPPED || result == SCAN_PMD_MIGRATION)
           return result;
 
       /*
@@ -2716,6 +2722,7 @@ static int madvise_collapse_errno(enum scan_result r)
       case SCAN_PAGE_LRU:
       case SCAN_DEL_PAGE_LRU:
       case SCAN_PAGE_FILLED:
+    case SCAN_PMD_MIGRATION:
           return -EAGAIN;
       /*
        * Other: Trying again likely not to succeed / error intrinsic to
@@ -2802,6 +2809,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
               goto handle_result;
           /* Whitelisted set of results where continuing OK */
           case SCAN_PMD_NULL:
+        case SCAN_PMD_MIGRATION:
           case SCAN_PTE_NON_PRESENT:
           case SCAN_PTE_UFFD_WP:
           case SCAN_PAGE_RO:
Signature · 1 line
2.30.2