Skip to contentSkip to conversation

khugepaged: Reduce race probability between migration and khugepaged

PATCH Superseded Dev Jain 6 participants 9 messages Jul 3, 2025 – Jul 4, 2025

Subsystems (2)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Dev Jain

author reviewer
#0 lore raw

PATCH v2 1/1 khugepaged: Reduce race probability between migration and khugepaged

Suppose a folio is under migration, and khugepaged is also trying to collapse it. collapse_pte_mapped_thp() will retrieve the folio from the page cache via filemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in
__folio_migrate_mapping -> folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.

This fixes the migration-shared-anon-thp testcase failure on Apple M3.

Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".

---

v1->v2:
- Remove SCAN_PMD_MIGRATION, merge into SCAN_PMD_MAPPED (David, Anshuman)
- Add a comment (Lorenzo)

v1:
- https://lore.kernel.org/all/[email protected]/

 mm/khugepaged.c | 9 +++++++++
 1 file changed, 9 insertions(+)
mm/khugepaged.c +9 -0
index 1aa7ca67c756..3fdefc4f4984 100644
@@ -941,6 +941,15 @@ static inline int check_pmd_state(pmd_t *pmd)
 
 	if (pmd_none(pmde))
 		return SCAN_PMD_NONE;
+
+	/*
+	 * The folio may be under migration when khugepaged is trying to
+	 * collapse it. Migration success or failure will eventually end
+	 * up with the PMD still pointing to a PMD-order folio, so return
+	 * SCAN_PMD_MAPPED.
+	 */
+	if (is_pmd_migration_entry(pmde))
+		return SCAN_PMD_MAPPED;
 	if (!pmd_present(pmde))
 		return SCAN_PMD_NULL;
 	if (pmd_trans_huge(pmde))
Signature · 1 line
2.30.2
Acked-by on v2 1/1
#1 lore raw
Quoted 33 lines · Dev Jain

Suppose a folio is under migration, and khugepaged is also trying to collapse it. collapse_pte_mapped_thp() will retrieve the folio from the page cache via filemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in
__folio_migrate_mapping -> folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.

This fixes the migration-shared-anon-thp testcase failure on Apple M3.

Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".

---

v1->v2:
- Remove SCAN_PMD_MIGRATION, merge into SCAN_PMD_MAPPED (David, Anshuman)
- Add a comment (Lorenzo)

v1:
- https://lore.kernel.org/all/[email protected]/

  mm/khugepaged.c | 9 +++++++++
  1 file changed, 9 insertions(+)
mm/khugepaged.c +0 -0
index 1aa7ca67c756..3fdefc4f4984 100644

mm/khugepaged.c static inline int check_pmd_state(pmd_t *pmd) · line 949

Quoted 9 lines
mm/khugepaged.c +6 -0
@@ -941,6 +941,15 @@ static inline int check_pmd_state(pmd_t *pmd)
 
  	if (pmd_none(pmde))
  		return SCAN_PMD_NONE;
+
+	/*
+	 * The folio may be under migration when khugepaged is trying to
+	 * collapse it. Migration success or failure will eventually end
+	 * up with the PMD still pointing to a PMD-order folio, so return
+	 * SCAN_PMD_MAPPED.

Nit: the last part (, so return ..) is obvious from the code.

I would have written

/*
* The folio may be under migration when khugepaged is trying to
* collapse it. Migration success or failure will eventually end
* up with a present PMD entry again.
*/

Signature · 3 lines
Cheers,

David / dhildenb
Reviewed-by
#2 lore raw
Quoted 45 lines · David Hildenbrand
Quoted 33 lines · Dev Jain

Suppose a folio is under migration, and khugepaged is also trying to collapse it. collapse_pte_mapped_thp() will retrieve the folio from the page cache via filemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in
__folio_migrate_mapping -> folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.

This fixes the migration-shared-anon-thp testcase failure on Apple M3.

Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".

---

v1->v2:
  - Remove SCAN_PMD_MIGRATION, merge into SCAN_PMD_MAPPED (David, Anshuman)
  - Add a comment (Lorenzo)

v1:
  - https://lore.kernel.org/all/[email protected]/

  mm/khugepaged.c | 9 +++++++++

  1 file changed, 9 insertions(+)
mm/khugepaged.c +0 -0
index 1aa7ca67c756..3fdefc4f4984 100644

mm/khugepaged.c static inline int check_pmd_state(pmd_t *pmd) · line 948

Quoted 8 lines
mm/khugepaged.c +6 -0
@@ -941,6 +941,15 @@ static inline int check_pmd_state(pmd_t *pmd)
         if (pmd_none(pmde))
           return SCAN_PMD_NONE;
+
+    /*
+     * The folio may be under migration when khugepaged is trying to
+     * collapse it. Migration success or failure will eventually end
+     * up with the PMD still pointing to a PMD-order folio, so return
+     * SCAN_PMD_MAPPED.

Nit: the last part (, so return ..) is obvious from the code.

Quoted 7 lines

I would have written

/*
 * The folio may be under migration when khugepaged is trying to
 * collapse it. Migration success or failure will eventually end
 * up with a present PMD entry again.
 */

+1

Quoted 4 lines
author reviewer
#3 lore raw
In reply to David Hildenbrand
Quoted 45 lines · David Hildenbrand
Quoted 33 lines · Dev Jain

Suppose a folio is under migration, and khugepaged is also trying to collapse it. collapse_pte_mapped_thp() will retrieve the folio from the page cache via filemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in
__folio_migrate_mapping -> folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.

This fixes the migration-shared-anon-thp testcase failure on Apple M3.

Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".

---

v1->v2:
  - Remove SCAN_PMD_MIGRATION, merge into SCAN_PMD_MAPPED (David, Anshuman)
  - Add a comment (Lorenzo)

v1:
  - https://lore.kernel.org/all/[email protected]/

  mm/khugepaged.c | 9 +++++++++

  1 file changed, 9 insertions(+)
mm/khugepaged.c +0 -0
index 1aa7ca67c756..3fdefc4f4984 100644

mm/khugepaged.c static inline int check_pmd_state(pmd_t *pmd) · line 948

Quoted 8 lines
mm/khugepaged.c +6 -0
@@ -941,6 +941,15 @@ static inline int check_pmd_state(pmd_t *pmd)
         if (pmd_none(pmde))
           return SCAN_PMD_NONE;
+
+    /*
+     * The folio may be under migration when khugepaged is trying to
+     * collapse it. Migration success or failure will eventually end
+     * up with the PMD still pointing to a PMD-order folio, so return
+     * SCAN_PMD_MAPPED.

Nit: the last part (, so return ..) is obvious from the code.

Quoted 8 lines

I would have written

/*
 * The folio may be under migration when khugepaged is trying to
 * collapse it. Migration success or failure will eventually end
 * up with a present PMD entry again.
 */

Thanks for the suggestion, but

PMD pointing to PMD-order folio necessarily implies present PMD entry,

but the converse is not true? For example it may point to a PTE table.

So I wanted to make it explicitly clear that the transient entry will be

converted to a PMD-THP in the end.

Quoted 2 lines

Thanks!

#4 lore raw
Quoted 57 lines · Dev Jain
Quoted 47 lines · David Hildenbrand
Quoted 35 lines · Dev Jain

Suppose a folio is under migration, and khugepaged is also trying to collapse it. collapse_pte_mapped_thp() will retrieve the folio from the page cache via filemap_lock_folio(), thus taking a reference on the folio
and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in
__folio_migrate_mapping -> folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.

This fixes the migration-shared-anon-thp testcase failure on Apple M3.

Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".

---

v1->v2:
  - Remove SCAN_PMD_MIGRATION, merge into SCAN_PMD_MAPPED (David,
Anshuman)
  - Add a comment (Lorenzo)

v1:
  - https://lore.kernel.org/all/[email protected]/

   mm/khugepaged.c | 9 +++++++++
   1 file changed, 9 insertions(+)
mm/khugepaged.c +0 -0
index 1aa7ca67c756..3fdefc4f4984 100644

mm/khugepaged.c static inline int check_pmd_state(pmd_t *pmd) · line 948

Quoted 8 lines
mm/khugepaged.c +6 -0
@@ -941,6 +941,15 @@ static inline int check_pmd_state(pmd_t *pmd)
         if (pmd_none(pmde))
           return SCAN_PMD_NONE;
+
+    /*
+     * The folio may be under migration when khugepaged is trying to
+     * collapse it. Migration success or failure will eventually end
+     * up with the PMD still pointing to a PMD-order folio, so return
+     * SCAN_PMD_MAPPED.

Nit: the last part (, so return ..) is obvious from the code.

Quoted 8 lines

I would have written

/*
 * The folio may be under migration when khugepaged is trying to
 * collapse it. Migration success or failure will eventually end
 * up with a present PMD entry again.
 */

Quoted 4 lines

Thanks for the suggestion, but

PMD pointing to PMD-order folio necessarily implies present PMD entry,

Quoted 1 line

but the converse is not true? For example it may point to a PTE table.

I see, talking about orders is confusing though.

"with a present PMD mapping a folio again."

Signature · 3 lines
Cheers,

David / dhildenb
author reviewer
#5 lore raw
Quoted 60 lines · David Hildenbrand
Quoted 59 lines · Dev Jain
Quoted 47 lines · David Hildenbrand
Quoted 35 lines · Dev Jain

Suppose a folio is under migration, and khugepaged is also trying to collapse it. collapse_pte_mapped_thp() will retrieve the folio from the page cache via filemap_lock_folio(), thus taking a reference on the folio
and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in
__folio_migrate_mapping -> folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.

This fixes the migration-shared-anon-thp testcase failure on Apple M3.

Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".

---

v1->v2:
   - Remove SCAN_PMD_MIGRATION, merge into SCAN_PMD_MAPPED (David, Anshuman)
   - Add a comment (Lorenzo)

v1:
   - https://lore.kernel.org/all/[email protected]/

   mm/khugepaged.c | 9 +++++++++

   1 file changed, 9 insertions(+)
mm/khugepaged.c +0 -0
index 1aa7ca67c756..3fdefc4f4984 100644

mm/khugepaged.c static inline int check_pmd_state(pmd_t *pmd) · line 948

Quoted 8 lines
mm/khugepaged.c +6 -0
@@ -941,6 +941,15 @@ static inline int check_pmd_state(pmd_t *pmd)
          if (pmd_none(pmde))
            return SCAN_PMD_NONE;
+
+    /*
+     * The folio may be under migration when khugepaged is trying to
+     * collapse it. Migration success or failure will eventually end
+     * up with the PMD still pointing to a PMD-order folio, so return
+     * SCAN_PMD_MAPPED.

Nit: the last part (, so return ..) is obvious from the code.

Quoted 8 lines

I would have written

/*
  * The folio may be under migration when khugepaged is trying to
  * collapse it. Migration success or failure will eventually end
  * up with a present PMD entry again.
  */

Thanks for the suggestion, but

Quoted 7 lines
Quoted 2 lines

PMD pointing to PMD-order folio necessarily implies present PMD entry,

Quoted 1 line

but the converse is not true? For example it may point to a PTE

table.

I see, talking about orders is confusing though.

"with a present PMD mapping a folio again."

Sure. Shall respin.

Acked-by on v2 1/1
#6 lore raw
In reply to Dev Jain
Quoted 9 lines · Dev Jain

Suppose a folio is under migration, and khugepaged is also trying to collapse it. collapse_pte_mapped_thp() will retrieve the folio from the page cache via filemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in
__folio_migrate_mapping -> folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.

Quoted 7 lines

This fixes the migration-shared-anon-thp testcase failure on Apple M3.

Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".

David's comment refering to a 'present PMD entry' seems more clear to me, but

Signature · 2 lines
Oscar Salvador
SUSE Labs
reviewer Reviewed-by on v2 1/1
#7 lore raw
In reply to Dev Jain
Quoted 22 lines · Dev Jain

Suppose a folio is under migration, and khugepaged is also trying to collapse it. collapse_pte_mapped_thp() will retrieve the folio from the page cache via filemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in
__folio_migrate_mapping -> folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.

This fixes the migration-shared-anon-thp testcase failure on Apple M3.

Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".

---

v1->v2:
- Remove SCAN_PMD_MIGRATION, merge into SCAN_PMD_MAPPED (David, Anshuman)
- Add a comment (Lorenzo)

Quoted 6 lines

v1:
- https://lore.kernel.org/all/[email protected]/

 mm/khugepaged.c | 9 +++++++++
 1 file changed, 9 insertions(+)

Best Regards,
Yan, Zi

reviewer Reviewed-by on v2 1/1
#8 lore raw
In reply to Dev Jain
Quoted 9 lines · Dev Jain

Suppose a folio is under migration, and khugepaged is also trying to collapse it. collapse_pte_mapped_thp() will retrieve the folio from the page cache via filemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in
__folio_migrate_mapping -> folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.

Quoted 7 lines

This fixes the migration-shared-anon-thp testcase failure on Apple M3.

Note that, this is not a "fix" since it only reduces the chance of interference of khugepaged with migration, wherein both the kernel functionalities are deemed "best-effort".

With David's comments addressed, LGTM.