mm: handle device-private PMDs in walk callbacks

queue_folios_pmd() is called under pmd_trans_huge_lock(), whose pmd_is_huge() check returns true for any non-present, non-none PMD softleaf. Passing such a PMD to pmd_folio() treats the softleaf encoding as a hardware PFN and can return a bogus folio pointer.

Mirror queue_folios_pte_range(): handle non-present entries before looking up a folio. Keep migration entries counted as failures, but skip other non-present PMDs such as device-private entries.

Potential trigger: an HMM-based GPU driver migrates an anonymous THP folio to device memory via migrate_vma_pages(), leaving a device-private PMD. Userspace then calls mbind(), migrate_pages() or set_mempolicy_home_node() on that range.

Reported-by sashiko-bot <[email protected]>

Link https://sashiko.dev/#/patchset/20260703173903.3789516-1-usama.arif%40linux.dev?part=6

Fixes 368076f52ebe ("mm/huge_memory: add device-private THP support to PMD operations")

Cc <[email protected]>

Reviewed-by Joshua Hahn <[email protected]>

Signed-off-by Usama Arif <[email protected]>

---

queue_folios_pmd() is called under pmd_trans_huge_lock(), whose pmd_is_huge() check returns true for any non-present, non-none PMD softleaf. Passing such a PMD to pmd_folio() treats the softleaf encoding as a hardware PFN and can return a bogus folio pointer.

Mirror queue_folios_pte_range(): handle non-present entries before looking up a folio. Keep migration entries counted as failures, but skip other non-present PMDs such as device-private entries.

Potential trigger: an HMM-based GPU driver migrates an anonymous THP folio to device memory via migrate_vma_pages(), leaving a device-private PMD. Userspace then calls mbind(), migrate_pages() or set_mempolicy_home_node() on that range.

Reported-by sashiko-bot <[email protected]>

Link https://sashiko.dev/#/patchset/20260703173903.3789516-1-usama.arif%40linux.dev?part=6

Fixes 368076f52ebe ("mm/huge_memory: add device-private THP support to PMD operations")

Cc <[email protected]>

Reviewed-by Joshua Hahn <[email protected]>

Signed-off-by Usama Arif <[email protected]>

---
 mm/mempolicy.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

mm/mempolicy.c +0 -0

index 914f81863db5..4785b55c02da 100644

mm/mempolicy.c +4 -3

-	if (unlikely(pmd_is_migration_entry(*pmd))) {
-		qp->nr_failed++;
+	if (unlikely(!pmd_present(pmdval))) {
+		if (pmd_is_migration_entry(pmdval))
+			qp->nr_failed++;
 		return;
 	}
-	folio = pmd_folio(*pmd);
+	folio = pmd_folio(pmdval);
 	if (is_huge_zero_folio(folio)) {
 		walk->action = ACTION_CONTINUE;
 		return;

Signature · 1 line

2.53.0-Meta

madvise_cold_or_pageout_pte_range() takes pmd_trans_huge_lock(), whose pmd_is_huge() check returns true for a device-private PMD. The subsequent !pmd_present() branch has a VM_BUG_ON() asserting migration is the only allowed non-present case; a device-private PMD trips it.

Allow device-private PMDs in that non-present assertion and continue to huge_unlock before calling pmd_folio(). This keeps the assertion for unexpected PMD softleafs while skipping device-private PMDs like other non-present PMDs in this path.

Potential trigger: an HMM-based GPU driver races with madvise(MADV_COLD)/MADV_PAGEOUT: pmd_trans_huge(*pmd) reads true, then migrate_vma_pages() flips the PMD to a device-private entry before the PMD lock is acquired.

Reported-by sashiko-bot <[email protected]>

Link https://sashiko.dev/#/patchset/20260703173903.3789516-1-usama.arif%40linux.dev?part=6

Fixes 368076f52ebe ("mm/huge_memory: add device-private THP support to PMD operations")

Cc <[email protected]>

Reviewed-by Joshua Hahn <[email protected]>

Reviewed-by Zi Yan <[email protected]>

Reviewed-by Balbir Singh <[email protected]>

Signed-off-by Usama Arif <[email protected]>

---
 mm/madvise.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

mm/madvise.c +0 -0

index 9292f60b19aa..1065b5a84ea7 100644

madvise_free_pte_range() checks pmd_trans_huge(*pmd) unlocked, then madvise_free_huge_pmd() takes pmd_trans_huge_lock(). pmd_is_huge() returns true for a device-private PMD, so orig_pmd can be device-private and enter the !pmd_present() branch.

Allow device-private PMDs in that non-present assertion and continue to out before calling pmd_folio(). This keeps the assertion for unexpected PMD softleafs while skipping device-private PMDs like other non-present PMDs in this path.

Potential trigger: an HMM-based GPU driver races with madvise(MADV_FREE):
migrate_vma_pages() flips the PMD to a device-private entry between the caller's pmd_trans_huge() check and the callee's pmd_trans_huge_lock().

Fixes 368076f52ebe ("mm/huge_memory: add device-private THP support to PMD operations")

Cc <[email protected]>

Reviewed-by Joshua Hahn <[email protected]>

Reviewed-by Zi Yan <[email protected]>

Reviewed-by Balbir Singh <[email protected]>

Signed-off-by Usama Arif <[email protected]>

---
 mm/huge_memory.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

mm/huge_memory.c +0 -0

index c0892cc533a9..ddbdc83b4cae 100644