Skip to contentSkip to conversation

mm: fixes for THP migration, swap-in and device-private smaps

PATCH In review Usama Arif 5 participants 14 messages Nov 16 – Nov 17

Subsystems (7)
lore
Share & tools
lore

Usama Arif

author
#0 lore raw

v2 cover mm: fixes for THP migration, swap-in and device-private smaps

This series contains fixes for existing THP migration, PTE swap-in and smaps paths. They were extracted from the work on PMD swap entries for anonymous THPs [1] and can be merged independently.

1. Avoid a folio reference leak when concurrent MADV_DONTNEED clears or splits a migration PMD before the device migration split helper acquires the PMD lock.

2. Prevent PTE swap-in batching from mapping a hardware-poisoned page of a large anon swap-cache folio.

3. Stop reading the accessed and dirty bits out of a non-present PMD in smaps, where they encode the softleaf entry rather than accessed/dirty state.

4. Account device-private PMDs using mapcounts in smaps, matching the PTE path. Shared mappings get proportional PSS, and exclusive mappings are counted as private.

This applies on top of mm-new (0bce4c3c7589).

v1 -> v2: https://lore.kernel.org/r/[email protected]/
- Patch 2: use the existing folio_contain_hwpoisoned_page()
rather than adding a helper that scans every page of the folio (Lorenzo, David). Only skip batching for anon folios, since falling back for a freshly read folio frees it with the poisoned page in it, and the retried fault can get the same block back.
- Collect the Reviewed-by and Acked-by tags from v1.

[1] PMD-level swap entries for anonymous THPs:
https://lore.kernel.org/r/[email protected]/

Usama Arif (4):
mm/migrate_device: avoid leaking folio references during THP splitting
mm: don't PTE-batch a swap-in of a folio with a hardware-poisoned page fs/proc/task_mmu: don't read young/dirty from a non-present PMD in
smaps
fs/proc/task_mmu: account device-private PMDs as present in smaps

 fs/proc/task_mmu.c  | 12 ++++++++----
 mm/memory.c         |  5 +++++
 mm/migrate_device.c |  7 +------
 3 files changed, 14 insertions(+), 10 deletions(-)
Signature · 1 line
2.53.1-Meta
author
#1 lore raw

PATCH v2 1/4 mm/migrate_device: avoid leaking folio references during THP splitting

migrate_vma_split_unmapped_folio() takes an extra folio reference because split_huge_pmd_address(..., true) drops one when it splits a PMD.

However, migrate_vma only holds mmap_lock for reading. A concurrent MADV_DONTNEED can clear the migration PMD, or split it into PTE migration entries, before the split helper takes the PMD lock. The helper then does nothing and leaves the extra reference behind. folio_split_unmapped() fails with -EAGAIN due to the unexpected reference, and the reference is never dropped, so the source THP is leaked.

If a THP is faulted in at the same address inside that window it is
worse: freeze = true replaces the new folio's mappings with migration entries that nothing will ever remove, and drops a reference the caller never took.

An existing migration PMD is expanded into PTE migration entries even with freeze = false. Use that mode and remove the extra folio_get(). The reference already held by migration keeps the source folio alive whether or not its PMD migration entry still exists.

Fixes 4265d67e405a ("mm/migrate_device: add THP splitting during migration")
---
 mm/migrate_device.c | 7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)
mm/migrate_device.c +1 -6
index 9e19fadfa65c6..3286bb8a4b73e 100644
@@ -920,12 +920,7 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,
 	unsigned long flags;
 	int ret = 0;
 
-	/*
-	 * take a reference, since split_huge_pmd_address() with freeze = true
-	 * drops a reference at the end.
-	 */
-	folio_get(folio);
-	split_huge_pmd_address(migrate->vma, addr, true);
+	split_huge_pmd_address(migrate->vma, addr, false);
 	ret = folio_split_unmapped(folio, 0);
 	if (ret)
 		return ret;
Signature · 1 line
2.53.1-Meta
maintainer Reviewed-by on v2 1/4
#2 lore raw

Re: [PATCH v2 1/4] mm/migrate_device: avoid leaking folio references during THP splitting

Quoted 37 lines · Usama Arif

migrate_vma_split_unmapped_folio() takes an extra folio reference because split_huge_pmd_address(..., true) drops one when it splits a PMD.

However, migrate_vma only holds mmap_lock for reading. A concurrent MADV_DONTNEED can clear the migration PMD, or split it into PTE migration entries, before the split helper takes the PMD lock. The helper then does nothing and leaves the extra reference behind. folio_split_unmapped() fails with -EAGAIN due to the unexpected reference, and the reference is never dropped, so the source THP is leaked.

If a THP is faulted in at the same address inside that window it is
worse: freeze = true replaces the new folio's mappings with migration entries that nothing will ever remove, and drops a reference the caller never took.

An existing migration PMD is expanded into PTE migration entries even with freeze = false. Use that mode and remove the extra folio_get(). The reference already held by migration keeps the source folio alive whether or not its PMD migration entry still exists.

Fixes 4265d67e405a ("mm/migrate_device: add THP splitting during migration")
---
 mm/migrate_device.c | 7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)
mm/migrate_device.c +0 -0
index 9e19fadfa65c6..3286bb8a4b73e 100644
@@ -920,12 +920,7 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,
 	unsigned long flags;
 	int ret = 0;
 

mm/migrate_device.c static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate, · line 926

Quoted 10 lines
mm/migrate_device.c +1 -6
-	/*
-	 * take a reference, since split_huge_pmd_address() with freeze = true
-	 * drops a reference at the end.
-	 */
-	folio_get(folio);
-	split_huge_pmd_address(migrate->vma, addr, true);
+	split_huge_pmd_address(migrate->vma, addr, false);
 	ret = folio_split_unmapped(folio, 0);
 	if (ret)
 		return ret;

Oh, that means we can remove the parameter from split_huge_pmd_address no? Best as a cleanup patch once this is in for-next-fixes.

Signature · 3 lines
Cheers,

David
#3 lore raw

Re: [PATCH v2 1/4] mm/migrate_device: avoid leaking folio references during THP splitting

Quoted 38 lines · David Hildenbrand (Arm)
Quoted 37 lines · Usama Arif

migrate_vma_split_unmapped_folio() takes an extra folio reference because split_huge_pmd_address(..., true) drops one when it splits a PMD.

However, migrate_vma only holds mmap_lock for reading. A concurrent MADV_DONTNEED can clear the migration PMD, or split it into PTE migration entries, before the split helper takes the PMD lock. The helper then does nothing and leaves the extra reference behind. folio_split_unmapped() fails with -EAGAIN due to the unexpected reference, and the reference is never dropped, so the source THP is leaked.

If a THP is faulted in at the same address inside that window it is
worse: freeze = true replaces the new folio's mappings with migration entries that nothing will ever remove, and drops a reference the caller never took.

An existing migration PMD is expanded into PTE migration entries even with freeze = false. Use that mode and remove the extra folio_get(). The reference already held by migration keeps the source folio alive whether or not its PMD migration entry still exists.

Fixes 4265d67e405a ("mm/migrate_device: add THP splitting during migration")
---
 mm/migrate_device.c | 7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)
mm/migrate_device.c +0 -0
index 9e19fadfa65c6..3286bb8a4b73e 100644
@@ -920,12 +920,7 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,
 	unsigned long flags;
 	int ret = 0;
 
Quoted 13 lines

mm/migrate_device.c static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate, · line 926

Quoted 10 lines
mm/migrate_device.c +1 -6
-	/*
-	 * take a reference, since split_huge_pmd_address() with freeze = true
-	 * drops a reference at the end.
-	 */
-	folio_get(folio);
-	split_huge_pmd_address(migrate->vma, addr, true);
+	split_huge_pmd_address(migrate->vma, addr, false);
 	ret = folio_split_unmapped(folio, 0);
 	if (ret)
 		return ret;

Oh, that means we can remove the parameter from split_huge_pmd_address no? Best as a cleanup patch once this is in for-next-fixes.

Yeah, that's would be great.

Quoted 1 line

The patch looks good to me too:

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
reviewer Reviewed-by on v2 1/4
#4 lore raw

Re: [PATCH v2 1/4] mm/migrate_device: avoid leaking folio references during THP splitting

In reply to Usama Arif
Quoted 20 lines · Usama Arif

migrate_vma_split_unmapped_folio() takes an extra folio reference because split_huge_pmd_address(..., true) drops one when it splits a PMD.

However, migrate_vma only holds mmap_lock for reading. A concurrent MADV_DONTNEED can clear the migration PMD, or split it into PTE migration entries, before the split helper takes the PMD lock. The helper then does nothing and leaves the extra reference behind. folio_split_unmapped() fails with -EAGAIN due to the unexpected reference, and the reference is never dropped, so the source THP is leaked.

If a THP is faulted in at the same address inside that window it is
worse: freeze = true replaces the new folio's mappings with migration entries that nothing will ever remove, and drops a reference the caller never took.

An existing migration PMD is expanded into PTE migration entries even with freeze = false. Use that mode and remove the extra folio_get(). The reference already held by migration keeps the source folio alive whether or not its PMD migration entry still exists.

Quoted 9 lines
Fixes 4265d67e405a ("mm/migrate_device: add THP splitting during migration")
---
 mm/migrate_device.c | 7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)

Thank you for fixing it.

Best Regards,
Yan, Zi

author
#5 lore raw

PATCH v2 2/4 mm: don't PTE-batch a swap-in of a folio with a hardware-poisoned page

In reply to Usama Arif

do_swap_page() checks PageHWPoison() on the faulting page, but the large-folio batching path then maps the rest of the folio without looking at its other pages. A poisoned page that is not the one being faulted gets mapped anyway, and the fault that would have reported it never happens.

Skip batching for such an anon folio and let each page take its own fault, where the existing check catches it. do_set_pmd() already backs off the same way.

Leave a freshly read folio alone: it is not anon yet and can only be mapped as a whole, and falling back would free it with the poisoned page still in it, which the retried fault can then get back from the page allocator.

Fixes 508758960b8d ("mm: swap: entirely map large folios found in swapcache")
---
 mm/memory.c | 5 +++++
 1 file changed, 5 insertions(+)
mm/memory.c +5 -0
index d477a4f986bc8..51e72fb40655f 100644
@@ -5158,6 +5158,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 		    swap_pte_batch(folio_ptep, nr, folio_pte) != nr)
 			goto check_folio;
 
+		/* A poisoned page in the folio must take its own fault. */
+		if (folio_test_anon(folio) &&
+		    folio_contain_hwpoisoned_page(folio))
+			goto check_folio;
+
 		page_idx = idx;
 		address = folio_start;
 		ptep = folio_ptep;
Signature · 1 line
2.53.1-Meta
reviewer Reviewed-by on v2 2/4
#6 lore raw

Re: [PATCH v2 2/4] mm: don't PTE-batch a swap-in of a folio with a hardware-poisoned page

Quoted 6 lines · Usama Arif

do_swap_page() checks PageHWPoison() on the faulting page, but the large-folio batching path then maps the rest of the folio without looking at its other pages. A poisoned page that is not the one being faulted gets mapped anyway, and the fault that would have reported it never happens.

Quoted 8 lines

Skip batching for such an anon folio and let each page take its own fault, where the existing check catches it. do_set_pmd() already backs off the same way.

Leave a freshly read folio alone: it is not anon yet and can only be mapped as a whole, and falling back would free it with the poisoned page still in it, which the retried fault can then get back from the page allocator.

LGTM!

Quoted 26 lines
Fixes 508758960b8d ("mm: swap: entirely map large folios found in swapcache")
---
 mm/memory.c | 5 +++++
 1 file changed, 5 insertions(+)
mm/memory.c +5 -0
index d477a4f986bc8..51e72fb40655f 100644
@@ -5158,6 +5158,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 		    swap_pte_batch(folio_ptep, nr, folio_pte) != nr)
 			goto check_folio;
 
+		/* A poisoned page in the folio must take its own fault. */
+		if (folio_test_anon(folio) &&
+		    folio_contain_hwpoisoned_page(folio))
+			goto check_folio;
+
 		page_idx = idx;
 		address = folio_start;
 		ptep = folio_ptep;
Signature · 1 line
2.53.1-Meta
#7 lore raw

Re: [PATCH v2 2/4] mm: don't PTE-batch a swap-in of a folio with a hardware-poisoned page

In reply to Usama Arif
Quoted 28 lines · Usama Arif

do_swap_page() checks PageHWPoison() on the faulting page, but the large-folio batching path then maps the rest of the folio without looking at its other pages. A poisoned page that is not the one being faulted gets mapped anyway, and the fault that would have reported it never happens.

Skip batching for such an anon folio and let each page take its own fault, where the existing check catches it. do_set_pmd() already backs off the same way.

Leave a freshly read folio alone: it is not anon yet and can only be mapped as a whole, and falling back would free it with the poisoned page still in it, which the retried fault can then get back from the page allocator.

Fixes 508758960b8d ("mm: swap: entirely map large folios found in swapcache")
---
 mm/memory.c | 5 +++++
 1 file changed, 5 insertions(+)
mm/memory.c +0 -0
index d477a4f986bc8..51e72fb40655f 100644
@@ -5158,6 +5158,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 		    swap_pte_batch(folio_ptep, nr, folio_pte) != nr)

mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 5168

Quoted 10 lines
mm/memory.c +5 -0
 			goto check_folio;
 
+		/* A poisoned page in the folio must take its own fault. */
+		if (folio_test_anon(folio) &&
+		    folio_contain_hwpoisoned_page(folio))
+			goto check_folio;
+
 		page_idx = idx;
 		address = folio_start;
 		ptep = folio_ptep;

Why not check before swap_pte_batch() ?

Signature · 3 lines
Cheers,

David
reviewer Reviewed-by on v2 2/4
#8 lore raw

Re: [PATCH v2 2/4] mm: don't PTE-batch a swap-in of a folio with a hardware-poisoned page

In reply to Usama Arif
Quoted 15 lines · Usama Arif

do_swap_page() checks PageHWPoison() on the faulting page, but the large-folio batching path then maps the rest of the folio without looking at its other pages. A poisoned page that is not the one being faulted gets mapped anyway, and the fault that would have reported it never happens.

Skip batching for such an anon folio and let each page take its own fault, where the existing check catches it. do_set_pmd() already backs off the same way.

Leave a freshly read folio alone: it is not anon yet and can only be mapped as a whole, and falling back would free it with the poisoned page still in it, which the retried fault can then get back from the page allocator.

Quoted 7 lines
Fixes 508758960b8d ("mm: swap: entirely map large folios found in swapcache")
---
 mm/memory.c | 5 +++++
 1 file changed, 5 insertions(+)

Like David suggested, it probably can be placed before folio_pte_range(). With that change, feel free to add

Thanks.

Best Regards,
Yan, Zi

Reviewed-by on v2 2/4
#9 lore raw

Re: [PATCH v2 2/4] mm: don't PTE-batch a swap-in of a folio with a hardware-poisoned page

In reply to Usama Arif
Quoted 10 lines · Usama Arif

do_swap_page() checks PageHWPoison() on the faulting page, but the large-folio batching path then maps the rest of the folio without looking at its other pages. A poisoned page that is not the one being faulted gets mapped anyway, and the fault that would have reported it never happens.

Skip batching for such an anon folio and let each page take its own fault, where the existing check catches it. do_set_pmd() already backs off the same way.

Quoted 8 lines

Leave a freshly read folio alone: it is not anon yet and can only be mapped as a whole, and falling back would free it with the poisoned page still in it, which the retried fault can then get back from the page allocator.

Fixes 508758960b8d ("mm: swap: entirely map large folios found in swapcache")

With the check moved up as David and Zi suggested:

Do we have the same problem on the file side?

If memory_failure() cannot split a large file folio, it stays in the page cache with the poisoned page in it.

Consider a write fault through MAP_SHARED on a good page of that folio:

do_shared_fault()

    __do_fault()
      PageHWPoison(vmf->page)
        // checks the faulting page only
    finish_fault()
      do_set_pmd()
        folio_test_has_hwpoisoned(folio)
          // backs off with VM_FAULT_FALLBACK
      set_pte_range(vmf, folio, &folio->page, folio_nr_pages(folio), addr)
        // maps the poisoned page as well

Possible fix is below. I can post a proper patch if the analysis looks correct.

mm/memory.c +4 -0
index fb76d8922e37..25b53847943f 100644
@@ -5893,6 +5893,10 @@ vm_fault_t finish_fault(struct vm_fault *vmf)
 			return ret;
 	}
 
+	/* A poisoned page must take its own fault */
+	if (folio_contain_hwpoisoned_page(folio))
+		needs_fallback = true;
+
 	if (!needs_fallback && vma->vm_file) {
 		struct address_space *mapping = vma->vm_file->f_mapping;
 		pgoff_t file_end;
Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
author
#10 lore raw

PATCH v2 3/4 fs/proc/task_mmu: don't read young/dirty from a non-present PMD in smaps

In reply to Usama Arif

smaps_pmd_entry() passes pmd_young(*pmd) and pmd_dirty(*pmd) to smaps_account() for every PMD, including softleaf entries (migration and device-private), where those bits encode the entry rather than accessed/dirty state. smaps_pte_entry() only reads them when the entry is present.

On x86_64 the entry's offset is stored inverted across bits 9-58, which overlap _PAGE_SAVED_DIRTY, so pmd_dirty() returns garbage: a clean 2MB device-private mapping is accounted as 2048 kB dirty, and lazyfree accounting is suppressed. pgtable_64.h also warns that the hardware can set the accessed and dirty bits on non-present entries by mistake.

Only read the bits when the PMD is present.

Fixes c94b6923fa0a ("/proc/PID/smaps: Add PMD migration entry parsing")
---
 fs/proc/task_mmu.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
fs/proc/task_mmu.c +4 -3
index 164b4efc5e093..ad9fdcdaa2c40 100644
@@ -1029,13 +1029,15 @@ static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr,
 	struct vm_area_struct *vma = walk->vma;
 	bool locked = !!(vma->vm_flags & VM_LOCKED);
 	struct page *page = NULL;
-	bool present = false;
+	bool present = false, young = false, dirty = false;
 	struct folio *folio;
 
 	if (pmd_none(*pmd))
 		return;
 	if (pmd_present(*pmd)) {
 		page = vm_normal_page_pmd(vma, addr, *pmd);
+		young = pmd_young(*pmd);
+		dirty = pmd_dirty(*pmd);
 		present = true;
 	} else if (unlikely(thp_migration_supported())) {
 		const softleaf_t entry = softleaf_from_pmd(*pmd);
@@ -1055,8 +1057,7 @@ static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr,
 	else
 		mss->file_thp += HPAGE_PMD_SIZE;
 
-	smaps_account(mss, page, true, pmd_young(*pmd), pmd_dirty(*pmd),
-		      locked, present);
+	smaps_account(mss, page, true, young, dirty, locked, present);
 }
 #else
 static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr,
Signature · 1 line
2.53.1-Meta
#11 lore raw

Re: [PATCH v2 3/4] fs/proc/task_mmu: don't read young/dirty from a non-present PMD in smaps

Quoted 11 lines · Usama Arif
fs/proc/task_mmu.c +1 -1
index 164b4efc5e093..ad9fdcdaa2c40 100644
@@ -1029,13 +1029,15 @@ static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr,
 	struct vm_area_struct *vma = walk->vma;
 	bool locked = !!(vma->vm_flags & VM_LOCKED);
 	struct page *page = NULL;
-	bool present = false;
+	bool present = false, young = false, dirty = false;
 	struct folio *folio;

fs/proc/task_mmu.c static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr, · line 1043

Quoted 10 lines
fs/proc/task_mmu.c +2 -0
 
 	if (pmd_none(*pmd))
 		return;
 	if (pmd_present(*pmd)) {
 		page = vm_normal_page_pmd(vma, addr, *pmd);
+		young = pmd_young(*pmd);
+		dirty = pmd_dirty(*pmd);
 		present = true;
 	} else if (unlikely(thp_migration_supported())) {
 		const softleaf_t entry = softleaf_from_pmd(*pmd);

smaps_pmd_entry() dereferences pmd six times :/

Could you read it once with pmdp_get() in a separate patch before this one, the way smaps_pte_entry() does with ptep_get()?

This patch would then only move pmd_young() and pmd_dirty() under the present branch.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
author
#12 lore raw

PATCH v2 4/4 fs/proc/task_mmu: account device-private PMDs as present in smaps

In reply to Usama Arif

Device-private PMDs represent mapped pages, but smaps_pmd_entry() passes present=false to smaps_account(). That treats them like migration
entries: mapcounts are ignored, PSS is charged in full, and exclusive mappings are accounted as shared.

Match smaps_pte_entry() and mark device-private PMDs as present so smaps_account() uses mapcounts for PSS and private/shared accounting.

Fixes a30b48bf1b24 ("mm/migrate_device: implement THP migration of zone device pages")
---
 fs/proc/task_mmu.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
fs/proc/task_mmu.c +4 -1
index ad9fdcdaa2c40..6345bb143f200 100644
@@ -1042,8 +1042,11 @@ static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr,
 	} else if (unlikely(thp_migration_supported())) {
 		const softleaf_t entry = softleaf_from_pmd(*pmd);
 
-		if (softleaf_has_pfn(entry))
+		if (softleaf_has_pfn(entry)) {
+			if (softleaf_is_device_private(entry))
+				present = true;
 			page = softleaf_to_page(entry);
+		}
 	}
 	if (IS_ERR_OR_NULL(page))
 		return;
Signature · 1 line
2.53.1-Meta
Reviewed-by on v2 4/4
#13 lore raw

Re: [PATCH v2 4/4] fs/proc/task_mmu: account device-private PMDs as present in smaps

Quoted 8 lines · Usama Arif

Device-private PMDs represent mapped pages, but smaps_pmd_entry() passes present=false to smaps_account(). That treats them like migration
entries: mapcounts are ignored, PSS is charged in full, and exclusive mappings are accounted as shared.

Match smaps_pte_entry() and mark device-private PMDs as present so smaps_account() uses mapcounts for PSS and private/shared accounting.

Quoted 6 lines
Fixes a30b48bf1b24 ("mm/migrate_device: implement THP migration of zone device pages")
Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov