mm: fixes for THP migration, swap-in and device-private smaps
Subsystems (7)
v2 cover mm: fixes for THP migration, swap-in and device-private smaps
This series contains fixes for existing THP migration, PTE swap-in and smaps paths. They were extracted from the work on PMD swap entries for anonymous THPs [1] and can be merged independently.
1. Avoid a folio reference leak when concurrent MADV_DONTNEED clears or splits a migration PMD before the device migration split helper acquires the PMD lock.
2. Prevent PTE swap-in batching from mapping a hardware-poisoned page of a large anon swap-cache folio.
3. Stop reading the accessed and dirty bits out of a non-present PMD in smaps, where they encode the softleaf entry rather than accessed/dirty state.
4. Account device-private PMDs using mapcounts in smaps, matching the PTE path. Shared mappings get proportional PSS, and exclusive mappings are counted as private.
This applies on top of mm-new (0bce4c3c7589).
v1 -> v2: https://lore.kernel.org/r/[email protected]/
- Patch 2: use the existing folio_contain_hwpoisoned_page()
rather than adding a helper that scans every page of the folio (Lorenzo, David). Only skip batching for anon folios, since falling back for a freshly read folio frees it with the poisoned page in it, and the retried fault can get the same block back.
- Collect the Reviewed-by and Acked-by tags from v1.
[1] PMD-level swap entries for anonymous THPs:
https://lore.kernel.org/r/[email protected]/
Usama Arif (4):
mm/migrate_device: avoid leaking folio references during THP splitting
mm: don't PTE-batch a swap-in of a folio with a hardware-poisoned page fs/proc/task_mmu: don't read young/dirty from a non-present PMD in
smaps
fs/proc/task_mmu: account device-private PMDs as present in smaps
fs/proc/task_mmu.c | 12 ++++++++---- mm/memory.c | 5 +++++ mm/migrate_device.c | 7 +------ 3 files changed, 14 insertions(+), 10 deletions(-)
Signature · 1 line
2.53.1-Meta
PATCH v2 1/4 mm/migrate_device: avoid leaking folio references during THP splitting
migrate_vma_split_unmapped_folio() takes an extra folio reference because split_huge_pmd_address(..., true) drops one when it splits a PMD.
However, migrate_vma only holds mmap_lock for reading. A concurrent MADV_DONTNEED can clear the migration PMD, or split it into PTE migration entries, before the split helper takes the PMD lock. The helper then does nothing and leaves the extra reference behind. folio_split_unmapped() fails with -EAGAIN due to the unexpected reference, and the reference is never dropped, so the source THP is leaked.
If a THP is faulted in at the same address inside that window it is
worse: freeze = true replaces the new folio's mappings with migration entries that nothing will ever remove, and drops a reference the caller never took.
An existing migration PMD is expanded into PTE migration entries even with freeze = false. Use that mode and remove the extra folio_get(). The reference already held by migration keeps the source folio alive whether or not its PMD migration entry still exists.
--- mm/migrate_device.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-)
mm/migrate_device.c +1 -6
@@ -920,12 +920,7 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,unsigned long flags;int ret =0 ; -/* -* take a reference, since split_huge_pmd_address() with freeze = true -* drops a reference at the end. -*/ -folio_get (folio); -split_huge_pmd_address (migrate->vma, addr, true); +split_huge_pmd_address (migrate->vma, addr, false); ret =folio_split_unmapped (folio,0 );if (ret)return ret;
Signature · 1 line
2.53.1-Meta
Re: [PATCH v2 1/4] mm/migrate_device: avoid leaking folio references during THP splitting
Quoted 37 lines · Usama Arif
migrate_vma_split_unmapped_folio()takes an extra folio reference becausesplit_huge_pmd_address(..., true)drops one when it splits a PMD.However,
migrate_vmaonly holdsmmap_lockfor reading. A concurrentMADV_DONTNEEDcan clear the migration PMD, or split it into PTE migration entries, before the split helper takes the PMD lock. The helper then does nothing and leaves the extra reference behind.folio_split_unmapped()fails with-EAGAINdue to the unexpected reference, and the reference is never dropped, so the source THP is leaked.If a THP is faulted in at the same address inside that window it is
worse: freeze = true replaces the new folio's mappings with migration entries that nothing will ever remove, and drops a reference the caller never took.An existing migration PMD is expanded into PTE migration entries even with freeze = false. Use that mode and remove the extra
folio_get(). The reference already held by migration keeps the source folio alive whether or not its PMD migration entry still exists.Reported-by Zi Yan <[email protected]>Fixes 4265d67e405a ("mm/migrate_device: add THP splitting during migration")Signed-off-by Usama Arif <[email protected]>--- mm/migrate_device.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-)mm/migrate_device.c +0
-0@@ -920,12 +920,7 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,unsigned long flags;int ret =0 ;
mm/migrate_device.c static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate, · line 926
Quoted 10 lines
mm/migrate_device.c +1
-6-/* -* take a reference, since split_huge_pmd_address() with freeze = true -* drops a reference at the end. -*/ -folio_get (folio); -split_huge_pmd_address (migrate->vma, addr, true); +split_huge_pmd_address (migrate->vma, addr, false); ret =folio_split_unmapped (folio,0 );if (ret)return ret;
Oh, that means we can remove the parameter from split_huge_pmd_address no? Best as a cleanup patch once this is in for-next-fixes.
Signature · 3 lines
Cheers, David
Kiryl Shutsemau (Meta)
Reviewed-byRe: [PATCH v2 1/4] mm/migrate_device: avoid leaking folio references during THP splitting
Quoted 38 lines · David Hildenbrand (Arm)
Quoted 37 lines · Usama Arif
migrate_vma_split_unmapped_folio()takes an extra folio reference becausesplit_huge_pmd_address(..., true)drops one when it splits a PMD.However,
migrate_vmaonly holdsmmap_lockfor reading. A concurrentMADV_DONTNEEDcan clear the migration PMD, or split it into PTE migration entries, before the split helper takes the PMD lock. The helper then does nothing and leaves the extra reference behind.folio_split_unmapped()fails with-EAGAINdue to the unexpected reference, and the reference is never dropped, so the source THP is leaked.If a THP is faulted in at the same address inside that window it is
worse: freeze = true replaces the new folio's mappings with migration entries that nothing will ever remove, and drops a reference the caller never took.An existing migration PMD is expanded into PTE migration entries even with freeze = false. Use that mode and remove the extra
folio_get(). The reference already held by migration keeps the source folio alive whether or not its PMD migration entry still exists.Reported-by Zi Yan <[email protected]>Fixes 4265d67e405a ("mm/migrate_device: add THP splitting during migration")Signed-off-by Usama Arif <[email protected]>--- mm/migrate_device.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-)mm/migrate_device.c +0
-0@@ -920,12 +920,7 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,unsigned long flags;int ret =0 ;
Quoted 13 lines
mm/migrate_device.c static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate, · line 926
Quoted 10 lines
mm/migrate_device.c +1
-6-/* -* take a reference, since split_huge_pmd_address() with freeze = true -* drops a reference at the end. -*/ -folio_get (folio); -split_huge_pmd_address (migrate->vma, addr, true); +split_huge_pmd_address (migrate->vma, addr, false); ret =folio_split_unmapped (folio,0 );if (ret)return ret;Oh, that means we can remove the parameter from
split_huge_pmd_addressno? Best as a cleanup patch once this is in for-next-fixes.
Yeah, that's would be great.
Quoted 1 line
Reviewed-by David Hildenbrand (Arm) <[email protected]>
The patch looks good to me too:
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Re: [PATCH v2 1/4] mm/migrate_device: avoid leaking folio references during THP splitting
In reply to Usama ArifQuoted 20 lines · Usama Arif
migrate_vma_split_unmapped_folio()takes an extra folio reference becausesplit_huge_pmd_address(..., true)drops one when it splits a PMD.However,
migrate_vmaonly holdsmmap_lockfor reading. A concurrentMADV_DONTNEEDcan clear the migration PMD, or split it into PTE migration entries, before the split helper takes the PMD lock. The helper then does nothing and leaves the extra reference behind.folio_split_unmapped()fails with-EAGAINdue to the unexpected reference, and the reference is never dropped, so the source THP is leaked.If a THP is faulted in at the same address inside that window it is
worse: freeze = true replaces the new folio's mappings with migration entries that nothing will ever remove, and drops a reference the caller never took.An existing migration PMD is expanded into PTE migration entries even with freeze = false. Use that mode and remove the extra
folio_get(). The reference already held by migration keeps the source folio alive whether or not its PMD migration entry still exists.
Quoted 9 lines
Reported-by Zi Yan <[email protected]>Fixes 4265d67e405a ("mm/migrate_device: add THP splitting during migration")Signed-off-by Usama Arif <[email protected]>--- mm/migrate_device.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-)
Thank you for fixing it.
Best Regards,
Yan, Zi
PATCH v2 2/4 mm: don't PTE-batch a swap-in of a folio with a hardware-poisoned page
In reply to Usama Arifdo_swap_page() checks PageHWPoison() on the faulting page, but the large-folio batching path then maps the rest of the folio without looking at its other pages. A poisoned page that is not the one being faulted gets mapped anyway, and the fault that would have reported it never happens.
Skip batching for such an anon folio and let each page take its own fault, where the existing check catches it. do_set_pmd() already backs off the same way.
Leave a freshly read folio alone: it is not anon yet and can only be mapped as a whole, and falling back would free it with the poisoned page still in it, which the retried fault can then get back from the page allocator.
--- mm/memory.c | 5 +++++ 1 file changed, 5 insertions(+)
mm/memory.c +5 -0
@@ -5158,6 +5158,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) swap_pte_batch(folio_ptep, nr, folio_pte) != nr) goto check_folio; +/* A poisoned page in the folio must take its own fault. */ +if (folio_test_anon (folio) && +folio_contain_hwpoisoned_page (folio)) + goto check_folio; + page_idx = idx; address = folio_start; ptep = folio_ptep;
Signature · 1 line
2.53.1-Meta
Re: [PATCH v2 2/4] mm: don't PTE-batch a swap-in of a folio with a hardware-poisoned page
Quoted 6 lines · Usama Arif
do_swap_page()checksPageHWPoison()on the faulting page, but the large-folio batching path then maps the rest of the folio without looking at its other pages. A poisoned page that is not the one being faulted gets mapped anyway, and the fault that would have reported it never happens.
Quoted 8 lines
Skip batching for such an anon folio and let each page take its own fault, where the existing check catches it.
do_set_pmd()already backs off the same way.Leave a freshly read folio alone: it is not anon yet and can only be mapped as a whole, and falling back would free it with the poisoned page still in it, which the retried fault can then get back from the page allocator.
LGTM!
Quoted 26 lines
Fixes 508758960b8d ("mm: swap: entirely map large folios found in swapcache")Signed-off-by Usama Arif <[email protected]>--- mm/memory.c | 5 +++++ 1 file changed, 5 insertions(+)mm/memory.c +5
-0@@ -5158,6 +5158,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) swap_pte_batch(folio_ptep, nr, folio_pte) != nr) goto check_folio; +/* A poisoned page in the folio must take its own fault. */ +if (folio_test_anon (folio) && +folio_contain_hwpoisoned_page (folio)) + goto check_folio; + page_idx = idx; address = folio_start; ptep = folio_ptep;Signature · 1 line
2.53.1-Meta
David Hildenbrand (Arm)
maintainerRe: [PATCH v2 2/4] mm: don't PTE-batch a swap-in of a folio with a hardware-poisoned page
In reply to Usama ArifQuoted 28 lines · Usama Arif
do_swap_page()checksPageHWPoison()on the faulting page, but the large-folio batching path then maps the rest of the folio without looking at its other pages. A poisoned page that is not the one being faulted gets mapped anyway, and the fault that would have reported it never happens.Skip batching for such an anon folio and let each page take its own fault, where the existing check catches it.
do_set_pmd()already backs off the same way.Leave a freshly read folio alone: it is not anon yet and can only be mapped as a whole, and falling back would free it with the poisoned page still in it, which the retried fault can then get back from the page allocator.
Fixes 508758960b8d ("mm: swap: entirely map large folios found in swapcache")Signed-off-by Usama Arif <[email protected]>--- mm/memory.c | 5 +++++ 1 file changed, 5 insertions(+)mm/memory.c +0
-0@@ -5158,6 +5158,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) swap_pte_batch(folio_ptep, nr, folio_pte) != nr)
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 5168
Quoted 10 lines
mm/memory.c +5
-0goto check_folio; +/* A poisoned page in the folio must take its own fault. */ +if (folio_test_anon (folio) && +folio_contain_hwpoisoned_page (folio)) + goto check_folio; + page_idx = idx; address = folio_start; ptep = folio_ptep;
Why not check before swap_pte_batch() ?
Signature · 3 lines
Cheers, David
Re: [PATCH v2 2/4] mm: don't PTE-batch a swap-in of a folio with a hardware-poisoned page
In reply to Usama ArifQuoted 15 lines · Usama Arif
do_swap_page()checksPageHWPoison()on the faulting page, but the large-folio batching path then maps the rest of the folio without looking at its other pages. A poisoned page that is not the one being faulted gets mapped anyway, and the fault that would have reported it never happens.Skip batching for such an anon folio and let each page take its own fault, where the existing check catches it.
do_set_pmd()already backs off the same way.Leave a freshly read folio alone: it is not anon yet and can only be mapped as a whole, and falling back would free it with the poisoned page still in it, which the retried fault can then get back from the page allocator.
Quoted 7 lines
Fixes 508758960b8d ("mm: swap: entirely map large folios found in swapcache")Signed-off-by Usama Arif <[email protected]>--- mm/memory.c | 5 +++++ 1 file changed, 5 insertions(+)
Like David suggested, it probably can be placed before folio_pte_range(). With that change, feel free to add
Thanks.
Best Regards,
Yan, Zi
Re: [PATCH v2 2/4] mm: don't PTE-batch a swap-in of a folio with a hardware-poisoned page
In reply to Usama ArifQuoted 10 lines · Usama Arif
do_swap_page()checksPageHWPoison()on the faulting page, but the large-folio batching path then maps the rest of the folio without looking at its other pages. A poisoned page that is not the one being faulted gets mapped anyway, and the fault that would have reported it never happens.Skip batching for such an anon folio and let each page take its own fault, where the existing check catches it.
do_set_pmd()already backs off the same way.
Quoted 8 lines
Leave a freshly read folio alone: it is not anon yet and can only be mapped as a whole, and falling back would free it with the poisoned page still in it, which the retried fault can then get back from the page allocator.
Fixes 508758960b8d ("mm: swap: entirely map large folios found in swapcache")Signed-off-by Usama Arif <[email protected]>
With the check moved up as David and Zi suggested:
Do we have the same problem on the file side?
If memory_failure() cannot split a large file folio, it stays in the page cache with the poisoned page in it.
Consider a write fault through MAP_SHARED on a good page of that folio:
do_shared_fault()
__do_fault()
PageHWPoison(vmf->page)
// checks the faulting page only
finish_fault()
do_set_pmd()
folio_test_has_hwpoisoned(folio)
// backs off with VM_FAULT_FALLBACK
set_pte_range(vmf, folio, &folio->page, folio_nr_pages(folio), addr)
// maps the poisoned page as wellPossible fix is below. I can post a proper patch if the analysis looks correct.
mm/memory.c +4 -0
@@ -5893,6 +5893,10 @@ vm_fault_t finish_fault(struct vm_fault *vmf)return ret; } +/* A poisoned page must take its own fault */ +if (folio_contain_hwpoisoned_page (folio)) + needs_fallback = true; +if (!needs_fallback && vma->vm_file) {struct address_space *mapping = vma->vm_file->f_mapping;pgoff_t file_end;
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
PATCH v2 3/4 fs/proc/task_mmu: don't read young/dirty from a non-present PMD in smaps
In reply to Usama Arifsmaps_pmd_entry() passes pmd_young(*pmd) and pmd_dirty(*pmd) to smaps_account() for every PMD, including softleaf entries (migration and device-private), where those bits encode the entry rather than accessed/dirty state. smaps_pte_entry() only reads them when the entry is present.
On x86_64 the entry's offset is stored inverted across bits 9-58, which overlap _PAGE_SAVED_DIRTY, so pmd_dirty() returns garbage: a clean 2MB device-private mapping is accounted as 2048 kB dirty, and lazyfree accounting is suppressed. pgtable_64.h also warns that the hardware can set the accessed and dirty bits on non-present entries by mistake.
Only read the bits when the PMD is present.
--- fs/proc/task_mmu.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
fs/proc/task_mmu.c +4 -3
@@ -1029,13 +1029,15 @@ static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr,struct vm_area_struct *vma = walk->vma;bool locked = !!(vma->vm_flags &VM_LOCKED );struct page *page =NULL ; -bool present = false; +bool present = false, young = false, dirty = false;struct folio *folio;if (pmd_none (*pmd))return ;if (pmd_present (*pmd)) { page =vm_normal_page_pmd (vma, addr, *pmd); + young =pmd_young (*pmd); + dirty =pmd_dirty (*pmd); present = true; }else if (unlikely (thp_migration_supported ())) {const softleaf_t entry =softleaf_from_pmd (*pmd);
@@ -1055,8 +1057,7 @@ static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr, else mss->file_thp += HPAGE_PMD_SIZE; -smaps_account (mss, page, true,pmd_young (*pmd),pmd_dirty (*pmd), - locked, present); +smaps_account (mss, page, true, young, dirty, locked, present); }#else static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr,
Signature · 1 line
2.53.1-Meta
Re: [PATCH v2 3/4] fs/proc/task_mmu: don't read young/dirty from a non-present PMD in smaps
Quoted 11 lines · Usama Arif
fs/proc/task_mmu.c +1
-1@@ -1029,13 +1029,15 @@ static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr,struct vm_area_struct *vma = walk->vma;bool locked = !!(vma->vm_flags &VM_LOCKED );struct page *page =NULL ; -bool present = false; +bool present = false, young = false, dirty = false;struct folio *folio;
fs/proc/task_mmu.c static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr, · line 1043
Quoted 10 lines
fs/proc/task_mmu.c +2
-0if (pmd_none (*pmd))return ;if (pmd_present (*pmd)) { page =vm_normal_page_pmd (vma, addr, *pmd); + young =pmd_young (*pmd); + dirty =pmd_dirty (*pmd); present = true; }else if (unlikely (thp_migration_supported ())) {const softleaf_t entry =softleaf_from_pmd (*pmd);
smaps_pmd_entry() dereferences pmd six times :/
Could you read it once with pmdp_get() in a separate patch before this one, the way smaps_pte_entry() does with ptep_get()?
This patch would then only move pmd_young() and pmd_dirty() under the present branch.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
PATCH v2 4/4 fs/proc/task_mmu: account device-private PMDs as present in smaps
In reply to Usama ArifDevice-private PMDs represent mapped pages, but smaps_pmd_entry() passes present=false to smaps_account(). That treats them like migration
entries: mapcounts are ignored, PSS is charged in full, and exclusive mappings are accounted as shared.
Match smaps_pte_entry() and mark device-private PMDs as present so smaps_account() uses mapcounts for PSS and private/shared accounting.
--- fs/proc/task_mmu.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
fs/proc/task_mmu.c +4 -1
@@ -1042,8 +1042,11 @@ static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr, }else if (unlikely (thp_migration_supported ())) {const softleaf_t entry =softleaf_from_pmd (*pmd); -if (softleaf_has_pfn (entry)) +if (softleaf_has_pfn (entry)) { +if (softleaf_is_device_private (entry)) + present = true; page =softleaf_to_page (entry); + } }if (IS_ERR_OR_NULL (page))return ;
Signature · 1 line
2.53.1-Meta
Re: [PATCH v2 4/4] fs/proc/task_mmu: account device-private PMDs as present in smaps
Quoted 8 lines · Usama Arif
Device-private PMDs represent mapped pages, but
smaps_pmd_entry()passes present=false tosmaps_account(). That treats them like migration
entries: mapcounts are ignored, PSS is charged in full, and exclusive mappings are accounted as shared.Match
smaps_pte_entry()and mark device-private PMDs as present sosmaps_account()uses mapcounts for PSS and private/shared accounting.
Quoted 6 lines
Fixes a30b48bf1b24 ("mm/migrate_device: implement THP migration of zone device pages")Signed-off-by Usama Arif <[email protected]>Reviewed-by Zi Yan <[email protected]>
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov