mm/migrate_device: fix folio refcount leak on folio_split_unmapped failure
Subsystems (3)
PATCH 1/1 mm/migrate_device: fix folio refcount leak on folio_split_unmapped failure
From: Usama Arif <[email protected]>
migrate_vma_split_unmapped_folio() takes an extra reference via folio_get() before calling folio_split_unmapped(). On success, the split consumes this reference: __folio_freeze_and_split_unmapped() expects the +1 in its folio_ref_freeze() check, and distributes it across the resulting sub-folios via folio_ref_unfreeze(...+1), which are later balanced by folio_put() calls in __migrate_device_finalize().
If folio_split_unmapped() fails (e.g., unexpected pinning returns -EAGAIN), the function returns without calling folio_put(). The extra reference is never released.
Add the missing folio_put() on the error path.
--- mm/migrate_device.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
mm/migrate_device.c +3 -1
@@ -917,8 +917,10 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,folio_get (folio);split_huge_pmd_address (migrate->vma, addr, true); ret =folio_split_unmapped (folio,0 ); -if (ret) +if (ret) { +folio_put (folio);return ret; + } migrate->src[idx] &= ~MIGRATE_PFN_COMPOUND ; flags = migrate->src[idx] & ((1UL <<MIGRATE_PFN_SHIFT ) -1 ); pfn = migrate->src[idx] >>MIGRATE_PFN_SHIFT ;
Signature · 1 line
2.47.3
Kiryl Shutsemau
reviewerQuoted 8 lines · Usama Arif
From: Usama Arif <[email protected]>
migrate_vma_split_unmapped_folio()takes an extra reference viafolio_get()before callingfolio_split_unmapped(). On success, the split consumes this reference:__folio_freeze_and_split_unmapped()expects the +1 in itsfolio_ref_freeze()check, and distributes it across the resulting sub-folios viafolio_ref_unfreeze(...+1), which are later balanced byfolio_put()calls in__migrate_device_finalize().
Without this explanation folio_get() looks very random. And I still can't say I understand reference management for the folios here.
Who takes reference for the folio if it !THP that gets return in the _finalize()?
Can we get reference for THP and !THP at the same spot?
I think we should avoid spacial-casing THP where possible.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Zi Yan
reviewerQuoted 41 lines · Usama Arif
From: Usama Arif <[email protected]>
migrate_vma_split_unmapped_folio()takes an extra reference viafolio_get()before callingfolio_split_unmapped(). On success, the split consumes this reference:__folio_freeze_and_split_unmapped()expects the +1 in itsfolio_ref_freeze()check, and distributes it across the resulting sub-folios viafolio_ref_unfreeze(...+1), which are later balanced byfolio_put()calls in__migrate_device_finalize().If
folio_split_unmapped()fails (e.g., unexpected pinning returns-EAGAIN), the function returns without callingfolio_put(). The extra reference is never released.Add the missing
folio_put()on the error path.Fixes 4265d67e405a4 ("mm/migrate_device: add THP splitting during migration")Closes https://lore.kernel.org/all/CAA1CXcDyqPPwf_-W7B+PFQtL8HdoJGCEqVsVxq7DhOUB=L4PQA@mail.gmail.com/ open hereReported-by Nico Pache <[email protected]>Signed-off-by Usama Arif <[email protected]>--- mm/migrate_device.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/migrate_device.c +3
-1@@ -917,8 +917,10 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,folio_get (folio);split_huge_pmd_address (migrate->vma, addr, true); ret =folio_split_unmapped (folio,0 ); -if (ret) +if (ret) { +folio_put (folio);return ret; + } migrate->src[idx] &= ~MIGRATE_PFN_COMPOUND ; flags = migrate->src[idx] & ((1UL <<MIGRATE_PFN_SHIFT ) -1 ); pfn = migrate->src[idx] >>MIGRATE_PFN_SHIFT ;Signature · 1 line
2.47.3
Add Balbir, who wrote the code, to comment on this.
Best Regards,
Yan, Zi
Quoted 37 lines · Zi Yan
Quoted 35 lines · Usama Arif
From: Usama Arif <[email protected]>
migrate_vma_split_unmapped_folio()takes an extra reference viafolio_get()before callingfolio_split_unmapped(). On success, the split consumes this reference:__folio_freeze_and_split_unmapped()expects the +1 in itsfolio_ref_freeze()check, and distributes it across the resulting sub-folios viafolio_ref_unfreeze(...+1), which are later balanced byfolio_put()calls in__migrate_device_finalize().If
folio_split_unmapped()fails (e.g., unexpected pinning returns-EAGAIN), the function returns without callingfolio_put(). The extra reference is never released.Add the missing
folio_put()on the error path.Fixes 4265d67e405a4 ("mm/migrate_device: add THP splitting during migration")Closes https://lore.kernel.org/all/CAA1CXcDyqPPwf_-W7B+PFQtL8HdoJGCEqVsVxq7DhOUB=L4PQA@mail.gmail.com/ open hereReported-by Nico Pache <[email protected]>Signed-off-by Usama Arif <[email protected]>--- mm/migrate_device.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/migrate_device.c +2
-1@@ -917,8 +917,10 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,folio_get (folio);split_huge_pmd_address (migrate->vma, addr, true); ret =folio_split_unmapped (folio,0 ); -if (ret) +if (ret) { +folio_put (folio);return ret;
Quoted 9 lines
Quoted 6 lines
mm/migrate_device.c +1
-0+ } migrate->src[idx] &= ~MIGRATE_PFN_COMPOUND ; flags = migrate->src[idx] & ((1UL <<MIGRATE_PFN_SHIFT ) -1 ); pfn = migrate->src[idx] >>MIGRATE_PFN_SHIFT ;Signature · 1 line
2.47.3Add Balbir, who wrote the code, to comment on this.
Thanks Zi!
Just wondering if there is a reproducer for the issue and how the fix was tested? I expect migrate_vma_finalize() to be called for folios, even when split failed and drop the lock.
Balbir
Zi Yan
reviewerQuoted 48 lines · Balbir Singh
Quoted 37 lines · Zi Yan
Quoted 35 lines · Usama Arif
From: Usama Arif <[email protected]>
migrate_vma_split_unmapped_folio()takes an extra reference viafolio_get()before callingfolio_split_unmapped(). On success, the split consumes this reference:__folio_freeze_and_split_unmapped()expects the +1 in itsfolio_ref_freeze()check, and distributes it across the resulting sub-folios viafolio_ref_unfreeze(...+1), which are later balanced byfolio_put()calls in__migrate_device_finalize().If
folio_split_unmapped()fails (e.g., unexpected pinning returns-EAGAIN), the function returns without callingfolio_put(). The extra reference is never released.Add the missing
folio_put()on the error path.Fixes 4265d67e405a4 ("mm/migrate_device: add THP splitting during migration")Closes https://lore.kernel.org/all/CAA1CXcDyqPPwf_-W7B+PFQtL8HdoJGCEqVsVxq7DhOUB=L4PQA@mail.gmail.com/ open hereReported-by Nico Pache <[email protected]>Signed-off-by Usama Arif <[email protected]>--- mm/migrate_device.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/migrate_device.c +2
-1@@ -917,8 +917,10 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,folio_get (folio);split_huge_pmd_address (migrate->vma, addr, true); ret =folio_split_unmapped (folio,0 ); -if (ret) +if (ret) { +folio_put (folio);return ret;Quoted 9 lines
Quoted 6 lines
mm/migrate_device.c +1
-0+ } migrate->src[idx] &= ~MIGRATE_PFN_COMPOUND ; flags = migrate->src[idx] & ((1UL <<MIGRATE_PFN_SHIFT ) -1 ); pfn = migrate->src[idx] >>MIGRATE_PFN_SHIFT ;Signature · 1 line
2.47.3Add Balbir, who wrote the code, to comment on this.
Quoted 5 lines
Thanks Zi!
Just wondering if there is a reproducer for the issue and how the fix was tested? I expect
migrate_vma_finalize()to be called for folios, even when split failed and drop the lock.
Does migrate_vma_finalize() do folio_put() for failed-to-split folios? If so, how does it distinguish between split folios and failed-to-split folios? By comparing source and destination folio orders?
What we see from migrate_vma_split_unmapped_folio() is that it adds a refcount for all input folios, but only drops a refcount for the split folio. Isn’t it cause failed-to-split folios to have additional refcount?
Best Regards,
Yan, Zi
Matthew Brost
reviewerQuoted 56 lines · Zi Yan
Quoted 48 lines · Balbir Singh
Quoted 37 lines · Zi Yan
Quoted 35 lines · Usama Arif
From: Usama Arif <[email protected]>
migrate_vma_split_unmapped_folio()takes an extra reference viafolio_get()before callingfolio_split_unmapped(). On success, the split consumes this reference:__folio_freeze_and_split_unmapped()expects the +1 in itsfolio_ref_freeze()check, and distributes it across the resulting sub-folios viafolio_ref_unfreeze(...+1), which are later balanced byfolio_put()calls in__migrate_device_finalize().If
folio_split_unmapped()fails (e.g., unexpected pinning returns-EAGAIN), the function returns without callingfolio_put(). The extra reference is never released.Add the missing
folio_put()on the error path.Fixes 4265d67e405a4 ("mm/migrate_device: add THP splitting during migration")Closes https://lore.kernel.org/all/CAA1CXcDyqPPwf_-W7B+PFQtL8HdoJGCEqVsVxq7DhOUB=L4PQA@mail.gmail.com/ open hereReported-by Nico Pache <[email protected]>Signed-off-by Usama Arif <[email protected]>--- mm/migrate_device.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/migrate_device.c +2
-1@@ -917,8 +917,10 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,folio_get (folio);split_huge_pmd_address (migrate->vma, addr, true); ret =folio_split_unmapped (folio,0 ); -if (ret) +if (ret) { +folio_put (folio);return ret;Quoted 9 lines
Quoted 6 lines
mm/migrate_device.c +1
-0+ } migrate->src[idx] &= ~MIGRATE_PFN_COMPOUND ; flags = migrate->src[idx] & ((1UL <<MIGRATE_PFN_SHIFT ) -1 ); pfn = migrate->src[idx] >>MIGRATE_PFN_SHIFT ;Signature · 1 line
2.47.3Add Balbir, who wrote the code, to comment on this.
Quoted 5 lines
Thanks Zi!
Just wondering if there is a reproducer for the issue and how the fix was tested? I expect
migrate_vma_finalize()to be called for folios, even when split failed and drop the lock.
Quoted 8 lines
Does
migrate_vma_finalize()dofolio_put()for failed-to-split folios? If so, how does it distinguish between split folios and failed-to-split folios? By comparing source and destination folio orders?What we see from
migrate_vma_split_unmapped_folio()is that it adds a refcount for all input folios, but only drops a refcount for the split folio. Isn’t it cause failed-to-split folios to have additional refcount?
I wonder if I’ve actually seen this bug. I’ve occasionally seen CPU page faults hang forever spinning, which could be caused by the page’s refcount accidentally being increased here. It’s quite difficult and random to reproduce, so I don’t have a real analysis of what’s happening in this case.
Matt
Quoted 4 lines
Best Regards,
Yan, Zi
Balbir Singh
Acked-byQuoted 52 lines · Zi Yan
Quoted 50 lines · Balbir Singh
Quoted 37 lines · Zi Yan
Quoted 35 lines · Usama Arif
From: Usama Arif <[email protected]>
migrate_vma_split_unmapped_folio()takes an extra reference viafolio_get()before callingfolio_split_unmapped(). On success, the split consumes this reference:__folio_freeze_and_split_unmapped()expects the +1 in itsfolio_ref_freeze()check, and distributes it across the resulting sub-folios viafolio_ref_unfreeze(...+1), which are later balanced byfolio_put()calls in__migrate_device_finalize().If
folio_split_unmapped()fails (e.g., unexpected pinning returns-EAGAIN), the function returns without callingfolio_put(). The extra reference is never released.Add the missing
folio_put()on the error path.Fixes 4265d67e405a4 ("mm/migrate_device: add THP splitting during migration")Closes https://lore.kernel.org/all/CAA1CXcDyqPPwf_-W7B+PFQtL8HdoJGCEqVsVxq7DhOUB=L4PQA@mail.gmail.com/ open hereReported-by Nico Pache <[email protected]>Signed-off-by Usama Arif <[email protected]>--- mm/migrate_device.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/migrate_device.c +2
-1@@ -917,8 +917,10 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,folio_get (folio);split_huge_pmd_address (migrate->vma, addr, true); ret =folio_split_unmapped (folio,0 ); -if (ret) +if (ret) { +folio_put (folio);return ret;Quoted 9 lines
Quoted 6 lines
mm/migrate_device.c +1
-0+ } migrate->src[idx] &= ~MIGRATE_PFN_COMPOUND ; flags = migrate->src[idx] & ((1UL <<MIGRATE_PFN_SHIFT ) -1 ); pfn = migrate->src[idx] >>MIGRATE_PFN_SHIFT ;Signature · 1 line
2.47.3Add Balbir, who wrote the code, to comment on this.
Thanks Zi!
Quoted 8 lines
Quoted 3 lines
Just wondering if there is a reproducer for the issue and how the fix was tested? I expect
migrate_vma_finalize()to be called for folios, even when split failed and drop the lock.Does
migrate_vma_finalize()dofolio_put()for failed-to-split folios? If so, how does it distinguish between split folios and failed-to-split folios? By comparing source and destination folio orders?
We reset the MIGRATE_PFN_MIGRATE flag for failing to migrate pfns. We do a folio_put on the src in finalize, if it is split then on all the split folios as well.
Quoted 5 lines
What we see from
migrate_vma_split_unmapped_folio()is that it adds a refcount for all input folios, but only drops a refcount for the split folio. Isn’t it cause failed-to-split folios to have additional refcount?
Thanks! Yes, the patch makes sense
Balbir
Usama Arif
reviewerQuoted 57 lines · Balbir Singh
Quoted 56 lines · Zi Yan
Quoted 53 lines · Balbir Singh
Quoted 37 lines · Zi Yan
Quoted 35 lines · Usama Arif
From: Usama Arif <[email protected]>
migrate_vma_split_unmapped_folio()takes an extra reference viafolio_get()before callingfolio_split_unmapped(). On success, the split consumes this reference:__folio_freeze_and_split_unmapped()expects the +1 in itsfolio_ref_freeze()check, and distributes it across the resulting sub-folios viafolio_ref_unfreeze(...+1), which are later balanced byfolio_put()calls in__migrate_device_finalize().If
folio_split_unmapped()fails (e.g., unexpected pinning returns-EAGAIN), the function returns without callingfolio_put(). The extra reference is never released.Add the missing
folio_put()on the error path.Fixes 4265d67e405a4 ("mm/migrate_device: add THP splitting during migration")Closes https://lore.kernel.org/all/CAA1CXcDyqPPwf_-W7B+PFQtL8HdoJGCEqVsVxq7DhOUB=L4PQA@mail.gmail.com/ open hereReported-by Nico Pache <[email protected]>Signed-off-by Usama Arif <[email protected]>--- mm/migrate_device.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/migrate_device.c +2
-1@@ -917,8 +917,10 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,folio_get (folio);split_huge_pmd_address (migrate->vma, addr, true); ret =folio_split_unmapped (folio,0 ); -if (ret) +if (ret) { +folio_put (folio);return ret;Quoted 9 lines
Quoted 6 lines
mm/migrate_device.c +1
-0+ } migrate->src[idx] &= ~MIGRATE_PFN_COMPOUND ; flags = migrate->src[idx] & ((1UL <<MIGRATE_PFN_SHIFT ) -1 ); pfn = migrate->src[idx] >>MIGRATE_PFN_SHIFT ;Signature · 1 line
2.47.3Add Balbir, who wrote the code, to comment on this.
Thanks Zi!
Just wondering if there is a reproducer for the issue and how the fix was tested? I expect
migrate_vma_finalize()to be called for folios, even when split failed and drop the lock.
Quoted 13 lines
Quoted 4 lines
Does
migrate_vma_finalize()dofolio_put()for failed-to-split folios? If so, how does it distinguish between split folios and failed-to-split folios? By comparing source and destination folio orders?We reset the
MIGRATE_PFN_MIGRATEflag for failing to migrate pfns. We do afolio_puton the src in finalize, if it is split then on all the split folios as well.Quoted 5 lines
What we see from
migrate_vma_split_unmapped_folio()is that it adds a refcount for all input folios, but only drops a refcount for the split folio. Isn’t it cause failed-to-split folios to have additional refcount?
Hello!
Thanks for reviewing everyone. So its very difficult to create a reproducer I think the extra reference would need to appear after migrate_device_unmap() but before folio_split_unmapped() in migrate_vma_pages()? That's hard to trigger reliably from userspace.
The fix came about when Nico indicated there might be an issue if split_huge_pmd_address fails in my patch [1].
Below is my understanding of how refcounting is working over here step by step. I might very well be wrong on this, and the refcounting is a bit all over the place and I might miss a reference change somewhere so would really appreciate if someone can confirm this!
1. migrate_vma_collect_huge_pmd():
a) folio_get(folio) -> +1 (collect reference)
2. migrate_device_unmap():
a) folio_isolate_lru() -> +1 (isolation reference)
b) folio_put() -> -1 (drops the collect reference)
Without this patch fix:
3. migrate_vma_split_unmapped_folio():
a) folio_get(folio) -> +1 (split reference)
b) folio_split_unmapped() -> fails
c) Returns error — without folio_put() which is the fix
4. Caller in migrate_vma_pages(): clears MIGRATE_PFN_MIGRATE | MIGRATE_PFN_COMPOUND
5. __migrate_device_finalize(): sees !(src_pfns[i] & MIGRATE_PFN_MIGRATE), restores the folio:
a) remove_migration_ptes(src, src) — re-establishes user PTEs
b) folio_unlock(src)
c) folio_put(src) -> -1 (drops the isolation reference)
The split reference in 3.a is never released and the folio has a permanently elevated refcount. Unless I missed a folio_put somewhere for the refcount increase in folio_isolate_lru() (2.b)?
Please let me know if this makes sense!
[1] https://lore.kernel.org/all/CAA1CXcDyqPPwf_-W7B+PFQtL8HdoJGCEqVsVxq7DhOUB=L4PQA@mail.gmail.com/
Quoted 6 lines
Thanks! Yes, the patch makes sense
Acked-by Balbir Singh <[email protected]>Balbir
Hi!
Quoted 103 lines · Usama Arif
Quoted 54 lines · Balbir Singh
Quoted 53 lines · Zi Yan
Quoted 51 lines · Balbir Singh
Quoted 36 lines · Zi Yan
Quoted 34 lines · Usama Arif
From: Usama Arif <[email protected]>
migrate_vma_split_unmapped_folio()takes an extra reference viafolio_get()before callingfolio_split_unmapped(). On success, the split consumes this reference:__folio_freeze_and_split_unmapped()expects the +1 in itsfolio_ref_freeze()check, and distributes it across the resulting sub-folios viafolio_ref_unfreeze(...+1), which are later balanced byfolio_put()calls in__migrate_device_finalize().If
folio_split_unmapped()fails (e.g., unexpected pinning returns-EAGAIN), the function returns without callingfolio_put(). The extra reference is never released.Add the missing
folio_put()on the error path.Fixes 4265d67e405a4 ("mm/migrate_device: add THP splitting during migration")Closes https://lore.kernel.org/all/CAA1CXcDyqPPwf_-W7B+PFQtL8HdoJGCEqVsVxq7DhOUB=L4PQA@mail.gmail.com/ open hereReported-by Nico Pache <[email protected]>Signed-off-by Usama Arif <[email protected]>--- mm/migrate_device.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/migrate_device.c +2
-1@@ -917,8 +917,10 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,folio_get (folio);split_huge_pmd_address (migrate->vma, addr, true); ret =folio_split_unmapped (folio,0 ); -if (ret) +if (ret) { +folio_put (folio);Quoted 9 lines
Quoted 7 lines
mm/migrate_device.c +1
-0return ret; + } migrate->src[idx] &= ~MIGRATE_PFN_COMPOUND ; flags = migrate->src[idx] & ((1UL <<MIGRATE_PFN_SHIFT ) -1 ); pfn = migrate->src[idx] >>MIGRATE_PFN_SHIFT ;Signature · 1 line
2.47.3Add Balbir, who wrote the code, to comment on this.
Thanks Zi!
Just wondering if there is a reproducer for the issue and how the fix was tested? I expect
migrate_vma_finalize()to be called for folios, even when split failed and drop the lock.Quoted 12 lines
Quoted 4 lines
Does
migrate_vma_finalize()dofolio_put()for failed-to-split folios? If so, how does it distinguish between split folios and failed-to-split folios? By comparing source and destination folio orders?We reset the
MIGRATE_PFN_MIGRATEflag for failing to migrate pfns. We do afolio_puton the src in finalize, if it is split then on all the split folios as well.Quoted 5 lines
What we see from
migrate_vma_split_unmapped_folio()is that it adds a refcount for all input folios, but only drops a refcount for the split folio. Isn’t it cause failed-to-split folios to have additional refcount?Hello!
Thanks for reviewing everyone. So its very difficult to create a reproducer I think the extra reference would need to appear after
migrate_device_unmap()but beforefolio_split_unmapped()inmigrate_vma_pages()? That's hard to trigger reliably from userspace.The fix came about when Nico indicated there might be an issue if
split_huge_pmd_addressfails in my patch [1].Below is my understanding of how refcounting is working over here step by step. I might very well be wrong on this, and the refcounting is a bit all over the place and I might miss a reference change somewhere so would really appreciate if someone can confirm this!
1.
migrate_vma_collect_huge_pmd():
a)folio_get(folio)-> +1 (collect reference)
2.migrate_device_unmap():
a)folio_isolate_lru()-> +1 (isolation reference)
b)folio_put()-> -1 (drops the collect reference)Without this patch fix:
3.
migrate_vma_split_unmapped_folio():
a)folio_get(folio)-> +1 (split reference)
b)folio_split_unmapped()-> fails
c) Returns error — withoutfolio_put()which is the fix
4. Caller inmigrate_vma_pages(): clearsMIGRATE_PFN_MIGRATE|MIGRATE_PFN_COMPOUND
5.__migrate_device_finalize(): sees !(src_pfns[i] &MIGRATE_PFN_MIGRATE), restores the folio:
a)remove_migration_ptes(src, src)— re-establishes user PTEs
b)folio_unlock(src)
c)folio_put(src)-> -1 (drops the isolation reference)
Quoted 12 lines
The split reference in 3.a is never released and the folio has a permanently elevated refcount. Unless I missed a
folio_putsomewhere for the refcount increase infolio_isolate_lru()(2.b)?Please let me know if this makes sense!
[1] https://lore.kernel.org/all/CAA1CXcDyqPPwf_-W7B+PFQtL8HdoJGCEqVsVxq7DhOUB=L4PQA@mail.gmail.com/
Quoted 5 lines
Thanks! Yes, the patch makes sense
Acked-by Balbir Singh <[email protected]>Balbir
I remember stumbling on this while ago also. The folio_get() in migrate_vma_split_unmapped_folio() is balanced with put_page() in __split_huge_pmd_locked() (freeze = true), can't fail for device pages. Folios at this point are unmapped but have 1 refcount from "collecting".
After folio_split_unmapped() the refcount(s) is still 1.
So it seems the code is good as is? A comment though would be good for the extra folio_get..
Thanks,
--Mika
Usama Arif
reviewerQuoted 118 lines · Mika Penttilä
Hi!
Quoted 114 lines · Usama Arif
Quoted 54 lines · Balbir Singh
Quoted 53 lines · Zi Yan
Quoted 51 lines · Balbir Singh
Quoted 36 lines · Zi Yan
Quoted 34 lines · Usama Arif
From: Usama Arif <[email protected]>
migrate_vma_split_unmapped_folio()takes an extra reference viafolio_get()before callingfolio_split_unmapped(). On success, the split consumes this reference:__folio_freeze_and_split_unmapped()expects the +1 in itsfolio_ref_freeze()check, and distributes it across the resulting sub-folios viafolio_ref_unfreeze(...+1), which are later balanced byfolio_put()calls in__migrate_device_finalize().If
folio_split_unmapped()fails (e.g., unexpected pinning returns-EAGAIN), the function returns without callingfolio_put(). The extra reference is never released.Add the missing
folio_put()on the error path.Fixes 4265d67e405a4 ("mm/migrate_device: add THP splitting during migration")Closes https://lore.kernel.org/all/CAA1CXcDyqPPwf_-W7B+PFQtL8HdoJGCEqVsVxq7DhOUB=L4PQA@mail.gmail.com/ open hereReported-by Nico Pache <[email protected]>Signed-off-by Usama Arif <[email protected]>--- mm/migrate_device.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/migrate_device.c +2
-1@@ -917,8 +917,10 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,folio_get (folio);split_huge_pmd_address (migrate->vma, addr, true); ret =folio_split_unmapped (folio,0 ); -if (ret) +if (ret) { +folio_put (folio);Quoted 9 lines
Quoted 7 lines
mm/migrate_device.c +1
-0return ret; + } migrate->src[idx] &= ~MIGRATE_PFN_COMPOUND ; flags = migrate->src[idx] & ((1UL <<MIGRATE_PFN_SHIFT ) -1 ); pfn = migrate->src[idx] >>MIGRATE_PFN_SHIFT ;Signature · 1 line
2.47.3Add Balbir, who wrote the code, to comment on this.
Thanks Zi!
Just wondering if there is a reproducer for the issue and how the fix was tested? I expect
migrate_vma_finalize()to be called for folios, even when split failed and drop the lock.Quoted 12 lines
Quoted 4 lines
Does
migrate_vma_finalize()dofolio_put()for failed-to-split folios? If so, how does it distinguish between split folios and failed-to-split folios? By comparing source and destination folio orders?We reset the
MIGRATE_PFN_MIGRATEflag for failing to migrate pfns. We do afolio_puton the src in finalize, if it is split then on all the split folios as well.Quoted 5 lines
What we see from
migrate_vma_split_unmapped_folio()is that it adds a refcount for all input folios, but only drops a refcount for the split folio. Isn’t it cause failed-to-split folios to have additional refcount?Hello!
Thanks for reviewing everyone. So its very difficult to create a reproducer I think the extra reference would need to appear after
migrate_device_unmap()but beforefolio_split_unmapped()inmigrate_vma_pages()? That's hard to trigger reliably from userspace.The fix came about when Nico indicated there might be an issue if
split_huge_pmd_addressfails in my patch [1].Below is my understanding of how refcounting is working over here step by step. I might very well be wrong on this, and the refcounting is a bit all over the place and I might miss a reference change somewhere so would really appreciate if someone can confirm this!
1.
migrate_vma_collect_huge_pmd():
a)folio_get(folio)-> +1 (collect reference)
2.migrate_device_unmap():
a)folio_isolate_lru()-> +1 (isolation reference)
b)folio_put()-> -1 (drops the collect reference)Without this patch fix:
3.
migrate_vma_split_unmapped_folio():
a)folio_get(folio)-> +1 (split reference)
b)folio_split_unmapped()-> fails
c) Returns error — withoutfolio_put()which is the fix
4. Caller inmigrate_vma_pages(): clearsMIGRATE_PFN_MIGRATE|MIGRATE_PFN_COMPOUND
5.__migrate_device_finalize(): sees !(src_pfns[i] &MIGRATE_PFN_MIGRATE), restores the folio:
a)remove_migration_ptes(src, src)— re-establishes user PTEs
b)folio_unlock(src)
c)folio_put(src)-> -1 (drops the isolation reference)The split reference in 3.a is never released and the folio has a permanently elevated refcount. Unless I missed a
folio_putsomewhere for the refcount increase infolio_isolate_lru()(2.b)?Please let me know if this makes sense!
[1] https://lore.kernel.org/all/CAA1CXcDyqPPwf_-W7B+PFQtL8HdoJGCEqVsVxq7DhOUB=L4PQA@mail.gmail.com/
Quoted 4 lines
Thanks! Yes, the patch makes sense
Acked-by Balbir Singh <[email protected]>
Quoted 9 lines
Quoted 1 line
Quoted 1 line
Balbir
I remember stumbling on this while ago also. The
folio_get()inmigrate_vma_split_unmapped_folio()is balanced withput_page()in__split_huge_pmd_locked()(freeze = true), can't fail for device pages. Folios at this point are unmapped but have 1 refcount from "collecting".
Afterfolio_split_unmapped()the refcount(s) is still 1.So it seems the code is good as is? A comment though would be good for the extra
folio_get..
hmm I dont think the put_page() in __split_huge_pmd_locked() is there to balance the folio_get() in migrate_vma_split_unmapped_folio(). There are other points where split_huge_pmd_locked() is called with freeze = true [1] and they don't get a reference before calling split_huge_pmd.
I think the folio_put() in __split_huge_pmd_locked() freeze = true case is there as migration entries are being installed?
[1] https://elixir.bootlin.com/linux/v6.19.3/source/mm/rmap.c#L2334
Quoted 127 lines · Usama Arif
Quoted 117 lines · Mika Penttilä
Hi!
Quoted 113 lines · Usama Arif
Quoted 54 lines · Balbir Singh
Quoted 53 lines · Zi Yan
Quoted 51 lines · Balbir Singh
Quoted 36 lines · Zi Yan
Quoted 34 lines · Usama Arif
From: Usama Arif <[email protected]>
migrate_vma_split_unmapped_folio()takes an extra reference viafolio_get()before callingfolio_split_unmapped(). On success, the split consumes this reference:__folio_freeze_and_split_unmapped()expects the +1 in itsfolio_ref_freeze()check, and distributes it across the resulting sub-folios viafolio_ref_unfreeze(...+1), which are later balanced byfolio_put()calls in__migrate_device_finalize().If
folio_split_unmapped()fails (e.g., unexpected pinning returns-EAGAIN), the function returns without callingfolio_put(). The extra reference is never released.Add the missing
folio_put()on the error path.Fixes 4265d67e405a4 ("mm/migrate_device: add THP splitting during migration")Closes https://lore.kernel.org/all/CAA1CXcDyqPPwf_-W7B+PFQtL8HdoJGCEqVsVxq7DhOUB=L4PQA@mail.gmail.com/ open hereReported-by Nico Pache <[email protected]>Signed-off-by Usama Arif <[email protected]>--- mm/migrate_device.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/migrate_device.c +2
-1@@ -917,8 +917,10 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,folio_get (folio);split_huge_pmd_address (migrate->vma, addr, true); ret =folio_split_unmapped (folio,0 ); -if (ret) +if (ret) { +folio_put (folio);Quoted 9 lines
Quoted 7 lines
mm/migrate_device.c +1
-0return ret; + } migrate->src[idx] &= ~MIGRATE_PFN_COMPOUND ; flags = migrate->src[idx] & ((1UL <<MIGRATE_PFN_SHIFT ) -1 ); pfn = migrate->src[idx] >>MIGRATE_PFN_SHIFT ;Signature · 1 line
2.47.3Add Balbir, who wrote the code, to comment on this.
Thanks Zi!
Just wondering if there is a reproducer for the issue and how the fix was tested? I expect
migrate_vma_finalize()to be called for folios, even when split failed and drop the lock.Quoted 12 lines
Quoted 4 lines
Does
migrate_vma_finalize()dofolio_put()for failed-to-split folios? If so, how does it distinguish between split folios and failed-to-split folios? By comparing source and destination folio orders?We reset the
MIGRATE_PFN_MIGRATEflag for failing to migrate pfns. We do afolio_puton the src in finalize, if it is split then on all the split folios as well.Quoted 5 lines
What we see from
migrate_vma_split_unmapped_folio()is that it adds a refcount for all input folios, but only drops a refcount for the split folio. Isn’t it cause failed-to-split folios to have additional refcount?Hello!
Thanks for reviewing everyone. So its very difficult to create a reproducer I think the extra reference would need to appear after
migrate_device_unmap()but beforefolio_split_unmapped()inmigrate_vma_pages()? That's hard to trigger reliably from userspace.The fix came about when Nico indicated there might be an issue if
split_huge_pmd_addressfails in my patch [1].Below is my understanding of how refcounting is working over here step by step. I might very well be wrong on this, and the refcounting is a bit all over the place and I might miss a reference change somewhere so would really appreciate if someone can confirm this!
1.
migrate_vma_collect_huge_pmd():
a)folio_get(folio)-> +1 (collect reference)
2.migrate_device_unmap():
a)folio_isolate_lru()-> +1 (isolation reference)
b)folio_put()-> -1 (drops the collect reference)Without this patch fix:
3.
migrate_vma_split_unmapped_folio():
a)folio_get(folio)-> +1 (split reference)
b)folio_split_unmapped()-> fails
c) Returns error — withoutfolio_put()which is the fix
4. Caller inmigrate_vma_pages(): clearsMIGRATE_PFN_MIGRATE|MIGRATE_PFN_COMPOUND
5.__migrate_device_finalize(): sees !(src_pfns[i] &MIGRATE_PFN_MIGRATE), restores the folio:
a)remove_migration_ptes(src, src)— re-establishes user PTEs
b)folio_unlock(src)
c)folio_put(src)-> -1 (drops the isolation reference)The split reference in 3.a is never released and the folio has a permanently elevated refcount. Unless I missed a
folio_putsomewhere for the refcount increase infolio_isolate_lru()(2.b)?Please let me know if this makes sense!
[1] https://lore.kernel.org/all/CAA1CXcDyqPPwf_-W7B+PFQtL8HdoJGCEqVsVxq7DhOUB=L4PQA@mail.gmail.com/
Quoted 4 lines
Thanks! Yes, the patch makes sense
Acked-by Balbir Singh <[email protected]>Quoted 8 lines
Quoted 1 line
Quoted 1 line
Balbir
I remember stumbling on this while ago also. The
folio_get()inmigrate_vma_split_unmapped_folio()is balanced withput_page()in__split_huge_pmd_locked()(freeze = true), can't fail for device pages. Folios at this point are unmapped but have 1 refcount from "collecting".
Afterfolio_split_unmapped()the refcount(s) is still 1.So it seems the code is good as is? A comment though would be good for the extra
folio_get..
Quoted 10 lines
hmm I dont think the
put_page()in__split_huge_pmd_locked()is there to balance thefolio_get()inmigrate_vma_split_unmapped_folio(). There are other points wheresplit_huge_pmd_locked()is called with freeze = true [1] and they don't get a reference before callingsplit_huge_pmd.I think the
folio_put()in__split_huge_pmd_locked()freeze = true case is there as migration entries are being installed?[1] https://elixir.bootlin.com/linux/v6.19.3/source/mm/rmap.c#L2334
Yes normally you want to drop the reference when installing migration entries but in this context you have already done the collecting for the THP folio and you want to balance with the folio_get() the put_page() to keep the refs unchanged. Is that right Balbir?
--Mika
Usama Arif
reviewerQuoted 136 lines · Mika Penttilä
Quoted 134 lines · Usama Arif
Quoted 117 lines · Mika Penttilä
Hi!
Quoted 113 lines · Usama Arif
Quoted 54 lines · Balbir Singh
Quoted 53 lines · Zi Yan
Quoted 51 lines · Balbir Singh
Quoted 36 lines · Zi Yan
Quoted 34 lines · Usama Arif
From: Usama Arif <[email protected]>
migrate_vma_split_unmapped_folio()takes an extra reference viafolio_get()before callingfolio_split_unmapped(). On success, the split consumes this reference:__folio_freeze_and_split_unmapped()expects the +1 in itsfolio_ref_freeze()check, and distributes it across the resulting sub-folios viafolio_ref_unfreeze(...+1), which are later balanced byfolio_put()calls in__migrate_device_finalize().If
folio_split_unmapped()fails (e.g., unexpected pinning returns-EAGAIN), the function returns without callingfolio_put(). The extra reference is never released.Add the missing
folio_put()on the error path.Fixes 4265d67e405a4 ("mm/migrate_device: add THP splitting during migration")Closes https://lore.kernel.org/all/CAA1CXcDyqPPwf_-W7B+PFQtL8HdoJGCEqVsVxq7DhOUB=L4PQA@mail.gmail.com/ open hereReported-by Nico Pache <[email protected]>Signed-off-by Usama Arif <[email protected]>--- mm/migrate_device.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/migrate_device.c +2
-1@@ -917,8 +917,10 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,folio_get (folio);split_huge_pmd_address (migrate->vma, addr, true); ret =folio_split_unmapped (folio,0 ); -if (ret) +if (ret) { +folio_put (folio);Quoted 9 lines
Quoted 7 lines
mm/migrate_device.c +1
-0return ret; + } migrate->src[idx] &= ~MIGRATE_PFN_COMPOUND ; flags = migrate->src[idx] & ((1UL <<MIGRATE_PFN_SHIFT ) -1 ); pfn = migrate->src[idx] >>MIGRATE_PFN_SHIFT ;Signature · 1 line
2.47.3Add Balbir, who wrote the code, to comment on this.
Thanks Zi!
Just wondering if there is a reproducer for the issue and how the fix was tested? I expect
migrate_vma_finalize()to be called for folios, even when split failed and drop the lock.Quoted 12 lines
Quoted 4 lines
Does
migrate_vma_finalize()dofolio_put()for failed-to-split folios? If so, how does it distinguish between split folios and failed-to-split folios? By comparing source and destination folio orders?We reset the
MIGRATE_PFN_MIGRATEflag for failing to migrate pfns. We do afolio_puton the src in finalize, if it is split then on all the split folios as well.Quoted 5 lines
What we see from
migrate_vma_split_unmapped_folio()is that it adds a refcount for all input folios, but only drops a refcount for the split folio. Isn’t it cause failed-to-split folios to have additional refcount?Hello!
Thanks for reviewing everyone. So its very difficult to create a reproducer I think the extra reference would need to appear after
migrate_device_unmap()but beforefolio_split_unmapped()inmigrate_vma_pages()? That's hard to trigger reliably from userspace.The fix came about when Nico indicated there might be an issue if
split_huge_pmd_addressfails in my patch [1].Below is my understanding of how refcounting is working over here step by step. I might very well be wrong on this, and the refcounting is a bit all over the place and I might miss a reference change somewhere so would really appreciate if someone can confirm this!
1.
migrate_vma_collect_huge_pmd():
a)folio_get(folio)-> +1 (collect reference)
2.migrate_device_unmap():
a)folio_isolate_lru()-> +1 (isolation reference)
b)folio_put()-> -1 (drops the collect reference)Without this patch fix:
3.
migrate_vma_split_unmapped_folio():
a)folio_get(folio)-> +1 (split reference)
b)folio_split_unmapped()-> fails
c) Returns error — withoutfolio_put()which is the fix
4. Caller inmigrate_vma_pages(): clearsMIGRATE_PFN_MIGRATE|MIGRATE_PFN_COMPOUND
5.__migrate_device_finalize(): sees !(src_pfns[i] &MIGRATE_PFN_MIGRATE), restores the folio:
a)remove_migration_ptes(src, src)— re-establishes user PTEs
b)folio_unlock(src)
c)folio_put(src)-> -1 (drops the isolation reference)The split reference in 3.a is never released and the folio has a permanently elevated refcount. Unless I missed a
folio_putsomewhere for the refcount increase infolio_isolate_lru()(2.b)?Please let me know if this makes sense!
[1] https://lore.kernel.org/all/CAA1CXcDyqPPwf_-W7B+PFQtL8HdoJGCEqVsVxq7DhOUB=L4PQA@mail.gmail.com/
Quoted 4 lines
Thanks! Yes, the patch makes sense
Acked-by Balbir Singh <[email protected]>Quoted 8 lines
Quoted 1 line
Quoted 1 line
Balbir
I remember stumbling on this while ago also. The
folio_get()inmigrate_vma_split_unmapped_folio()is balanced withput_page()in__split_huge_pmd_locked()(freeze = true), can't fail for device pages. Folios at this point are unmapped but have 1 refcount from "collecting".
Afterfolio_split_unmapped()the refcount(s) is still 1.So it seems the code is good as is? A comment though would be good for the extra
folio_get..hmm I dont think the
put_page()in__split_huge_pmd_locked()is there to balance thefolio_get()inmigrate_vma_split_unmapped_folio(). There are other points wheresplit_huge_pmd_locked()is called with freeze = true [1] and they don't get a reference before callingsplit_huge_pmd.I think the
folio_put()in__split_huge_pmd_locked()freeze = true case is there as migration entries are being installed?
Quoted 9 lines
Quoted 3 lines
[1] https://elixir.bootlin.com/linux/v6.19.3/source/mm/rmap.c#L2334
Yes normally you want to drop the reference when installing migration entries but in this context you have already done the collecting for the THP folio and you want to balance with the
folio_get()theput_page()to keep the refs unchanged. Is that right Balbir?--Mika
Hi Mika,
You are right, This patch is wrong. I tried the below diff to force folio_split_unmapped to return -EAGAIN. I ran tools/testing/selftests/mm/hmm-tests -r hmm.hmm_device_private.migrate_anon_huge_err to trigger the path for folio_split_unmapped.
mm/huge_memory.c +2 -0
@@ -4140,6 +4140,8 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)if (folio_expected_ref_count (folio) !=folio_ref_count (folio) -1 )return -EAGAIN ; +return -EAGAIN ; +local_irq_disable (); ret = __folio_freeze_and_split_unmapped(folio, new_order, &folio->page, NULL, NULL, false, NULL, SPLIT_TYPE_UNIFORM,
I inserted a lot of traces to keep track of refcounts [1]. Without this patch, I get
....
hmm-tests-129 [000] ..... 1.476233: __migrate_device_pages: SPLIT_UNMAPPED: folio=ffc536e2c4100000 refcount=0 AFTER error NO folio_put
hmm-tests-129 [000] ..... 1.476234: __migrate_device_pages: PAGES: split FAILED folio=ffc536e2c4100000 refcount=0
hmm-tests-129 [000] ..... 1.476236: __migrate_device_finalize: FINALIZE[0]: src=ffc536e2c4100000 dst=ffc536e2c4100000 src==dst=1 refcount_src=1 mapcount_src=0 order_src=0 migrate=0 BEFORE remove_migration_ptes
hmm-tests-129 [000] ..... 1.476237: __migrate_device_finalize: FINALIZE[0]: src=ffc536e2c4100000 refcount=1 mapcount=0 AFTER remove_migration_ptes
hmm-tests-129 [000] ..... 1.476237: __migrate_device_finalize: FINALIZE[0]: src=ffc536e2c4100000 refcount=0 AFTER folio_put(src)i.e. refcount = 512, which is correct as split_huge_pmd_address was successful. Full output is at [2].
With this patch, I get:
BUG: Bad rss-counter state mm:00000000cfe88d5e type:MM_FILEPAGES val:-511 Comm:bash Pid:63
BUG: Bad rss-counter state mm:00000000cfe88d5e type:MM_ANONPAGES val:511 Comm:bash Pid:63
...
hmm-tests-129 [000] ..... 1.468315: __migrate_device_pages: SPLIT_UNMAPPED: folio=ffed210c840f0000 refcount=1 AFTER error folio_put FIX PRESENT
hmm-tests-129 [000] ..... 1.468315: __migrate_device_pages: PAGES: split FAILED folio=ffed210c840f0000 refcount=1
hmm-tests-129 [000] ..... 1.468318: __migrate_device_finalize: FINALIZE[0]: src=ffed210c840f0000 dst=ffed210c840f0000 src==dst=1 refcount_src=1 mapcount_src=0 order_src=9 migrate=0 BEFORE remove_migration_ptes
hmm-tests-129 [000] ..... 1.468357: __migrate_device_finalize: FINALIZE[0]: src=ffed210c840f0000 refcount=513 mapcount=512 AFTER remove_migration_ptes
hmm-tests-129 [000] ..... 1.468357: __migrate_device_finalize: FINALIZE[0]: src=ffed210c840f0000 refcount=512 AFTER folio_put(src)refcount=0 means the folio would be freed which is not correct. The full output is at [3].
Thank you for clearing this up!
[1] https://gist.github.com/uarif1/65e1e816af7aa0ae38dd6ec64d62a993
[2] https://gist.github.com/uarif1/79ea9500667daa4e2ef09cb5d308f041
[3] https://gist.github.com/uarif1/8a35a6c65ba8b3a1c1dfe72dc30e821d
Zi Yan
reviewerQuoted 191 lines · Usama Arif
Quoted 136 lines · Mika Penttilä
Quoted 134 lines · Usama Arif
Quoted 117 lines · Mika Penttilä
Hi!
Quoted 113 lines · Usama Arif
Quoted 54 lines · Balbir Singh
Quoted 53 lines · Zi Yan
Quoted 51 lines · Balbir Singh
Quoted 36 lines · Zi Yan
Quoted 34 lines · Usama Arif
From: Usama Arif <[email protected]>
migrate_vma_split_unmapped_folio()takes an extra reference viafolio_get()before callingfolio_split_unmapped(). On success, the split consumes this reference:__folio_freeze_and_split_unmapped()expects the +1 in itsfolio_ref_freeze()check, and distributes it across the resulting sub-folios viafolio_ref_unfreeze(...+1), which are later balanced byfolio_put()calls in__migrate_device_finalize().If
folio_split_unmapped()fails (e.g., unexpected pinning returns-EAGAIN), the function returns without callingfolio_put(). The extra reference is never released.Add the missing
folio_put()on the error path.Fixes 4265d67e405a4 ("mm/migrate_device: add THP splitting during migration")Closes https://lore.kernel.org/all/CAA1CXcDyqPPwf_-W7B+PFQtL8HdoJGCEqVsVxq7DhOUB=L4PQA@mail.gmail.com/ open hereReported-by Nico Pache <[email protected]>Signed-off-by Usama Arif <[email protected]>--- mm/migrate_device.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/migrate_device.c +2
-1@@ -917,8 +917,10 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,folio_get (folio);split_huge_pmd_address (migrate->vma, addr, true); ret =folio_split_unmapped (folio,0 ); -if (ret) +if (ret) { +folio_put (folio);Quoted 9 lines
Quoted 7 lines
mm/migrate_device.c +1
-0return ret; + } migrate->src[idx] &= ~MIGRATE_PFN_COMPOUND ; flags = migrate->src[idx] & ((1UL <<MIGRATE_PFN_SHIFT ) -1 ); pfn = migrate->src[idx] >>MIGRATE_PFN_SHIFT ;Signature · 1 line
2.47.3Add Balbir, who wrote the code, to comment on this.
Thanks Zi!
Just wondering if there is a reproducer for the issue and how the fix was tested? I expect
migrate_vma_finalize()to be called for folios, even when split failed and drop the lock.Quoted 12 lines
Quoted 4 lines
Does
migrate_vma_finalize()dofolio_put()for failed-to-split folios? If so, how does it distinguish between split folios and failed-to-split folios? By comparing source and destination folio orders?We reset the
MIGRATE_PFN_MIGRATEflag for failing to migrate pfns. We do afolio_puton the src in finalize, if it is split then on all the split folios as well.Quoted 5 lines
What we see from
migrate_vma_split_unmapped_folio()is that it adds a refcount for all input folios, but only drops a refcount for the split folio. Isn’t it cause failed-to-split folios to have additional refcount?Hello!
Thanks for reviewing everyone. So its very difficult to create a reproducer I think the extra reference would need to appear after
migrate_device_unmap()but beforefolio_split_unmapped()inmigrate_vma_pages()? That's hard to trigger reliably from userspace.The fix came about when Nico indicated there might be an issue if
split_huge_pmd_addressfails in my patch [1].Below is my understanding of how refcounting is working over here step by step. I might very well be wrong on this, and the refcounting is a bit all over the place and I might miss a reference change somewhere so would really appreciate if someone can confirm this!
1.
migrate_vma_collect_huge_pmd():
a)folio_get(folio)-> +1 (collect reference)
2.migrate_device_unmap():
a)folio_isolate_lru()-> +1 (isolation reference)
b)folio_put()-> -1 (drops the collect reference)Without this patch fix:
3.
migrate_vma_split_unmapped_folio():
a)folio_get(folio)-> +1 (split reference)
b)folio_split_unmapped()-> fails
c) Returns error — withoutfolio_put()which is the fix
4. Caller inmigrate_vma_pages(): clearsMIGRATE_PFN_MIGRATE|MIGRATE_PFN_COMPOUND
5.__migrate_device_finalize(): sees !(src_pfns[i] &MIGRATE_PFN_MIGRATE), restores the folio:
a)remove_migration_ptes(src, src)— re-establishes user PTEs
b)folio_unlock(src)
c)folio_put(src)-> -1 (drops the isolation reference)The split reference in 3.a is never released and the folio has a permanently elevated refcount. Unless I missed a
folio_putsomewhere for the refcount increase infolio_isolate_lru()(2.b)?Please let me know if this makes sense!
[1] https://lore.kernel.org/all/CAA1CXcDyqPPwf_-W7B+PFQtL8HdoJGCEqVsVxq7DhOUB=L4PQA@mail.gmail.com/
Quoted 4 lines
Thanks! Yes, the patch makes sense
Acked-by Balbir Singh <[email protected]>Quoted 8 lines
Quoted 1 line
Quoted 1 line
Balbir
I remember stumbling on this while ago also. The
folio_get()inmigrate_vma_split_unmapped_folio()is balanced withput_page()in__split_huge_pmd_locked()(freeze = true), can't fail for device pages. Folios at this point are unmapped but have 1 refcount from "collecting".
Afterfolio_split_unmapped()the refcount(s) is still 1.So it seems the code is good as is? A comment though would be good for the extra
folio_get..hmm I dont think the
put_page()in__split_huge_pmd_locked()is there to balance thefolio_get()inmigrate_vma_split_unmapped_folio(). There are other points wheresplit_huge_pmd_locked()is called with freeze = true [1] and they don't get a reference before callingsplit_huge_pmd.I think the
folio_put()in__split_huge_pmd_locked()freeze = true case is there as migration entries are being installed?Quoted 9 lines
Quoted 3 lines
[1] https://elixir.bootlin.com/linux/v6.19.3/source/mm/rmap.c#L2334
Yes normally you want to drop the reference when installing migration entries but in this context you have already done the collecting for the THP folio and you want to balance with the
folio_get()theput_page()to keep the refs unchanged. Is that right Balbir?--Mika
Hi Mika,
You are right, This patch is wrong. I tried the below diff to force
folio_split_unmappedto return-EAGAIN. I rantools/testing/selftests/mm/hmm-tests-r hmm.hmm_device_private.migrate_anon_huge_err to trigger the path forfolio_split_unmapped.mm/huge_memory.c +2
-0@@ -4140,6 +4140,8 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)if (folio_expected_ref_count (folio) !=folio_ref_count (folio) -1 )return -EAGAIN ; +return -EAGAIN ; +local_irq_disable (); ret = __folio_freeze_and_split_unmapped(folio, new_order, &folio->page, NULL, NULL, false, NULL, SPLIT_TYPE_UNIFORM,I inserted a lot of traces to keep track of refcounts [1]. Without this patch, I get
....hmm-tests-129 [000] ..... 1.476233: __migrate_device_pages: SPLIT_UNMAPPED: folio=ffc536e2c4100000 refcount=0 AFTER error NO folio_put hmm-tests-129 [000] ..... 1.476234: __migrate_device_pages: PAGES: split FAILED folio=ffc536e2c4100000 refcount=0 hmm-tests-129 [000] ..... 1.476236: __migrate_device_finalize: FINALIZE[0]: src=ffc536e2c4100000 dst=ffc536e2c4100000 src==dst=1 refcount_src=1 mapcount_src=0 order_src=0 migrate=0 BEFORE remove_migration_ptes hmm-tests-129 [000] ..... 1.476237: __migrate_device_finalize: FINALIZE[0]: src=ffc536e2c4100000 refcount=1 mapcount=0 AFTER remove_migration_ptes hmm-tests-129 [000] ..... 1.476237: __migrate_device_finalize: FINALIZE[0]: src=ffc536e2c4100000 refcount=0 AFTER folio_put(src)i.e. refcount = 512, which is correct as
split_huge_pmd_addresswas successful. Full output is at [2].With this patch, I get:
BUG: Bad rss-counter state mm:00000000cfe88d5e type:MM_FILEPAGES val:-511 Comm:bash Pid:63 BUG: Bad rss-counter state mm:00000000cfe88d5e type:MM_ANONPAGES val:511 Comm:bash Pid:63 ... hmm-tests-129 [000] ..... 1.468315: __migrate_device_pages: SPLIT_UNMAPPED: folio=ffed210c840f0000 refcount=1 AFTER error folio_put FIX PRESENT hmm-tests-129 [000] ..... 1.468315: __migrate_device_pages: PAGES: split FAILED folio=ffed210c840f0000 refcount=1 hmm-tests-129 [000] ..... 1.468318: __migrate_device_finalize: FINALIZE[0]: src=ffed210c840f0000 dst=ffed210c840f0000 src==dst=1 refcount_src=1 mapcount_src=0 order_src=9 migrate=0 BEFORE remove_migration_ptes hmm-tests-129 [000] ..... 1.468357: __migrate_device_finalize: FINALIZE[0]: src=ffed210c840f0000 refcount=513 mapcount=512 AFTER remove_migration_ptes hmm-tests-129 [000] ..... 1.468357: __migrate_device_finalize: FINALIZE[0]: src=ffed210c840f0000 refcount=512 AFTER folio_put(src)
Quoted 3 lines
refcount=0 means the folio would be freed which is not correct. The full output is at [3].
Thank you for clearing this up!
Thank you for doing the investigation. Can you send a patch to add a comment in migrate_vma_split_unmapped_folio() about this to avoid the confusion in the future?
Quoted 5 lines
[1] https://gist.github.com/uarif1/65e1e816af7aa0ae38dd6ec64d62a993
[2] https://gist.github.com/uarif1/79ea9500667daa4e2ef09cb5d308f041
[3] https://gist.github.com/uarif1/8a35a6c65ba8b3a1c1dfe72dc30e821d
Best Regards,
Yan, Zi
Usama Arif
reviewerQuoted 193 lines · Zi Yan
Quoted 191 lines · Usama Arif
Quoted 136 lines · Mika Penttilä
Quoted 134 lines · Usama Arif
Quoted 117 lines · Mika Penttilä
Hi!
Quoted 113 lines · Usama Arif
Quoted 54 lines · Balbir Singh
Quoted 53 lines · Zi Yan
Quoted 51 lines · Balbir Singh
Quoted 36 lines · Zi Yan
Quoted 34 lines · Usama Arif
From: Usama Arif <[email protected]>
migrate_vma_split_unmapped_folio()takes an extra reference viafolio_get()before callingfolio_split_unmapped(). On success, the split consumes this reference:__folio_freeze_and_split_unmapped()expects the +1 in itsfolio_ref_freeze()check, and distributes it across the resulting sub-folios viafolio_ref_unfreeze(...+1), which are later balanced byfolio_put()calls in__migrate_device_finalize().If
folio_split_unmapped()fails (e.g., unexpected pinning returns-EAGAIN), the function returns without callingfolio_put(). The extra reference is never released.Add the missing
folio_put()on the error path.Fixes 4265d67e405a4 ("mm/migrate_device: add THP splitting during migration")Closes https://lore.kernel.org/all/CAA1CXcDyqPPwf_-W7B+PFQtL8HdoJGCEqVsVxq7DhOUB=L4PQA@mail.gmail.com/ open hereReported-by Nico Pache <[email protected]>Signed-off-by Usama Arif <[email protected]>--- mm/migrate_device.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/migrate_device.c +2
-1@@ -917,8 +917,10 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,folio_get (folio);split_huge_pmd_address (migrate->vma, addr, true); ret =folio_split_unmapped (folio,0 ); -if (ret) +if (ret) { +folio_put (folio);Quoted 9 lines
Quoted 7 lines
mm/migrate_device.c +1
-0return ret; + } migrate->src[idx] &= ~MIGRATE_PFN_COMPOUND ; flags = migrate->src[idx] & ((1UL <<MIGRATE_PFN_SHIFT ) -1 ); pfn = migrate->src[idx] >>MIGRATE_PFN_SHIFT ;Signature · 1 line
2.47.3Add Balbir, who wrote the code, to comment on this.
Thanks Zi!
Just wondering if there is a reproducer for the issue and how the fix was tested? I expect
migrate_vma_finalize()to be called for folios, even when split failed and drop the lock.Quoted 12 lines
Quoted 4 lines
Does
migrate_vma_finalize()dofolio_put()for failed-to-split folios? If so, how does it distinguish between split folios and failed-to-split folios? By comparing source and destination folio orders?We reset the
MIGRATE_PFN_MIGRATEflag for failing to migrate pfns. We do afolio_puton the src in finalize, if it is split then on all the split folios as well.Quoted 5 lines
What we see from
migrate_vma_split_unmapped_folio()is that it adds a refcount for all input folios, but only drops a refcount for the split folio. Isn’t it cause failed-to-split folios to have additional refcount?Hello!
Thanks for reviewing everyone. So its very difficult to create a reproducer I think the extra reference would need to appear after
migrate_device_unmap()but beforefolio_split_unmapped()inmigrate_vma_pages()? That's hard to trigger reliably from userspace.The fix came about when Nico indicated there might be an issue if
split_huge_pmd_addressfails in my patch [1].Below is my understanding of how refcounting is working over here step by step. I might very well be wrong on this, and the refcounting is a bit all over the place and I might miss a reference change somewhere so would really appreciate if someone can confirm this!
1.
migrate_vma_collect_huge_pmd():
a)folio_get(folio)-> +1 (collect reference)
2.migrate_device_unmap():
a)folio_isolate_lru()-> +1 (isolation reference)
b)folio_put()-> -1 (drops the collect reference)Without this patch fix:
3.
migrate_vma_split_unmapped_folio():
a)folio_get(folio)-> +1 (split reference)
b)folio_split_unmapped()-> fails
c) Returns error — withoutfolio_put()which is the fix
4. Caller inmigrate_vma_pages(): clearsMIGRATE_PFN_MIGRATE|MIGRATE_PFN_COMPOUND
5.__migrate_device_finalize(): sees !(src_pfns[i] &MIGRATE_PFN_MIGRATE), restores the folio:
a)remove_migration_ptes(src, src)— re-establishes user PTEs
b)folio_unlock(src)
c)folio_put(src)-> -1 (drops the isolation reference)The split reference in 3.a is never released and the folio has a permanently elevated refcount. Unless I missed a
folio_putsomewhere for the refcount increase infolio_isolate_lru()(2.b)?Please let me know if this makes sense!
[1] https://lore.kernel.org/all/CAA1CXcDyqPPwf_-W7B+PFQtL8HdoJGCEqVsVxq7DhOUB=L4PQA@mail.gmail.com/
Quoted 4 lines
Thanks! Yes, the patch makes sense
Acked-by Balbir Singh <[email protected]>Quoted 8 lines
Quoted 1 line
Quoted 1 line
Balbir
I remember stumbling on this while ago also. The
folio_get()inmigrate_vma_split_unmapped_folio()is balanced withput_page()in__split_huge_pmd_locked()(freeze = true), can't fail for device pages. Folios at this point are unmapped but have 1 refcount from "collecting".
Afterfolio_split_unmapped()the refcount(s) is still 1.So it seems the code is good as is? A comment though would be good for the extra
folio_get..hmm I dont think the
put_page()in__split_huge_pmd_locked()is there to balance thefolio_get()inmigrate_vma_split_unmapped_folio(). There are other points wheresplit_huge_pmd_locked()is called with freeze = true [1] and they don't get a reference before callingsplit_huge_pmd.I think the
folio_put()in__split_huge_pmd_locked()freeze = true case is there as migration entries are being installed?Quoted 9 lines
Quoted 3 lines
[1] https://elixir.bootlin.com/linux/v6.19.3/source/mm/rmap.c#L2334
Yes normally you want to drop the reference when installing migration entries but in this context you have already done the collecting for the THP folio and you want to balance with the
folio_get()theput_page()to keep the refs unchanged. Is that right Balbir?--Mika
Hi Mika,
You are right, This patch is wrong. I tried the below diff to force
folio_split_unmappedto return-EAGAIN. I rantools/testing/selftests/mm/hmm-tests-r hmm.hmm_device_private.migrate_anon_huge_err to trigger the path forfolio_split_unmapped.mm/huge_memory.c +2
-0@@ -4140,6 +4140,8 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)if (folio_expected_ref_count (folio) !=folio_ref_count (folio) -1 )return -EAGAIN ; +return -EAGAIN ; +local_irq_disable (); ret = __folio_freeze_and_split_unmapped(folio, new_order, &folio->page, NULL, NULL, false, NULL, SPLIT_TYPE_UNIFORM,I inserted a lot of traces to keep track of refcounts [1]. Without this patch, I get
....hmm-tests-129 [000] ..... 1.476233: __migrate_device_pages: SPLIT_UNMAPPED: folio=ffc536e2c4100000 refcount=0 AFTER error NO folio_put hmm-tests-129 [000] ..... 1.476234: __migrate_device_pages: PAGES: split FAILED folio=ffc536e2c4100000 refcount=0 hmm-tests-129 [000] ..... 1.476236: __migrate_device_finalize: FINALIZE[0]: src=ffc536e2c4100000 dst=ffc536e2c4100000 src==dst=1 refcount_src=1 mapcount_src=0 order_src=0 migrate=0 BEFORE remove_migration_ptes hmm-tests-129 [000] ..... 1.476237: __migrate_device_finalize: FINALIZE[0]: src=ffc536e2c4100000 refcount=1 mapcount=0 AFTER remove_migration_ptes hmm-tests-129 [000] ..... 1.476237: __migrate_device_finalize: FINALIZE[0]: src=ffc536e2c4100000 refcount=0 AFTER folio_put(src)i.e. refcount = 512, which is correct as
split_huge_pmd_addresswas successful. Full output is at [2].With this patch, I get:
BUG: Bad rss-counter state mm:00000000cfe88d5e type:MM_FILEPAGES val:-511 Comm:bash Pid:63 BUG: Bad rss-counter state mm:00000000cfe88d5e type:MM_ANONPAGES val:511 Comm:bash Pid:63 ... hmm-tests-129 [000] ..... 1.468315: __migrate_device_pages: SPLIT_UNMAPPED: folio=ffed210c840f0000 refcount=1 AFTER error folio_put FIX PRESENT hmm-tests-129 [000] ..... 1.468315: __migrate_device_pages: PAGES: split FAILED folio=ffed210c840f0000 refcount=1 hmm-tests-129 [000] ..... 1.468318: __migrate_device_finalize: FINALIZE[0]: src=ffed210c840f0000 dst=ffed210c840f0000 src==dst=1 refcount_src=1 mapcount_src=0 order_src=9 migrate=0 BEFORE remove_migration_ptes hmm-tests-129 [000] ..... 1.468357: __migrate_device_finalize: FINALIZE[0]: src=ffed210c840f0000 refcount=513 mapcount=512 AFTER remove_migration_ptes hmm-tests-129 [000] ..... 1.468357: __migrate_device_finalize: FINALIZE[0]: src=ffed210c840f0000 refcount=512 AFTER folio_put(src)
Quoted 8 lines
Quoted 3 lines
refcount=0 means the folio would be freed which is not correct. The full output is at [3].
Thank you for clearing this up!
Thank you for doing the investigation. Can you send a patch to add a comment in
migrate_vma_split_unmapped_folio()about this to avoid the confusion in the future?
Yeah this was really confusing.
Does something like below look good?
mm/migrate_device.c +5 -0
@@ -910,6 +910,11 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,folio_get (folio);split_huge_pmd_address (migrate->vma, addr, true); +/* +* split_huge_pmd_address consumes the folio_get reference above. +* Therefore no folio_put is needed on the folio_split_unmapped +* error path. +*/ ret =folio_split_unmapped (folio,0 );if (ret)return ret;
Quoted 9 lines
Quoted 5 lines
[1] https://gist.github.com/uarif1/65e1e816af7aa0ae38dd6ec64d62a993
[2] https://gist.github.com/uarif1/79ea9500667daa4e2ef09cb5d308f041
[3] https://gist.github.com/uarif1/8a35a6c65ba8b3a1c1dfe72dc30e821dBest Regards,
Yan, Zi
Zi Yan
reviewerQuoted 213 lines · Usama Arif
Quoted 193 lines · Zi Yan
Quoted 191 lines · Usama Arif
Quoted 136 lines · Mika Penttilä
Quoted 134 lines · Usama Arif
Quoted 117 lines · Mika Penttilä
Hi!
Quoted 113 lines · Usama Arif
Quoted 54 lines · Balbir Singh
Quoted 53 lines · Zi Yan
Quoted 51 lines · Balbir Singh
Quoted 36 lines · Zi Yan
Quoted 34 lines · Usama Arif
From: Usama Arif <[email protected]>
migrate_vma_split_unmapped_folio()takes an extra reference viafolio_get()before callingfolio_split_unmapped(). On success, the split consumes this reference:__folio_freeze_and_split_unmapped()expects the +1 in itsfolio_ref_freeze()check, and distributes it across the resulting sub-folios viafolio_ref_unfreeze(...+1), which are later balanced byfolio_put()calls in__migrate_device_finalize().If
folio_split_unmapped()fails (e.g., unexpected pinning returns-EAGAIN), the function returns without callingfolio_put(). The extra reference is never released.Add the missing
folio_put()on the error path.Fixes 4265d67e405a4 ("mm/migrate_device: add THP splitting during migration")Closes https://lore.kernel.org/all/CAA1CXcDyqPPwf_-W7B+PFQtL8HdoJGCEqVsVxq7DhOUB=L4PQA@mail.gmail.com/ open hereReported-by Nico Pache <[email protected]>Signed-off-by Usama Arif <[email protected]>--- mm/migrate_device.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/migrate_device.c +2
-1@@ -917,8 +917,10 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,folio_get (folio);split_huge_pmd_address (migrate->vma, addr, true); ret =folio_split_unmapped (folio,0 ); -if (ret) +if (ret) { +folio_put (folio);Quoted 9 lines
Quoted 7 lines
mm/migrate_device.c +1
-0return ret; + } migrate->src[idx] &= ~MIGRATE_PFN_COMPOUND ; flags = migrate->src[idx] & ((1UL <<MIGRATE_PFN_SHIFT ) -1 ); pfn = migrate->src[idx] >>MIGRATE_PFN_SHIFT ;Signature · 1 line
2.47.3Add Balbir, who wrote the code, to comment on this.
Thanks Zi!
Just wondering if there is a reproducer for the issue and how the fix was tested? I expect
migrate_vma_finalize()to be called for folios, even when split failed and drop the lock.Quoted 12 lines
Quoted 4 lines
Does
migrate_vma_finalize()dofolio_put()for failed-to-split folios? If so, how does it distinguish between split folios and failed-to-split folios? By comparing source and destination folio orders?We reset the
MIGRATE_PFN_MIGRATEflag for failing to migrate pfns. We do afolio_puton the src in finalize, if it is split then on all the split folios as well.Quoted 5 lines
What we see from
migrate_vma_split_unmapped_folio()is that it adds a refcount for all input folios, but only drops a refcount for the split folio. Isn’t it cause failed-to-split folios to have additional refcount?Hello!
Thanks for reviewing everyone. So its very difficult to create a reproducer I think the extra reference would need to appear after
migrate_device_unmap()but beforefolio_split_unmapped()inmigrate_vma_pages()? That's hard to trigger reliably from userspace.The fix came about when Nico indicated there might be an issue if
split_huge_pmd_addressfails in my patch [1].Below is my understanding of how refcounting is working over here step by step. I might very well be wrong on this, and the refcounting is a bit all over the place and I might miss a reference change somewhere so would really appreciate if someone can confirm this!
1.
migrate_vma_collect_huge_pmd():
a)folio_get(folio)-> +1 (collect reference)
2.migrate_device_unmap():
a)folio_isolate_lru()-> +1 (isolation reference)
b)folio_put()-> -1 (drops the collect reference)Without this patch fix:
3.
migrate_vma_split_unmapped_folio():
a)folio_get(folio)-> +1 (split reference)
b)folio_split_unmapped()-> fails
c) Returns error — withoutfolio_put()which is the fix
4. Caller inmigrate_vma_pages(): clearsMIGRATE_PFN_MIGRATE|MIGRATE_PFN_COMPOUND
5.__migrate_device_finalize(): sees !(src_pfns[i] &MIGRATE_PFN_MIGRATE), restores the folio:
a)remove_migration_ptes(src, src)— re-establishes user PTEs
b)folio_unlock(src)
c)folio_put(src)-> -1 (drops the isolation reference)The split reference in 3.a is never released and the folio has a permanently elevated refcount. Unless I missed a
folio_putsomewhere for the refcount increase infolio_isolate_lru()(2.b)?Please let me know if this makes sense!
[1] https://lore.kernel.org/all/CAA1CXcDyqPPwf_-W7B+PFQtL8HdoJGCEqVsVxq7DhOUB=L4PQA@mail.gmail.com/
Quoted 4 lines
Thanks! Yes, the patch makes sense
Acked-by Balbir Singh <[email protected]>Quoted 8 lines
Quoted 1 line
Quoted 1 line
Balbir
I remember stumbling on this while ago also. The
folio_get()inmigrate_vma_split_unmapped_folio()is balanced withput_page()in__split_huge_pmd_locked()(freeze = true), can't fail for device pages. Folios at this point are unmapped but have 1 refcount from "collecting".
Afterfolio_split_unmapped()the refcount(s) is still 1.So it seems the code is good as is? A comment though would be good for the extra
folio_get..hmm I dont think the
put_page()in__split_huge_pmd_locked()is there to balance thefolio_get()inmigrate_vma_split_unmapped_folio(). There are other points wheresplit_huge_pmd_locked()is called with freeze = true [1] and they don't get a reference before callingsplit_huge_pmd.I think the
folio_put()in__split_huge_pmd_locked()freeze = true case is there as migration entries are being installed?Quoted 9 lines
Quoted 3 lines
[1] https://elixir.bootlin.com/linux/v6.19.3/source/mm/rmap.c#L2334
Yes normally you want to drop the reference when installing migration entries but in this context you have already done the collecting for the THP folio and you want to balance with the
folio_get()theput_page()to keep the refs unchanged. Is that right Balbir?--Mika
Hi Mika,
You are right, This patch is wrong. I tried the below diff to force
folio_split_unmappedto return-EAGAIN. I rantools/testing/selftests/mm/hmm-tests-r hmm.hmm_device_private.migrate_anon_huge_err to trigger the path forfolio_split_unmapped.mm/huge_memory.c +2
-0@@ -4140,6 +4140,8 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)if (folio_expected_ref_count (folio) !=folio_ref_count (folio) -1 )return -EAGAIN ; +return -EAGAIN ; +local_irq_disable (); ret = __folio_freeze_and_split_unmapped(folio, new_order, &folio->page, NULL, NULL, false, NULL, SPLIT_TYPE_UNIFORM,I inserted a lot of traces to keep track of refcounts [1]. Without this patch, I get
....hmm-tests-129 [000] ..... 1.476233: __migrate_device_pages: SPLIT_UNMAPPED: folio=ffc536e2c4100000 refcount=0 AFTER error NO folio_put hmm-tests-129 [000] ..... 1.476234: __migrate_device_pages: PAGES: split FAILED folio=ffc536e2c4100000 refcount=0 hmm-tests-129 [000] ..... 1.476236: __migrate_device_finalize: FINALIZE[0]: src=ffc536e2c4100000 dst=ffc536e2c4100000 src==dst=1 refcount_src=1 mapcount_src=0 order_src=0 migrate=0 BEFORE remove_migration_ptes hmm-tests-129 [000] ..... 1.476237: __migrate_device_finalize: FINALIZE[0]: src=ffc536e2c4100000 refcount=1 mapcount=0 AFTER remove_migration_ptes hmm-tests-129 [000] ..... 1.476237: __migrate_device_finalize: FINALIZE[0]: src=ffc536e2c4100000 refcount=0 AFTER folio_put(src)i.e. refcount = 512, which is correct as
split_huge_pmd_addresswas successful. Full output is at [2].With this patch, I get:
BUG: Bad rss-counter state mm:00000000cfe88d5e type:MM_FILEPAGES val:-511 Comm:bash Pid:63 BUG: Bad rss-counter state mm:00000000cfe88d5e type:MM_ANONPAGES val:511 Comm:bash Pid:63 ... hmm-tests-129 [000] ..... 1.468315: __migrate_device_pages: SPLIT_UNMAPPED: folio=ffed210c840f0000 refcount=1 AFTER error folio_put FIX PRESENT hmm-tests-129 [000] ..... 1.468315: __migrate_device_pages: PAGES: split FAILED folio=ffed210c840f0000 refcount=1 hmm-tests-129 [000] ..... 1.468318: __migrate_device_finalize: FINALIZE[0]: src=ffed210c840f0000 dst=ffed210c840f0000 src==dst=1 refcount_src=1 mapcount_src=0 order_src=9 migrate=0 BEFORE remove_migration_ptes hmm-tests-129 [000] ..... 1.468357: __migrate_device_finalize: FINALIZE[0]: src=ffed210c840f0000 refcount=513 mapcount=512 AFTER remove_migration_ptes hmm-tests-129 [000] ..... 1.468357: __migrate_device_finalize: FINALIZE[0]: src=ffed210c840f0000 refcount=512 AFTER folio_put(src)Quoted 8 lines
Quoted 3 lines
refcount=0 means the folio would be freed which is not correct. The full output is at [3].
Thank you for clearing this up!
Thank you for doing the investigation. Can you send a patch to add a comment in
migrate_vma_split_unmapped_folio()about this to avoid the confusion in the future?Yeah this was really confusing.
Does something like below look good?
mm/migrate_device.c +0
-0@@ -910,6 +910,11 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,
mm/migrate_device.c static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate, · line 920
Quoted 10 lines
mm/migrate_device.c +5
-0folio_get (folio);split_huge_pmd_address (migrate->vma, addr, true); +/* +* split_huge_pmd_address consumes the folio_get reference above. +* Therefore no folio_put is needed on the folio_split_unmapped +* error path. +*/ ret =folio_split_unmapped (folio,0 );if (ret)return ret;
I do not think there is a need to explain why there is no folio_put() below. How about below?
1. it makes sure the folio has the right ref count,
2. it explains folio_get() is for split_huge_pmd_address() instead of folio_split_unmapped().
mm/migrate_device.c +6 -0
@@ -914,8 +914,14 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,unsigned long flags;int ret =0 ; +VM_WARN_ON_ONCE (folio_ref_count (folio) ==1 ); +/* +* take a reference, since split_huge_pmd_address() with freeze = true +* drops a reference at the end. +*/ folio_get (folio);split_huge_pmd_address (migrate->vma, addr, true); + ret =folio_split_unmapped (folio,0 );if (ret)return ret;
Quoted 10 lines
Quoted 10 lines
Quoted 6 lines
[1] https://gist.github.com/uarif1/65e1e816af7aa0ae38dd6ec64d62a993
[2] https://gist.github.com/uarif1/79ea9500667daa4e2ef09cb5d308f041
[3] https://gist.github.com/uarif1/8a35a6c65ba8b3a1c1dfe72dc30e821dBest Regards,
Yan, Zi
Best Regards,
Yan, Zi
Usama Arif
reviewerQuoted 14 lines
I do not think there is a need to explain why there is no
folio_put()below. How about below?1. it makes sure the folio has the right ref count,
2. it explainsfolio_get()is forsplit_huge_pmd_address()instead offolio_split_unmapped().mm/migrate_device.c +0
-0
mm/migrate_device.c static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate, · line 917
Quoted 4 lines
mm/migrate_device.c +1
-0@@ -914,8 +914,14 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,unsigned long flags;int ret =0 ; +VM_WARN_ON_ONCE (folio_ref_count (folio) ==1 );
Thanks! I have sent it in [1]. One thing is, I think you mean over here folio_ref_count(folio) != 1, right? Thats what the patch has.
Quoted 26 lines
mm/migrate_device.c +5
-0+/* +* take a reference, since split_huge_pmd_address() with freeze = true +* drops a reference at the end. +*/ folio_get (folio);split_huge_pmd_address (migrate->vma, addr, true); + ret =folio_split_unmapped (folio,0 );if (ret)return ret;Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
[1] https://gist.github.com/uarif1/65e1e816af7aa0ae38dd6ec64d62a993
[2] https://gist.github.com/uarif1/79ea9500667daa4e2ef09cb5d308f041
[3] https://gist.github.com/uarif1/8a35a6c65ba8b3a1c1dfe72dc30e821dBest Regards,
Yan, ZiBest Regards,
Yan, Zi
Quoted 216 lines · Usama Arif
Quoted 193 lines · Zi Yan
Quoted 191 lines · Usama Arif
Quoted 136 lines · Mika Penttilä
Quoted 134 lines · Usama Arif
Quoted 117 lines · Mika Penttilä
Hi!
Quoted 113 lines · Usama Arif
Quoted 54 lines · Balbir Singh
Quoted 53 lines · Zi Yan
Quoted 51 lines · Balbir Singh
Quoted 36 lines · Zi Yan
Quoted 34 lines · Usama Arif
From: Usama Arif <[email protected]>
migrate_vma_split_unmapped_folio()takes an extra reference viafolio_get()before callingfolio_split_unmapped(). On success, the split consumes this reference:__folio_freeze_and_split_unmapped()expects the +1 in itsfolio_ref_freeze()check, and distributes it across the resulting sub-folios viafolio_ref_unfreeze(...+1), which are later balanced byfolio_put()calls in__migrate_device_finalize().If
folio_split_unmapped()fails (e.g., unexpected pinning returns-EAGAIN), the function returns without callingfolio_put(). The extra reference is never released.Add the missing
folio_put()on the error path.Fixes 4265d67e405a4 ("mm/migrate_device: add THP splitting during migration")Closes https://lore.kernel.org/all/CAA1CXcDyqPPwf_-W7B+PFQtL8HdoJGCEqVsVxq7DhOUB=L4PQA@mail.gmail.com/ open hereReported-by Nico Pache <[email protected]>Signed-off-by Usama Arif <[email protected]>--- mm/migrate_device.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/migrate_device.c +2
-1@@ -917,8 +917,10 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,folio_get (folio);split_huge_pmd_address (migrate->vma, addr, true); ret =folio_split_unmapped (folio,0 ); -if (ret) +if (ret) { +folio_put (folio);Quoted 9 lines
Quoted 7 lines
mm/migrate_device.c +1
-0return ret; + } migrate->src[idx] &= ~MIGRATE_PFN_COMPOUND ; flags = migrate->src[idx] & ((1UL <<MIGRATE_PFN_SHIFT ) -1 ); pfn = migrate->src[idx] >>MIGRATE_PFN_SHIFT ;Signature · 1 line
2.47.3Add Balbir, who wrote the code, to comment on this.
Thanks Zi!
Just wondering if there is a reproducer for the issue and how the fix was tested? I expect
migrate_vma_finalize()to be called for folios, even when split failed and drop the lock.Quoted 12 lines
Quoted 4 lines
Does
migrate_vma_finalize()dofolio_put()for failed-to-split folios? If so, how does it distinguish between split folios and failed-to-split folios? By comparing source and destination folio orders?We reset the
MIGRATE_PFN_MIGRATEflag for failing to migrate pfns. We do afolio_puton the src in finalize, if it is split then on all the split folios as well.Quoted 5 lines
What we see from
migrate_vma_split_unmapped_folio()is that it adds a refcount for all input folios, but only drops a refcount for the split folio. Isn’t it cause failed-to-split folios to have additional refcount?Hello!
Thanks for reviewing everyone. So its very difficult to create a reproducer I think the extra reference would need to appear after
migrate_device_unmap()but beforefolio_split_unmapped()inmigrate_vma_pages()? That's hard to trigger reliably from userspace.The fix came about when Nico indicated there might be an issue if
split_huge_pmd_addressfails in my patch [1].Below is my understanding of how refcounting is working over here step by step. I might very well be wrong on this, and the refcounting is a bit all over the place and I might miss a reference change somewhere so would really appreciate if someone can confirm this!
1.
migrate_vma_collect_huge_pmd():
a)folio_get(folio)-> +1 (collect reference)
2.migrate_device_unmap():
a)folio_isolate_lru()-> +1 (isolation reference)
b)folio_put()-> -1 (drops the collect reference)Without this patch fix:
3.
migrate_vma_split_unmapped_folio():
a)folio_get(folio)-> +1 (split reference)
b)folio_split_unmapped()-> fails
c) Returns error — withoutfolio_put()which is the fix
4. Caller inmigrate_vma_pages(): clearsMIGRATE_PFN_MIGRATE|MIGRATE_PFN_COMPOUND
5.__migrate_device_finalize(): sees !(src_pfns[i] &MIGRATE_PFN_MIGRATE), restores the folio:
a)remove_migration_ptes(src, src)— re-establishes user PTEs
b)folio_unlock(src)
c)folio_put(src)-> -1 (drops the isolation reference)The split reference in 3.a is never released and the folio has a permanently elevated refcount. Unless I missed a
folio_putsomewhere for the refcount increase infolio_isolate_lru()(2.b)?Please let me know if this makes sense!
[1] https://lore.kernel.org/all/CAA1CXcDyqPPwf_-W7B+PFQtL8HdoJGCEqVsVxq7DhOUB=L4PQA@mail.gmail.com/
Quoted 4 lines
Thanks! Yes, the patch makes sense
Acked-by Balbir Singh <[email protected]>Quoted 8 lines
Quoted 1 line
Quoted 1 line
Balbir
I remember stumbling on this while ago also. The
folio_get()inmigrate_vma_split_unmapped_folio()is balanced withput_page()in__split_huge_pmd_locked()(freeze = true), can't fail for device pages. Folios at this point are unmapped but have 1 refcount from "collecting".
Afterfolio_split_unmapped()the refcount(s) is still 1.So it seems the code is good as is? A comment though would be good for the extra
folio_get..hmm I dont think the
put_page()in__split_huge_pmd_locked()is there to balance thefolio_get()inmigrate_vma_split_unmapped_folio(). There are other points wheresplit_huge_pmd_locked()is called with freeze = true [1] and they don't get a reference before callingsplit_huge_pmd.I think the
folio_put()in__split_huge_pmd_locked()freeze = true case is there as migration entries are being installed?Quoted 9 lines
Quoted 3 lines
[1] https://elixir.bootlin.com/linux/v6.19.3/source/mm/rmap.c#L2334
Yes normally you want to drop the reference when installing migration entries but in this context you have already done the collecting for the THP folio and you want to balance with the
folio_get()theput_page()to keep the refs unchanged. Is that right Balbir?--Mika
Hi Mika,
You are right, This patch is wrong. I tried the below diff to force
folio_split_unmappedto return-EAGAIN. I rantools/testing/selftests/mm/hmm-tests-r hmm.hmm_device_private.migrate_anon_huge_err to trigger the path forfolio_split_unmapped.mm/huge_memory.c +2
-0@@ -4140,6 +4140,8 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)if (folio_expected_ref_count (folio) !=folio_ref_count (folio) -1 )return -EAGAIN ; +return -EAGAIN ; +local_irq_disable (); ret = __folio_freeze_and_split_unmapped(folio, new_order, &folio->page, NULL, NULL, false, NULL, SPLIT_TYPE_UNIFORM,I inserted a lot of traces to keep track of refcounts [1]. Without this patch, I get
....hmm-tests-129 [000] ..... 1.476233: __migrate_device_pages: SPLIT_UNMAPPED: folio=ffc536e2c4100000 refcount=0 AFTER error NO folio_put hmm-tests-129 [000] ..... 1.476234: __migrate_device_pages: PAGES: split FAILED folio=ffc536e2c4100000 refcount=0 hmm-tests-129 [000] ..... 1.476236: __migrate_device_finalize: FINALIZE[0]: src=ffc536e2c4100000 dst=ffc536e2c4100000 src==dst=1 refcount_src=1 mapcount_src=0 order_src=0 migrate=0 BEFORE remove_migration_ptes hmm-tests-129 [000] ..... 1.476237: __migrate_device_finalize: FINALIZE[0]: src=ffc536e2c4100000 refcount=1 mapcount=0 AFTER remove_migration_ptes hmm-tests-129 [000] ..... 1.476237: __migrate_device_finalize: FINALIZE[0]: src=ffc536e2c4100000 refcount=0 AFTER folio_put(src)i.e. refcount = 512, which is correct as
split_huge_pmd_addresswas successful. Full output is at [2].With this patch, I get:
BUG: Bad rss-counter state mm:00000000cfe88d5e type:MM_FILEPAGES val:-511 Comm:bash Pid:63 BUG: Bad rss-counter state mm:00000000cfe88d5e type:MM_ANONPAGES val:511 Comm:bash Pid:63 ... hmm-tests-129 [000] ..... 1.468315: __migrate_device_pages: SPLIT_UNMAPPED: folio=ffed210c840f0000 refcount=1 AFTER error folio_put FIX PRESENT hmm-tests-129 [000] ..... 1.468315: __migrate_device_pages: PAGES: split FAILED folio=ffed210c840f0000 refcount=1 hmm-tests-129 [000] ..... 1.468318: __migrate_device_finalize: FINALIZE[0]: src=ffed210c840f0000 dst=ffed210c840f0000 src==dst=1 refcount_src=1 mapcount_src=0 order_src=9 migrate=0 BEFORE remove_migration_ptes hmm-tests-129 [000] ..... 1.468357: __migrate_device_finalize: FINALIZE[0]: src=ffed210c840f0000 refcount=513 mapcount=512 AFTER remove_migration_ptes hmm-tests-129 [000] ..... 1.468357: __migrate_device_finalize: FINALIZE[0]: src=ffed210c840f0000 refcount=512 AFTER folio_put(src)Quoted 8 lines
Quoted 3 lines
refcount=0 means the folio would be freed which is not correct. The full output is at [3].
Thank you for clearing this up!
Thank you for doing the investigation. Can you send a patch to add a comment in
migrate_vma_split_unmapped_folio()about this to avoid the confusion in the future?Yeah this was really confusing.
Does something like below look good?
mm/migrate_device.c +0
-0@@ -910,6 +910,11 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,
mm/migrate_device.c static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate, · line 920
Quoted 10 lines
mm/migrate_device.c +5
-0folio_get (folio);split_huge_pmd_address (migrate->vma, addr, true); +/* +* split_huge_pmd_address consumes the folio_get reference above. +* Therefore no folio_put is needed on the folio_split_unmapped +* error path. +*/ ret =folio_split_unmapped (folio,0 );if (ret)return ret;
The comment should come above folio_get and state that
/*
* folio_get() is required because this reference is to be consumed
* by split_huge_pmd_address() for frozen entries
*/
Quoted 11 lines
Quoted 9 lines
Quoted 5 lines
[1] https://gist.github.com/uarif1/65e1e816af7aa0ae38dd6ec64d62a993
[2] https://gist.github.com/uarif1/79ea9500667daa4e2ef09cb5d308f041
[3] https://gist.github.com/uarif1/8a35a6c65ba8b3a1c1dfe72dc30e821dBest Regards,
Yan, Zi
@Usama have you figured out where the original issue you are seeing stems from?
Balbir
Usama Arif
reviewerQuoted 6 lines
@Usama have you figured out where the original issue you are seeing stems from?
There was no issue or bug report. split_huge_pmd_address currently cannot fail. I am making lazy page table allocation work [1], which would allow it to fail. One of the review comments was if I need a folio_put if split_huge_pmd_address fails. Its not needed if folio_split_unmapped fails, but is needed if split_huge_pmd_address fails [2]. Actually it would be very nice if you could review the patch in [1], as its modifying the code you wrote!
Thanks!
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
Quoted 3 lines
Balbir
Quoted 9 lines · On Wed, 4 Mar 2026 04:01:32 -0800 Usama Arif <[email protected]> wrote:
From: Usama Arif <[email protected]>
migrate_vma_split_unmapped_folio()takes an extra reference viafolio_get()before callingfolio_split_unmapped(). On success, the split consumes this reference:__folio_freeze_and_split_unmapped()expects the +1 in itsfolio_ref_freeze()check, and distributes it across the resulting sub-folios viafolio_ref_unfreeze(...+1), which are later balanced byfolio_put()calls in__migrate_device_finalize().
Quoted 5 lines
If
folio_split_unmapped()fails (e.g., unexpected pinning returns-EAGAIN), the function returns without callingfolio_put(). The extra reference is never released.Add the missing
folio_put()on the error path.
Agreed with Kiryl that maybe there is an opportunity to do additional cleanup. But as a fix for the issue, I think that this patch looks good to me. We can send a follow-up patch in the future if we want to clean this area up : -)
Quoted 26 lines
Fixes 4265d67e405a4 ("mm/migrate_device: add THP splitting during migration")Closes https://lore.kernel.org/all/CAA1CXcDyqPPwf_-W7B+PFQtL8HdoJGCEqVsVxq7DhOUB=L4PQA@mail.gmail.com/ open hereReported-by Nico Pache <[email protected]>Signed-off-by Usama Arif <[email protected]>--- mm/migrate_device.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/migrate_device.c +3
-1@@ -917,8 +917,10 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,folio_get (folio);split_huge_pmd_address (migrate->vma, addr, true); ret =folio_split_unmapped (folio,0 ); -if (ret) +if (ret) { +folio_put (folio);return ret; + } migrate->src[idx] &= ~MIGRATE_PFN_COMPOUND ; flags = migrate->src[idx] & ((1UL <<MIGRATE_PFN_SHIFT ) -1 ); pfn = migrate->src[idx] >>MIGRATE_PFN_SHIFT ;Signature · 1 line
2.47.3