mm/migrate_device: Clear stale mapping after freeing swapcache
Subsystems (2)
PATCH v4 1/1 mm/migrate_device: Clear stale mapping after freeing swapcache
__migrate_device_pages() reads the folio mapping before calling folio_free_swap(). When folio_free_swap() succeeds, the folio is removed from the swap cache, but the saved mapping still points to swap_space.
Passing the stale mapping to folio_migrate_mapping() makes it use the mapped-folio path for a folio that is no longer in swapcache. It can then operate on swap_space.i_pages with invalid reference accounting, eventually triggering a folio reference count BUG.
After a successful split, nr still contains the number of pages in the original large folio, although each resulting page is now a separate order-0 folio. Reset nr to 1 so each split folio is processed separately, including its own swapcache removal and mapping lookup.
Refresh the saved mapping after folio_free_swap() so the current folio state is used during migration.
v2:
- Refresh the mapping using folio_mapping(), as suggested by Zi Yan.
v3:
- Reset nr to 1 after a successful split so each resulting folio is processed independently, as suggested by Zi Yan.
v4:
- Re-read each source folio's mapping immediately before folio_migrate_mapping(), as suggested by Balbir Singh.
- Add a warning to validate the post-split order-0 invariant, as suggested by Balbir Singh.
--- mm/migrate_device.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
mm/migrate_device.c +13 -0
@@ -1182,6 +1182,13 @@ static void __migrate_device_pages(unsigned long *src_pfns, MIGRATE_PFN_COMPOUND); goto next; } + +/* +* reset nr so that only first after-split folio +* is processed below +*/ +VM_WARN_ON_ONCE (folio_test_large (folio)); + nr =1 ; }else if ((src_pfns[i] &MIGRATE_PFN_MIGRATE ) && (dst_pfns[i] &MIGRATE_PFN_COMPOUND ) && !(src_pfns[i] &MIGRATE_PFN_COMPOUND )) {
@@ -1221,6 +1228,12 @@ static void __migrate_device_pages(unsigned long *src_pfns, folio =page_folio (migrate_pfn_to_page (src_pfns[i+j])); newfolio =page_folio (migrate_pfn_to_page (dst_pfns[i+j])); +/* +* folio_free_swap() removed the folio from the swap +* cache. Refresh the saved mapping before migration. +*/ + mapping =folio_mapping (folio); + r =folio_migrate_mapping (mapping, newfolio, folio, extra_cnt);if (r) src_pfns[i+j] &= ~MIGRATE_PFN_MIGRATE ;
Signature · 1 line
2.43.0
Quoted 43 lines · Arvind Yadav
__migrate_device_pages()reads the folio mapping before callingfolio_free_swap(). Whenfolio_free_swap()succeeds, the folio is removed from the swap cache, but the saved mapping still points toswap_space.Passing the stale mapping to
folio_migrate_mapping()makes it use the mapped-folio path for a folio that is no longer in swapcache. It can then operate on swap_space.i_pages with invalid reference accounting, eventually triggering a folio reference count BUG.After a successful split, nr still contains the number of pages in the original large folio, although each resulting page is now a separate order-0 folio. Reset nr to 1 so each split folio is processed separately, including its own swapcache removal and mapping lookup.
Refresh the saved mapping after
folio_free_swap()so the current folio state is used during migration.v2:
- Refresh the mapping usingfolio_mapping(), as suggested by Zi Yan.v3:
- Reset nr to 1 after a successful split so each resulting folio is processed independently, as suggested by Zi Yan.v4:
- Re-read each source folio's mapping immediately beforefolio_migrate_mapping(), as suggested by Balbir Singh.
- Add a warning to validate the post-split order-0 invariant, as suggested by Balbir Singh.Fixes df263d9a7dff ("mm/migrate_device: try to handle swapcache pages")Cc Andrew Morton <[email protected]>Cc David Hildenbrand <[email protected]>Cc Matthew Brost <[email protected]>Cc Joshua Hahn <[email protected]>Cc Rakie Kim <[email protected]>Cc Byungchul Park <[email protected]>Cc Gregory Price <[email protected]>Cc Ying Huang <[email protected]>Cc Alistair Popple <[email protected]>Reviewed-by Zi Yan <[email protected]>Reviewed-by Balbir Singh <[email protected]>Signed-off-by Arvind Yadav <[email protected]>
Quoted 4 lines
--- mm/migrate_device.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
David Hildenbrand (Arm)
maintainerQuoted 52 lines · Arvind Yadav
__migrate_device_pages()reads the folio mapping before callingfolio_free_swap(). Whenfolio_free_swap()succeeds, the folio is removed from the swap cache, but the saved mapping still points toswap_space.Passing the stale mapping to
folio_migrate_mapping()makes it use the mapped-folio path for a folio that is no longer in swapcache. It can then operate on swap_space.i_pages with invalid reference accounting, eventually triggering a folio reference count BUG.After a successful split, nr still contains the number of pages in the original large folio, although each resulting page is now a separate order-0 folio. Reset nr to 1 so each split folio is processed separately, including its own swapcache removal and mapping lookup.
Refresh the saved mapping after
folio_free_swap()so the current folio state is used during migration.v2:
- Refresh the mapping usingfolio_mapping(), as suggested by Zi Yan.v3:
- Reset nr to 1 after a successful split so each resulting folio is processed independently, as suggested by Zi Yan.v4:
- Re-read each source folio's mapping immediately beforefolio_migrate_mapping(), as suggested by Balbir Singh.
- Add a warning to validate the post-split order-0 invariant, as suggested by Balbir Singh.Fixes df263d9a7dff ("mm/migrate_device: try to handle swapcache pages")Cc Andrew Morton <[email protected]>Cc David Hildenbrand <[email protected]>Cc Matthew Brost <[email protected]>Cc Joshua Hahn <[email protected]>Cc Rakie Kim <[email protected]>Cc Byungchul Park <[email protected]>Cc Gregory Price <[email protected]>Cc Ying Huang <[email protected]>Cc Alistair Popple <[email protected]>Reviewed-by Zi Yan <[email protected]>Reviewed-by Balbir Singh <[email protected]>Signed-off-by Arvind Yadav <[email protected]>--- mm/migrate_device.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)mm/migrate_device.c +0
-0
mm/migrate_device.c static void __migrate_device_pages(unsigned long *src_pfns, · line 1191
Quoted 10 lines
mm/migrate_device.c +7
-0@@ -1182,6 +1182,13 @@ static void __migrate_device_pages(unsigned long *src_pfns, MIGRATE_PFN_COMPOUND); goto next; } + +/* +* reset nr so that only first after-split folio +* is processed below +*/ +VM_WARN_ON_ONCE (folio_test_large (folio)); + nr =1 ;
This function is surely a beauty. (had to rephrase that sentence 3 times ;) )
migrate_vma_split_unmapped_folio() modifies the src_pfns() entries on success. (and somehow assumes that it's always a THP, what? After a MIGRATE_PFN_COMPOUND value is set? What? Why the "nr = 1 << folio_order(folio);" in the caller).
If we ended up modifying the current entry in such a way, shouldn't we just have
retry: label and restart at the very top of the function, where we just naturally re-read the entry/page/folio and do the right thing?
Quoted 6 lines
mm/migrate_device.c +0
-0}else if ((src_pfns[i] &MIGRATE_PFN_MIGRATE ) && (dst_pfns[i] &MIGRATE_PFN_COMPOUND ) && !(src_pfns[i] &MIGRATE_PFN_COMPOUND )) {@@ -1221,6 +1228,12 @@ static void __migrate_device_pages(unsigned long *src_pfns, folio =page_folio (migrate_pfn_to_page (src_pfns[i+j])); newfolio =page_folio (migrate_pfn_to_page (dst_pfns[i+j]));
mm/migrate_device.c static void __migrate_device_pages(unsigned long *src_pfns, · line 1239
Quoted 10 lines
mm/migrate_device.c +6
-0+/* +* folio_free_swap() removed the folio from the swap +* cache. Refresh the saved mapping before migration. +*/ + mapping =folio_mapping (folio); + r =folio_migrate_mapping (mapping, newfolio, folio, extra_cnt);if (r) src_pfns[i+j] &= ~MIGRATE_PFN_MIGRATE ;
While this looks good, I do wonder why do we have to supply the mapping here at all?
Is there a path where we call folio_migrate_mapping() and the old folio (folio) does no longer have the right mapping attached?
It would be a lot less error prone if the function would just obtain the mapping from the old folio.
Signature · 3 lines
Cheers, David
Quoted 65 lines · David Hildenbrand (Arm)
Quoted 52 lines · Arvind Yadav
__migrate_device_pages()reads the folio mapping before callingfolio_free_swap(). Whenfolio_free_swap()succeeds, the folio is removed from the swap cache, but the saved mapping still points toswap_space.Passing the stale mapping to
folio_migrate_mapping()makes it use the mapped-folio path for a folio that is no longer in swapcache. It can then operate on swap_space.i_pages with invalid reference accounting, eventually triggering a folio reference count BUG.After a successful split, nr still contains the number of pages in the original large folio, although each resulting page is now a separate order-0 folio. Reset nr to 1 so each split folio is processed separately, including its own swapcache removal and mapping lookup.
Refresh the saved mapping after
folio_free_swap()so the current folio state is used during migration.v2:
- Refresh the mapping usingfolio_mapping(), as suggested by Zi Yan.v3:
- Reset nr to 1 after a successful split so each resulting folio is processed independently, as suggested by Zi Yan.v4:
- Re-read each source folio's mapping immediately beforefolio_migrate_mapping(), as suggested by Balbir Singh.
- Add a warning to validate the post-split order-0 invariant, as suggested by Balbir Singh.Fixes df263d9a7dff ("mm/migrate_device: try to handle swapcache pages")Cc Andrew Morton <[email protected]>Cc David Hildenbrand <[email protected]>Cc Matthew Brost <[email protected]>Cc Joshua Hahn <[email protected]>Cc Rakie Kim <[email protected]>Cc Byungchul Park <[email protected]>Cc Gregory Price <[email protected]>Cc Ying Huang <[email protected]>Cc Alistair Popple <[email protected]>Reviewed-by Zi Yan <[email protected]>Reviewed-by Balbir Singh <[email protected]>Signed-off-by Arvind Yadav <[email protected]>--- mm/migrate_device.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)mm/migrate_device.c +0
-0mm/migrate_device.c static void __migrate_device_pages(unsigned long *src_pfns, · line 1191
Quoted 10 lines
mm/migrate_device.c +7
-0@@ -1182,6 +1182,13 @@ static void __migrate_device_pages(unsigned long *src_pfns, MIGRATE_PFN_COMPOUND); goto next; } + +/* +* reset nr so that only first after-split folio +* is processed below +*/ +VM_WARN_ON_ONCE (folio_test_large (folio)); + nr =1 ;This function is surely a beauty. (had to rephrase that sentence 3 times ;) )
Quoted 7 lines
migrate_vma_split_unmapped_folio()modifies thesrc_pfns()entries on success. (and somehow assumes that it's always a THP, what? After aMIGRATE_PFN_COMPOUNDvalue is set? What? Why the "nr = 1 <<folio_order(folio);" in the caller).If we ended up modifying the current entry in such a way, shouldn't we just have
retry: label and restart at the very top of the function, where we just naturally re-read the entry/page/folio and do the right thing?
Agreed. Since migrate_vma_split_unmapped_folio() rewrites src_pfns[], I will retry the current entry so all state is re-read.
I will keep nr set to the original folio size for the split-failure path, where goto next must skip the complete folio range.
Quoted 19 lines
Quoted 7 lines
mm/migrate_device.c +0
-0}else if ((src_pfns[i] &MIGRATE_PFN_MIGRATE ) && (dst_pfns[i] &MIGRATE_PFN_COMPOUND ) && !(src_pfns[i] &MIGRATE_PFN_COMPOUND )) {@@ -1221,6 +1228,12 @@ static void __migrate_device_pages(unsigned long *src_pfns, folio =page_folio (migrate_pfn_to_page (src_pfns[i+j])); newfolio =page_folio (migrate_pfn_to_page (dst_pfns[i+j]));mm/migrate_device.c static void __migrate_device_pages(unsigned long *src_pfns, · line 1239
Quoted 10 lines
mm/migrate_device.c +6
-0+/* +* folio_free_swap() removed the folio from the swap +* cache. Refresh the saved mapping before migration. +*/ + mapping =folio_mapping (folio); + r =folio_migrate_mapping (mapping, newfolio, folio, extra_cnt);if (r) src_pfns[i+j] &= ~MIGRATE_PFN_MIGRATE ;While this looks good, I do wonder why do we have to supply the mapping here at all?
Quoted 5 lines
Is there a path where we call
folio_migrate_mapping()and the old folio (folio) does no longer have the right mapping attached?It would be a lot less error prone if the function would just obtain the mapping from the old folio.
Agreed. I will make folio_migrate_mapping() obtain the mapping from the source folio and update its in-tree callers. Since this changes an exported helper, I will send it as a separate patch in v5.
Please correct me if I misunderstood your suggestion.
Thanks,
Arvind
Quoted 1 line
David Hildenbrand (Arm)
maintainerQuoted 7 lines
Quoted 7 lines
While this looks good, I do wonder why do we have to supply the mapping here at all?
Is there a path where we call
folio_migrate_mapping()and the old folio (folio) does no longer have the right mapping attached?
Quoted 7 lines
Quoted 2 lines
It would be a lot less error prone if the function would just obtain the mapping from the old folio.
Agreed. I will make
folio_migrate_mapping()obtain the mapping from the source folio and update its in-tree callers. Since this changes an exported helper, I will send it as a separate patch in v5.
I think it would be good to understand why we currently pass in the folio: was there ever a reason to do so? Or why do we pass in something that can just easily be had from the source folio?
But note that that should be a follow-up cleanup to the fix.
Signature · 3 lines
Cheers, David
Quoted 9 lines · David Hildenbrand (Arm)
Quoted 9 lines
Quoted 8 lines
While this looks good, I do wonder why do we have to supply the mapping here at all?
Is there a path where we call
folio_migrate_mapping()and the old folio (folio) does no longer have the right mapping attached?It would be a lot less error prone if the function would just obtain the mapping from the old folio.
Quoted 8 lines
Quoted 3 lines
Agreed. I will make
folio_migrate_mapping()obtain the mapping from the source folio and update its in-tree callers. Since this changes an exported helper, I will send it as a separate patch in v5.I think it would be good to understand why we currently pass in the folio: was there ever a reason to do so? Or why do we pass in something that can just easily be had from the source folio?
But note that that should be a follow-up cleanup to the fix.
Thanks for the suggestion. Once this is merged, I will review the history and callers to confirm whether any path intentionally uses a mapping other than folio_mapping(src) and follow up with a separate cleanup patch if needed.
Thanks,
Arvind