Skip to contentSkip to conversation

mm/migrate_device: Clear stale mapping after freeing swapcache

PATCH Applied Arvind Yadav 3 participants 6 messages Jul 28 – Jul 31

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 mm/migrate_device: Clear stale mapping after freeing swapcache

__migrate_device_pages() reads the folio mapping before calling folio_free_swap(). When folio_free_swap() succeeds, the folio is removed from the swap cache, but the saved mapping still points to swap_space.

Passing the stale mapping to folio_migrate_mapping() makes it use the mapped-folio path for a folio that is no longer in swapcache. It can then operate on swap_space.i_pages with invalid reference accounting, eventually triggering a folio reference count BUG.

After a successful split, nr still contains the number of pages in the original large folio, although each resulting page is now a separate order-0 folio. Reset nr to 1 so each split folio is processed separately, including its own swapcache removal and mapping lookup.

Refresh the saved mapping after folio_free_swap() so the current folio state is used during migration.

v2:
- Refresh the mapping using folio_mapping(), as suggested by Zi Yan.

v3:
- Reset nr to 1 after a successful split so each resulting folio is processed independently, as suggested by Zi Yan.

v4:
- Re-read each source folio's mapping immediately before folio_migrate_mapping(), as suggested by Balbir Singh.
- Add a warning to validate the post-split order-0 invariant, as suggested by Balbir Singh.

Fixes df263d9a7dff ("mm/migrate_device: try to handle swapcache pages")
Cc Andrew Morton <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Matthew Brost <[email protected]>
Cc Joshua Hahn <[email protected]>
Cc Rakie Kim <[email protected]>
Cc Byungchul Park <[email protected]>
Cc Gregory Price <[email protected]>
Cc Ying Huang <[email protected]>
Cc Alistair Popple <[email protected]>
---
 mm/migrate_device.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
mm/migrate_device.c +13 -0
index 554754eb26ff..0e6f00190fd8 100644
@@ -1182,6 +1182,13 @@ static void __migrate_device_pages(unsigned long *src_pfns,
 							 MIGRATE_PFN_COMPOUND);
 					goto next;
 				}
+
+				/*
+				 * reset nr so that only first after-split folio
+				 * is processed below
+				 */
+				VM_WARN_ON_ONCE(folio_test_large(folio));
+				nr = 1;
 			} else if ((src_pfns[i] & MIGRATE_PFN_MIGRATE) &&
 				(dst_pfns[i] & MIGRATE_PFN_COMPOUND) &&
 				!(src_pfns[i] & MIGRATE_PFN_COMPOUND)) {
@@ -1221,6 +1228,12 @@ static void __migrate_device_pages(unsigned long *src_pfns,
 			folio = page_folio(migrate_pfn_to_page(src_pfns[i+j]));
 			newfolio = page_folio(migrate_pfn_to_page(dst_pfns[i+j]));
 
+			/*
+			 * folio_free_swap() removed the folio from the swap
+			 * cache. Refresh the saved mapping before migration.
+			 */
+			mapping = folio_mapping(folio);
+
 			r = folio_migrate_mapping(mapping, newfolio, folio, extra_cnt);
 			if (r)
 				src_pfns[i+j] &= ~MIGRATE_PFN_MIGRATE;
Signature · 1 line
2.43.0
Reviewed-by on v4 1/1
#1 lore raw
Quoted 43 lines · Arvind Yadav

__migrate_device_pages() reads the folio mapping before calling folio_free_swap(). When folio_free_swap() succeeds, the folio is removed from the swap cache, but the saved mapping still points to swap_space.

Passing the stale mapping to folio_migrate_mapping() makes it use the mapped-folio path for a folio that is no longer in swapcache. It can then operate on swap_space.i_pages with invalid reference accounting, eventually triggering a folio reference count BUG.

After a successful split, nr still contains the number of pages in the original large folio, although each resulting page is now a separate order-0 folio. Reset nr to 1 so each split folio is processed separately, including its own swapcache removal and mapping lookup.

Refresh the saved mapping after folio_free_swap() so the current folio state is used during migration.

v2:
- Refresh the mapping using folio_mapping(), as suggested by Zi Yan.

v3:
- Reset nr to 1 after a successful split so each resulting folio is processed independently, as suggested by Zi Yan.

v4:
- Re-read each source folio's mapping immediately before folio_migrate_mapping(), as suggested by Balbir Singh.
- Add a warning to validate the post-split order-0 invariant, as suggested by Balbir Singh.

Fixes df263d9a7dff ("mm/migrate_device: try to handle swapcache pages")
Cc Andrew Morton <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Matthew Brost <[email protected]>
Cc Joshua Hahn <[email protected]>
Cc Rakie Kim <[email protected]>
Cc Byungchul Park <[email protected]>
Cc Gregory Price <[email protected]>
Cc Ying Huang <[email protected]>
Cc Alistair Popple <[email protected]>
Quoted 4 lines
---
 mm/migrate_device.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
#2 lore raw
In reply to Arvind Yadav
Quoted 52 lines · Arvind Yadav

__migrate_device_pages() reads the folio mapping before calling folio_free_swap(). When folio_free_swap() succeeds, the folio is removed from the swap cache, but the saved mapping still points to swap_space.

Passing the stale mapping to folio_migrate_mapping() makes it use the mapped-folio path for a folio that is no longer in swapcache. It can then operate on swap_space.i_pages with invalid reference accounting, eventually triggering a folio reference count BUG.

After a successful split, nr still contains the number of pages in the original large folio, although each resulting page is now a separate order-0 folio. Reset nr to 1 so each split folio is processed separately, including its own swapcache removal and mapping lookup.

Refresh the saved mapping after folio_free_swap() so the current folio state is used during migration.

v2:
- Refresh the mapping using folio_mapping(), as suggested by Zi Yan.

v3:
- Reset nr to 1 after a successful split so each resulting folio is processed independently, as suggested by Zi Yan.

v4:
- Re-read each source folio's mapping immediately before folio_migrate_mapping(), as suggested by Balbir Singh.
- Add a warning to validate the post-split order-0 invariant, as suggested by Balbir Singh.

Fixes df263d9a7dff ("mm/migrate_device: try to handle swapcache pages")
Cc Andrew Morton <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Matthew Brost <[email protected]>
Cc Joshua Hahn <[email protected]>
Cc Rakie Kim <[email protected]>
Cc Byungchul Park <[email protected]>
Cc Gregory Price <[email protected]>
Cc Ying Huang <[email protected]>
Cc Alistair Popple <[email protected]>
---
 mm/migrate_device.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
mm/migrate_device.c +0 -0
index 554754eb26ff..0e6f00190fd8 100644

mm/migrate_device.c static void __migrate_device_pages(unsigned long *src_pfns, · line 1191

Quoted 10 lines
mm/migrate_device.c +7 -0
@@ -1182,6 +1182,13 @@ static void __migrate_device_pages(unsigned long *src_pfns,
 							 MIGRATE_PFN_COMPOUND);
 					goto next;
 				}
+
+				/*
+				 * reset nr so that only first after-split folio
+				 * is processed below
+				 */
+				VM_WARN_ON_ONCE(folio_test_large(folio));
+				nr = 1;

This function is surely a beauty. (had to rephrase that sentence 3 times ;) )

migrate_vma_split_unmapped_folio() modifies the src_pfns() entries on success. (and somehow assumes that it's always a THP, what? After a MIGRATE_PFN_COMPOUND value is set? What? Why the "nr = 1 << folio_order(folio);" in the caller).

If we ended up modifying the current entry in such a way, shouldn't we just have
retry: label and restart at the very top of the function, where we just naturally re-read the entry/page/folio and do the right thing?

Quoted 6 lines
mm/migrate_device.c +0 -0
 			} else if ((src_pfns[i] & MIGRATE_PFN_MIGRATE) &&
 				(dst_pfns[i] & MIGRATE_PFN_COMPOUND) &&
 				!(src_pfns[i] & MIGRATE_PFN_COMPOUND)) {
@@ -1221,6 +1228,12 @@ static void __migrate_device_pages(unsigned long *src_pfns,
 			folio = page_folio(migrate_pfn_to_page(src_pfns[i+j]));
 			newfolio = page_folio(migrate_pfn_to_page(dst_pfns[i+j]));

mm/migrate_device.c static void __migrate_device_pages(unsigned long *src_pfns, · line 1239

Quoted 10 lines
mm/migrate_device.c +6 -0
 
+			/*
+			 * folio_free_swap() removed the folio from the swap
+			 * cache. Refresh the saved mapping before migration.
+			 */
+			mapping = folio_mapping(folio);
+
 			r = folio_migrate_mapping(mapping, newfolio, folio, extra_cnt);
 			if (r)
 				src_pfns[i+j] &= ~MIGRATE_PFN_MIGRATE;

While this looks good, I do wonder why do we have to supply the mapping here at all?

Is there a path where we call folio_migrate_mapping() and the old folio (folio) does no longer have the right mapping attached?

It would be a lot less error prone if the function would just obtain the mapping from the old folio.

Signature · 3 lines
Cheers,

David
#3 lore raw
Quoted 65 lines · David Hildenbrand (Arm)
Quoted 52 lines · Arvind Yadav

__migrate_device_pages() reads the folio mapping before calling folio_free_swap(). When folio_free_swap() succeeds, the folio is removed from the swap cache, but the saved mapping still points to swap_space.

Passing the stale mapping to folio_migrate_mapping() makes it use the mapped-folio path for a folio that is no longer in swapcache. It can then operate on swap_space.i_pages with invalid reference accounting, eventually triggering a folio reference count BUG.

After a successful split, nr still contains the number of pages in the original large folio, although each resulting page is now a separate order-0 folio. Reset nr to 1 so each split folio is processed separately, including its own swapcache removal and mapping lookup.

Refresh the saved mapping after folio_free_swap() so the current folio state is used during migration.

v2:
- Refresh the mapping using folio_mapping(), as suggested by Zi Yan.

v3:
- Reset nr to 1 after a successful split so each resulting folio is processed independently, as suggested by Zi Yan.

v4:
- Re-read each source folio's mapping immediately before folio_migrate_mapping(), as suggested by Balbir Singh.
- Add a warning to validate the post-split order-0 invariant, as suggested by Balbir Singh.

Fixes df263d9a7dff ("mm/migrate_device: try to handle swapcache pages")
Cc Andrew Morton <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Matthew Brost <[email protected]>
Cc Joshua Hahn <[email protected]>
Cc Rakie Kim <[email protected]>
Cc Byungchul Park <[email protected]>
Cc Gregory Price <[email protected]>
Cc Ying Huang <[email protected]>
Cc Alistair Popple <[email protected]>
---
  mm/migrate_device.c | 13 +++++++++++++
  1 file changed, 13 insertions(+)
mm/migrate_device.c +0 -0
index 554754eb26ff..0e6f00190fd8 100644

mm/migrate_device.c static void __migrate_device_pages(unsigned long *src_pfns, · line 1191

Quoted 10 lines
mm/migrate_device.c +7 -0
@@ -1182,6 +1182,13 @@ static void __migrate_device_pages(unsigned long *src_pfns,
  							 MIGRATE_PFN_COMPOUND);
  					goto next;
  				}
+
+				/*
+				 * reset nr so that only first after-split folio
+				 * is processed below
+				 */
+				VM_WARN_ON_ONCE(folio_test_large(folio));
+				nr = 1;

This function is surely a beauty. (had to rephrase that sentence 3 times ;) )

Quoted 7 lines

migrate_vma_split_unmapped_folio() modifies the src_pfns() entries on success. (and somehow assumes that it's always a THP, what? After a MIGRATE_PFN_COMPOUND value is set? What? Why the "nr = 1 << folio_order(folio);" in the caller).

If we ended up modifying the current entry in such a way, shouldn't we just have
retry: label and restart at the very top of the function, where we just naturally re-read the entry/page/folio and do the right thing?

Agreed. Since migrate_vma_split_unmapped_folio() rewrites src_pfns[], I will retry the current entry so all state is re-read.

I will keep nr set to the original folio size for the split-failure path, where goto next must skip the complete folio range.

Quoted 19 lines
Quoted 7 lines
mm/migrate_device.c +0 -0
  			} else if ((src_pfns[i] & MIGRATE_PFN_MIGRATE) &&
  				(dst_pfns[i] & MIGRATE_PFN_COMPOUND) &&
  				!(src_pfns[i] & MIGRATE_PFN_COMPOUND)) {
@@ -1221,6 +1228,12 @@ static void __migrate_device_pages(unsigned long *src_pfns,
  			folio = page_folio(migrate_pfn_to_page(src_pfns[i+j]));
  			newfolio = page_folio(migrate_pfn_to_page(dst_pfns[i+j]));

mm/migrate_device.c static void __migrate_device_pages(unsigned long *src_pfns, · line 1239

Quoted 10 lines
mm/migrate_device.c +6 -0
 
+			/*
+			 * folio_free_swap() removed the folio from the swap
+			 * cache. Refresh the saved mapping before migration.
+			 */
+			mapping = folio_mapping(folio);
+
  			r = folio_migrate_mapping(mapping, newfolio, folio, extra_cnt);
  			if (r)
  				src_pfns[i+j] &= ~MIGRATE_PFN_MIGRATE;

While this looks good, I do wonder why do we have to supply the mapping here at all?

Quoted 5 lines

Is there a path where we call folio_migrate_mapping() and the old folio (folio) does no longer have the right mapping attached?

It would be a lot less error prone if the function would just obtain the mapping from the old folio.

Agreed. I will make folio_migrate_mapping() obtain the mapping from the source folio and update its in-tree callers. Since this changes an exported helper, I will send it as a separate patch in v5.

Please correct me if I misunderstood your suggestion.

Thanks,
Arvind

Quoted 1 line
#4 lore raw
Quoted 7 lines
Quoted 7 lines

While this looks good, I do wonder why do we have to supply the mapping here at all?

Is there a path where we call folio_migrate_mapping() and the old folio (folio) does no longer have the right mapping attached?

Quoted 7 lines
Quoted 2 lines

It would be a lot less error prone if the function would just obtain the mapping from the old folio.

Agreed. I will make folio_migrate_mapping() obtain the mapping from the source folio and update its in-tree callers. Since this changes an exported helper, I will send it as a separate patch in v5.

I think it would be good to understand why we currently pass in the folio: was there ever a reason to do so? Or why do we pass in something that can just easily be had from the source folio?

But note that that should be a follow-up cleanup to the fix.

Signature · 3 lines
Cheers,

David
#5 lore raw
Quoted 9 lines · David Hildenbrand (Arm)
Quoted 9 lines
Quoted 8 lines

While this looks good, I do wonder why do we have to supply the mapping here at all?

Is there a path where we call folio_migrate_mapping() and the old folio (folio) does no longer have the right mapping attached?

It would be a lot less error prone if the function would just obtain the mapping from the old folio.

Quoted 8 lines
Quoted 3 lines

Agreed. I will make folio_migrate_mapping() obtain the mapping from the source folio and update its in-tree callers. Since this changes an exported helper, I will send it as a separate patch in v5.

I think it would be good to understand why we currently pass in the folio: was there ever a reason to do so? Or why do we pass in something that can just easily be had from the source folio?

But note that that should be a follow-up cleanup to the fix.

Thanks for the suggestion. Once this is merged, I will review the history and callers to confirm whether any path intentionally uses a mapping other than folio_mapping(src) and follow up with a separate cleanup patch if needed.

Thanks,
Arvind