mm/migrate: fix stale partially_mapped arg to deferred_split_folio()
Subsystems (2)
PATCH 1/1 mm/migrate: fix stale partially_mapped arg to deferred_split_folio()
In migrate_folio_move(), src_partially_mapped is sampled from the source folio before move_to_new_folio() is called:
if (folio_order(src) > 1 &&
!data_race(list_empty(&src->_deferred_list))) {
src_deferred_split = true;
src_partially_mapped = folio_test_partially_mapped(src);
}A concurrent thread can unmap pages from the source folio between this read and the actual migration, making the sampled value stale.
After move_to_new_folio() succeeds, __folio_migrate_mapping() has already copied all folio flags including PG_partially_mapped from src to dst. Passing the stale src_partially_mapped=false to deferred_split_folio(dst) while dst already has PG_partially_mapped=true triggers the invariant check in deferred_split_folio():
VM_WARN_ON_FOLIO(folio_test_partially_mapped(folio), folio)
at mm/huge_memory.c:4371, because the argument contradicts the flag already set on the folio.
Fix this by removing the src_partially_mapped variable entirely and reading PG_partially_mapped directly from dst after move_to_new_folio() completes, where it is authoritative and race-free.
--- mm/migrate.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
mm/migrate.c +5 -5
@@ -1361,7 +1361,6 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,int old_page_state =0 ;struct anon_vma *anon_vma =NULL ;bool src_deferred_split = false; -bool src_partially_mapped = false;struct list_head *prev;__migrate_folio_extract (dst, &old_page_state, &anon_vma);
@@ -1378,7 +1377,6 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,if (folio_order (src) >1 && !data_race (list_empty (&src->_deferred_list))) { src_deferred_split = true; - src_partially_mapped =folio_test_partially_mapped (src); } rc =move_to_new_folio (dst, src, mode);
@@ -1404,11 +1402,13 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,/* * Requeue the destination folio on the deferred split queue if * the source was on the queue. The source is unqueued in -* __folio_migrate_mapping(), so we recorded the state from -* before move_to_new_folio(). +* __folio_migrate_mapping(). Read PG_partially_mapped directly from +* dst: move_to_new_folio() copies all flags from src to dst, so dst +* now holds the correct authoritative state. The pre-migration value +* sampled from src is racy and must not be used. */ if (src_deferred_split) -deferred_split_folio (dst, src_partially_mapped); +deferred_split_folio (dst,folio_test_partially_mapped (dst)); out_unlock_both:folio_unlock (dst);
Signature · 1 line
2.43.0
David Hildenbrand (Arm)
maintainerQuoted 11 lines · Deepanshu Kartikey
In
migrate_folio_move(),src_partially_mappedis sampled from the source folio beforemove_to_new_folio()is called:if (folio_order(src) > 1 && !data_race(list_empty(&src->_deferred_list))) { src_deferred_split = true; src_partially_mapped = folio_test_partially_mapped(src); }A concurrent thread can unmap pages from the source folio between this read and the actual migration, making the sampled value stale.
Indeed. We could see "partially_mapped" getting set concurrently.
Quoted 46 lines
After
move_to_new_folio()succeeds,__folio_migrate_mapping()has already copied all folio flags includingPG_partially_mappedfrom src to dst. Passing the stale src_partially_mapped=false todeferred_split_folio(dst)while dst already has PG_partially_mapped=true triggers the invariant check indeferred_split_folio():
VM_WARN_ON_FOLIO(folio_test_partially_mapped(folio), folio)at
mm/huge_memory.c:4371, because the argument contradicts the flag already set on the folio.Fix this by removing the
src_partially_mappedvariable entirely and readingPG_partially_mappeddirectly from dst aftermove_to_new_folio()completes, where it is authoritative and race-free.Fixes 8a8ca142a488 ("mm: migrate: requeue destination folio on deferred split queue")Reported-by [email protected]Tested-by [email protected]Signed-off-by Deepanshu Kartikey <[email protected]>--- mm/migrate.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)mm/migrate.c +0
-2@@ -1361,7 +1361,6 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,int old_page_state =0 ;struct anon_vma *anon_vma =NULL ;bool src_deferred_split = false; -bool src_partially_mapped = false;struct list_head *prev;__migrate_folio_extract (dst, &old_page_state, &anon_vma);@@ -1378,7 +1377,6 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,if (folio_order (src) >1 && !data_race (list_empty (&src->_deferred_list))) { src_deferred_split = true; - src_partially_mapped =folio_test_partially_mapped (src); } rc =move_to_new_folio (dst, src, mode);
mm/migrate.c static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · line 1407
Quoted 8 lines
mm/migrate.c +3
-2@@ -1404,11 +1402,13 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,/* * Requeue the destination folio on the deferred split queue if * the source was on the queue. The source is unqueued in -* __folio_migrate_mapping(), so we recorded the state from -* before move_to_new_folio(). +* __folio_migrate_mapping(). Read PG_partially_mapped directly from +* dst: move_to_new_folio() copies all flags from src to dst, so dst +* now holds the correct authoritative state.
Where are we copying PG_partially_mapped exactly?
I don't see that happening in folio_migrate_flags().
Also, I wonder what happens if the folio gets partially mapped after already installing migration entries (zapping migration entries).
This is all extremely shaky.
Signature · 3 lines
Cheers, David
David Hildenbrand (Arm)
maintainerQuoted 11 lines · Deepanshu Kartikey
In
migrate_folio_move(),src_partially_mappedis sampled from the source folio beforemove_to_new_folio()is called:if (folio_order(src) > 1 && !data_race(list_empty(&src->_deferred_list))) { src_deferred_split = true; src_partially_mapped = folio_test_partially_mapped(src); }A concurrent thread can unmap pages from the source folio between this read and the actual migration, making the sampled value stale.
Trying to make sense of this.
In migrate_folio_move() don't we have the folio completely unmapped because there are only migration entries referencing the folio?
See migrate_folio_unmap(), where we check !folio_mapped().
Why should we suddenly have mapped folio here? Something is off.
Unmapping a migration entry will not involve rmap code and not mess with the partially-mapped flag.
Signature · 3 lines
Cheers, David
David Hildenbrand (Arm)
maintainerQuoted 18 lines · David Hildenbrand (Arm)
Quoted 11 lines · Deepanshu Kartikey
In
migrate_folio_move(),src_partially_mappedis sampled from the source folio beforemove_to_new_folio()is called:if (folio_order(src) > 1 && !data_race(list_empty(&src->_deferred_list))) { src_deferred_split = true; src_partially_mapped = folio_test_partially_mapped(src); }A concurrent thread can unmap pages from the source folio between this read and the actual migration, making the sampled value stale.
Trying to make sense of this.
In
migrate_folio_move()don't we have the folio completely unmapped because there are only migration entries referencing the folio?
Quoted 7 lines
See
migrate_folio_unmap(), where we check !folio_mapped().Why should we suddenly have mapped folio here? Something is off.
Unmapping a migration entry will not involve rmap code and not mess with the partially-mapped flag.
Okay, concluding that the above reasoning is all wrong.
Let's discuss it with Lance's proposal, which makes more sense.
https://lore.kernel.org/r/[email protected]
Signature · 3 lines
Cheers, David