mm: migrate: requeue destination folio on deferred split queue
Subsystems (2)
PATCH 1/1 mm: migrate: requeue destination folio on deferred split queue
During folio migration, __folio_migrate_mapping() removes the source folio from the deferred split queue, but the destination folio is never re-queued. This causes underutilized THPs to escape the shrinker after NUMA migration, since they silently drop off the deferred split list.
Fix this by calling deferred_split_folio() on the destination folio after a successful migration, for large rmappable folios.
--- mm/migrate.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
mm/migrate.c +11 -0
@@ -1393,6 +1393,17 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,if (old_page_state &PAGE_WAS_MAPPED )remove_migration_ptes (src, dst,0 ); +/* +* Requeue the destination folio on the deferred split queue if +* the source was a large folio that was on the queue. Without +* this, NUMA migration causes underutilized THPs to escape +* the shrinker since the source is unqueued in +* __folio_migrate_mapping() and the destination is never +* re-queued. +*/ +if (folio_test_large (dst) &&folio_test_large_rmappable (dst)) +deferred_split_folio (dst, false); + out_unlock_both:folio_unlock (dst);folio_set_owner_migrate_reason (dst, reason);
Signature · 1 line
2.47.3
David Hildenbrand (Arm)
maintainerQuoted 23 lines · Usama Arif
During folio migration,
__folio_migrate_mapping()removes the source folio from the deferred split queue, but the destination folio is never re-queued. This causes underutilized THPs to escape the shrinker after NUMA migration, since they silently drop off the deferred split list.Fix this by calling
deferred_split_folio()on the destination folio after a successful migration, for large rmappable folios.Reported-by Johannes Weiner <[email protected]>Fixes dafff3f4c850 ("mm: split underused THPs")Signed-off-by Usama Arif <[email protected]>--- mm/migrate.c | 11 +++++++++++ 1 file changed, 11 insertions(+)mm/migrate.c +0
-0@@ -1393,6 +1393,17 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,if (old_page_state &PAGE_WAS_MAPPED )remove_migration_ptes (src, dst,0 );
mm/migrate.c static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · line 1405
Quoted 10 lines
mm/migrate.c +10
-0+/* +* Requeue the destination folio on the deferred split queue if +* the source was a large folio that was on the queue. Without +* this, NUMA migration causes underutilized THPs to escape +* the shrinker since the source is unqueued in +* __folio_migrate_mapping() and the destination is never +* re-queued. +*/ +if (folio_test_large (dst) &&folio_test_large_rmappable (dst)) +deferred_split_folio (dst, false);
Doesn't that mean that you will readd any large folios, even if already previously taken off the list after scanning?
So I am not sure if your "if the source was a large folio that was on the queue." comment is accurate?
Signature · 3 lines
Cheers, David
Quoted 38 lines · David Hildenbrand (Arm)
Quoted 23 lines · Usama Arif
During folio migration,
__folio_migrate_mapping()removes the source folio from the deferred split queue, but the destination folio is never re-queued. This causes underutilized THPs to escape the shrinker after NUMA migration, since they silently drop off the deferred split list.Fix this by calling
deferred_split_folio()on the destination folio after a successful migration, for large rmappable folios.Reported-by Johannes Weiner <[email protected]>Fixes dafff3f4c850 ("mm: split underused THPs")Signed-off-by Usama Arif <[email protected]>--- mm/migrate.c | 11 +++++++++++ 1 file changed, 11 insertions(+)mm/migrate.c +0
-0@@ -1393,6 +1393,17 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,if (old_page_state &PAGE_WAS_MAPPED )remove_migration_ptes (src, dst,0 );mm/migrate.c static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · line 1405
Quoted 10 lines
mm/migrate.c +10
-0+/* +* Requeue the destination folio on the deferred split queue if +* the source was a large folio that was on the queue. Without +* this, NUMA migration causes underutilized THPs to escape +* the shrinker since the source is unqueued in +* __folio_migrate_mapping() and the destination is never +* re-queued. +*/ +if (folio_test_large (dst) &&folio_test_large_rmappable (dst)) +deferred_split_folio (dst, false);Doesn't that mean that you will readd any large folios, even if already previously taken off the list after scanning?
Quoted 3 lines
So I am not sure if your "if the source was a large folio that was on the queue." comment is accurate?
Yes you are right. How about something like below? We also won't need to check for anon and non-device folios with this as we only set the the flag if it was already on deferred_split list.
mm/migrate.c +21 -1
@@ -1137,7 +1137,9 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,enum {PAGE_WAS_MAPPED =BIT (0 ),PAGE_WAS_MLOCKED =BIT (1 ), -PAGE_OLD_STATES =PAGE_WAS_MAPPED |PAGE_WAS_MLOCKED , +PAGE_WAS_ON_DEFERRED_SPLIT =BIT (2 ), +PAGE_OLD_STATES =PAGE_WAS_MAPPED |PAGE_WAS_MLOCKED | +PAGE_WAS_ON_DEFERRED_SPLIT , }; static void __migrate_folio_record(struct folio *dst,
@@ -1373,6 +1375,15 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, goto out_unlock_both; } +/* +* Record whether the source folio is on the deferred split queue +* before move_to_new_folio(), which unqueues it via +* __folio_migrate_mapping(). +*/ +if (folio_test_large (src) &&folio_test_large_rmappable (src) && + !data_race (list_empty (&src->_deferred_list))) + old_page_state |=PAGE_WAS_ON_DEFERRED_SPLIT ; + rc =move_to_new_folio (dst, src, mode);if (rc) goto out;
@@ -1393,6 +1404,15 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,if (old_page_state &PAGE_WAS_MAPPED )remove_migration_ptes (src, dst,0 ); +/* +* Requeue the destination folio on the deferred split queue if +* the source was on the queue. The source is unqueued in +* __folio_migrate_mapping(), so we record and check the state +* from before move_to_new_folio(). +*/ +if (old_page_state &PAGE_WAS_ON_DEFERRED_SPLIT ) +deferred_split_folio (dst, false); + out_unlock_both:folio_unlock (dst);folio_set_owner_migrate_reason (dst, reason);
Zi Yan
reviewerQuoted 39 lines · Usama Arif
Quoted 38 lines · David Hildenbrand (Arm)
Quoted 23 lines · Usama Arif
During folio migration,
__folio_migrate_mapping()removes the source folio from the deferred split queue, but the destination folio is never re-queued. This causes underutilized THPs to escape the shrinker after NUMA migration, since they silently drop off the deferred split list.Fix this by calling
deferred_split_folio()on the destination folio after a successful migration, for large rmappable folios.Reported-by Johannes Weiner <[email protected]>Fixes dafff3f4c850 ("mm: split underused THPs")Signed-off-by Usama Arif <[email protected]>--- mm/migrate.c | 11 +++++++++++ 1 file changed, 11 insertions(+)mm/migrate.c +0
-0@@ -1393,6 +1393,17 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,if (old_page_state &PAGE_WAS_MAPPED )remove_migration_ptes (src, dst,0 );mm/migrate.c static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · line 1405
Quoted 10 lines
mm/migrate.c +10
-0+/* +* Requeue the destination folio on the deferred split queue if +* the source was a large folio that was on the queue. Without +* this, NUMA migration causes underutilized THPs to escape +* the shrinker since the source is unqueued in +* __folio_migrate_mapping() and the destination is never +* re-queued. +*/ +if (folio_test_large (dst) &&folio_test_large_rmappable (dst)) +deferred_split_folio (dst, false);Doesn't that mean that you will readd any large folios, even if already previously taken off the list after scanning?
Quoted 7 lines
Quoted 3 lines
So I am not sure if your "if the source was a large folio that was on the queue." comment is accurate?
Yes you are right. How about something like below? We also won't need to check for anon and non-device folios with this as we only set the the flag if it was already on
deferred_splitlist.
BTW, migrate_pages() tries to split partially mapped folios before migration[1], so what remains in the deferred_list would be:
1. partially mapped but with a pin,
2. fully mapped but potentially underused.
I wonder if you want to do an underused scan before migration and try to split underused THPs. Or to avoid this additional scan, find a way of detecting zero pages at page copy time and split it after migration.
Anyway, it seems that all large folios are in this deferred_list. Maybe, like David suggested in his LSFMM proposal, we should scan large folios on LRU lists at reclaim time instead, since there is not much difference between deferred_list and LRU lists right now.
[1] https://elixir.bootlin.com/linux/v6.19.3/source/mm/migrate.c#L1840
Quoted 39 lines
mm/migrate.c +14
-1@@ -1137,7 +1137,9 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,enum {PAGE_WAS_MAPPED =BIT (0 ),PAGE_WAS_MLOCKED =BIT (1 ), -PAGE_OLD_STATES =PAGE_WAS_MAPPED |PAGE_WAS_MLOCKED , +PAGE_WAS_ON_DEFERRED_SPLIT =BIT (2 ), +PAGE_OLD_STATES =PAGE_WAS_MAPPED |PAGE_WAS_MLOCKED | +PAGE_WAS_ON_DEFERRED_SPLIT , }; static void __migrate_folio_record(struct folio *dst,@@ -1373,6 +1375,15 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, goto out_unlock_both; } +/* +* Record whether the source folio is on the deferred split queue +* before move_to_new_folio(), which unqueues it via +* __folio_migrate_mapping(). +*/ +if (folio_test_large (src) &&folio_test_large_rmappable (src) && + !data_race (list_empty (&src->_deferred_list))) + old_page_state |=PAGE_WAS_ON_DEFERRED_SPLIT ; + rc =move_to_new_folio (dst, src, mode);if (rc) goto out;@@ -1393,6 +1404,15 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,if (old_page_state &PAGE_WAS_MAPPED )remove_migration_ptes (src, dst,0 ); +/* +* Requeue the destination folio on the deferred split queue if
mm/migrate.c static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · line 1418
Quoted 10 lines
mm/migrate.c +7
-0+* the source was on the queue. The source is unqueued in +* __folio_migrate_mapping(), so we record and check the state +* from before move_to_new_folio(). +*/ +if (old_page_state &PAGE_WAS_ON_DEFERRED_SPLIT ) +deferred_split_folio (dst, false); + out_unlock_both:folio_unlock (dst);folio_set_owner_migrate_reason (dst, reason);
Best Regards,
Yan, Zi
Quoted 49 lines · Zi Yan
Quoted 39 lines · Usama Arif
Quoted 38 lines · David Hildenbrand (Arm)
Quoted 23 lines · Usama Arif
During folio migration,
__folio_migrate_mapping()removes the source folio from the deferred split queue, but the destination folio is never re-queued. This causes underutilized THPs to escape the shrinker after NUMA migration, since they silently drop off the deferred split list.Fix this by calling
deferred_split_folio()on the destination folio after a successful migration, for large rmappable folios.Reported-by Johannes Weiner <[email protected]>Fixes dafff3f4c850 ("mm: split underused THPs")Signed-off-by Usama Arif <[email protected]>--- mm/migrate.c | 11 +++++++++++ 1 file changed, 11 insertions(+)mm/migrate.c +0
-0@@ -1393,6 +1393,17 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,if (old_page_state &PAGE_WAS_MAPPED )remove_migration_ptes (src, dst,0 );mm/migrate.c static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · line 1405
Quoted 10 lines
mm/migrate.c +10
-0+/* +* Requeue the destination folio on the deferred split queue if +* the source was a large folio that was on the queue. Without +* this, NUMA migration causes underutilized THPs to escape +* the shrinker since the source is unqueued in +* __folio_migrate_mapping() and the destination is never +* re-queued. +*/ +if (folio_test_large (dst) &&folio_test_large_rmappable (dst)) +deferred_split_folio (dst, false);Doesn't that mean that you will readd any large folios, even if already previously taken off the list after scanning?
Quoted 7 lines
Quoted 3 lines
So I am not sure if your "if the source was a large folio that was on the queue." comment is accurate?
Yes you are right. How about something like below? We also won't need to check for anon and non-device folios with this as we only set the the flag if it was already on
deferred_splitlist.
Quoted 6 lines
BTW,
migrate_pages()tries to split partially mapped folios before migration[1], so what remains in thedeferred_listwould be:1. partially mapped but with a pin,
2. fully mapped but potentially underused.
Yes, thats right.
Quoted 2 lines
I wonder if you want to do an underused scan before migration and try to split underused THPs.
hmm, I think we should keep THPs as is if there is no memory pressure (proactive or otherwise). Scanning THPs for zeros has a cost and we would also lose the benefit of THPs when we dont need memory.
Quoted 3 lines
Or to avoid this additional scan, find a way of detecting zero pages at page copy time and split it after migration.
Yeah but I think we lose the benefits of THPs after migration when we dont need additional memory?
Quoted 5 lines
Anyway, it seems that all large folios are in this
deferred_list. Maybe, like David suggested in his LSFMM proposal, we should scan large folios on LRU lists at reclaim time instead, since there is not much difference betweendeferred_listand LRU lists right now.
Yeah the THP shrinker is a very basic implementation and there are a lot of
Quoted 3 lines
[1] https://elixir.bootlin.com/linux/v6.19.3/source/mm/migrate.c#L1840
Also Johannes pointed out its not great storing this information in page flags, we can just keep it as local variable. This is what the patch would look like:
mm/migrate.c +14 -0
@@ -1360,6 +1360,7 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,int rc;int old_page_state =0 ;struct anon_vma *anon_vma =NULL ; +bool src_deferred_split = false;struct list_head *prev;__migrate_folio_extract (dst, &old_page_state, &anon_vma);
@@ -1373,6 +1374,10 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, goto out_unlock_both; } +if (folio_test_large (src) &&folio_test_large_rmappable (src) && + !data_race (list_empty (&src->_deferred_list))) + src_deferred_split = true; + rc =move_to_new_folio (dst, src, mode);if (rc) goto out;
@@ -1393,6 +1398,15 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,if (old_page_state &PAGE_WAS_MAPPED )remove_migration_ptes (src, dst,0 ); +/* +* Requeue the destination folio on the deferred split queue if +* the source was on the queue. The source is unqueued in +* __folio_migrate_mapping(), so we recorded the state from +* before move_to_new_folio(). +*/ +if (src_deferred_split) +deferred_split_folio (dst, false); + out_unlock_both:folio_unlock (dst);folio_set_owner_migrate_reason (dst, reason);
David Hildenbrand (Arm)
maintainerQuoted 39 lines · Usama Arif
Quoted 7 lines · Zi Yan
Quoted 4 lines · Usama Arif
Yes you are right. How about something like below? We also won't need to check for anon and non-device folios with this as we only set the the flag if it was already on
deferred_splitlist.Quoted 6 lines
BTW,
migrate_pages()tries to split partially mapped folios before migration[1], so what remains in thedeferred_listwould be:1. partially mapped but with a pin,
2. fully mapped but potentially underused.Yes, thats right.
Quoted 2 lines
I wonder if you want to do an underused scan before migration and try to split underused THPs.
hmm, I think we should keep THPs as is if there is no memory pressure (proactive or otherwise). Scanning THPs for zeros has a cost and we would also lose the benefit of THPs when we dont need memory.
Quoted 3 lines
Or to avoid this additional scan, find a way of detecting zero pages at page copy time and split it after migration.
Yeah but I think we lose the benefits of THPs after migration when we dont need additional memory?
Quoted 5 lines
Anyway, it seems that all large folios are in this
deferred_list. Maybe, like David suggested in his LSFMM proposal, we should scan large folios on LRU lists at reclaim time instead, since there is not much difference betweendeferred_listand LRU lists right now.
Quoted 8 lines
Yeah the THP shrinker is a very basic implementation and there are a lot of
Quoted 3 lines
[1] https://elixir.bootlin.com/linux/v6.19.3/source/mm/migrate.c#L1840
Also Johannes pointed out its not great storing this information in page flags, we can just keep it as local variable. This is what the patch would look like:
Much cleaner.
Signature · 3 lines
Cheers, David
Zi Yan
reviewerQuoted 57 lines · Usama Arif
Quoted 49 lines · Zi Yan
Quoted 39 lines · Usama Arif
Quoted 38 lines · David Hildenbrand (Arm)
Quoted 23 lines · Usama Arif
During folio migration,
__folio_migrate_mapping()removes the source folio from the deferred split queue, but the destination folio is never re-queued. This causes underutilized THPs to escape the shrinker after NUMA migration, since they silently drop off the deferred split list.Fix this by calling
deferred_split_folio()on the destination folio after a successful migration, for large rmappable folios.Reported-by Johannes Weiner <[email protected]>Fixes dafff3f4c850 ("mm: split underused THPs")Signed-off-by Usama Arif <[email protected]>--- mm/migrate.c | 11 +++++++++++ 1 file changed, 11 insertions(+)mm/migrate.c +0
-0@@ -1393,6 +1393,17 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,if (old_page_state &PAGE_WAS_MAPPED )remove_migration_ptes (src, dst,0 );mm/migrate.c static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · line 1405
Quoted 10 lines
mm/migrate.c +10
-0+/* +* Requeue the destination folio on the deferred split queue if +* the source was a large folio that was on the queue. Without +* this, NUMA migration causes underutilized THPs to escape +* the shrinker since the source is unqueued in +* __folio_migrate_mapping() and the destination is never +* re-queued. +*/ +if (folio_test_large (dst) &&folio_test_large_rmappable (dst)) +deferred_split_folio (dst, false);Doesn't that mean that you will readd any large folios, even if already previously taken off the list after scanning?
Quoted 7 lines
Quoted 3 lines
So I am not sure if your "if the source was a large folio that was on the queue." comment is accurate?
Yes you are right. How about something like below? We also won't need to check for anon and non-device folios with this as we only set the the flag if it was already on
deferred_splitlist.Quoted 6 lines
BTW,
migrate_pages()tries to split partially mapped folios before migration[1], so what remains in thedeferred_listwould be:1. partially mapped but with a pin,
2. fully mapped but potentially underused.
Quoted 8 lines
Yes, thats right.
Quoted 2 lines
I wonder if you want to do an underused scan before migration and try to split underused THPs.
hmm, I think we should keep THPs as is if there is no memory pressure (proactive or otherwise). Scanning THPs for zeros has a cost and we would also lose the benefit of THPs when we dont need memory.
Makes sense.
Quoted 7 lines
Quoted 4 lines
Or to avoid this additional scan, find a way of detecting zero pages at page copy time and split it after migration.
Yeah but I think we lose the benefits of THPs after migration when we dont need additional memory?
Right.
Quoted 46 lines
Quoted 6 lines
Anyway, it seems that all large folios are in this
deferred_list. Maybe, like David suggested in his LSFMM proposal, we should scan large folios on LRU lists at reclaim time instead, since there is not much difference betweendeferred_listand LRU lists right now.Yeah the THP shrinker is a very basic implementation and there are a lot of
Quoted 3 lines
[1] https://elixir.bootlin.com/linux/v6.19.3/source/mm/migrate.c#L1840
Also Johannes pointed out its not great storing this information in page flags, we can just keep it as local variable. This is what the patch would look like:
mm/migrate.c +7
-0@@ -1360,6 +1360,7 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,int rc;int old_page_state =0 ;struct anon_vma *anon_vma =NULL ; +bool src_deferred_split = false;struct list_head *prev;__migrate_folio_extract (dst, &old_page_state, &anon_vma);@@ -1373,6 +1374,10 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, goto out_unlock_both; } +if (folio_test_large (src) &&folio_test_large_rmappable (src) && + !data_race (list_empty (&src->_deferred_list))) + src_deferred_split = true; + rc =move_to_new_folio (dst, src, mode);if (rc) goto out;@@ -1393,6 +1398,15 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,if (old_page_state &PAGE_WAS_MAPPED )remove_migration_ptes (src, dst,0 ); +/* +* Requeue the destination folio on the deferred split queue if
mm/migrate.c static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · line 1412
Quoted 10 lines
mm/migrate.c +7
-0+* the source was on the queue. The source is unqueued in +* __folio_migrate_mapping(), so we recorded the state from +* before move_to_new_folio(). +*/ +if (src_deferred_split) +deferred_split_folio (dst, false); + out_unlock_both:folio_unlock (dst);folio_set_owner_migrate_reason (dst, reason);
LGTM. Thanks for improving it.
Best Regards,
Yan, Zi
mm/migrate.c static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · line 1384
Quoted 7 lines · Usama Arif
mm/migrate.c +7
-0+/* +* Record whether the source folio is on the deferred split queue +* before move_to_new_folio(), which unqueues it via +* __folio_migrate_mapping(). +*/ +if (folio_test_large (src) &&folio_test_large_rmappable (src) && + !data_race (list_empty (&src->_deferred_list)))
Why do you need data_race() here? list_empty() contains a READ_ONCE() so shouldn't be necessary?
Quoted 1 line
+
old_page_state|=PAGE_WAS_ON_DEFERRED_SPLIT;
You've done a great job of the naming. So much so that the comment seems entirely unnecessary?
mm/migrate.c static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · line 1414
Quoted 8 lines
mm/migrate.c +8
-0+/* +* Requeue the destination folio on the deferred split queue if +* the source was on the queue. The source is unqueued in +* __folio_migrate_mapping(), so we record and check the state +* from before move_to_new_folio(). +*/ +if (old_page_state &PAGE_WAS_ON_DEFERRED_SPLIT ) +deferred_split_folio (dst, false);
Again, I'm not sure the comment says anything that the code doesn't?
Quoted 11 lines · Matthew Wilcox
mm/migrate.c static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · line 1384
Quoted 7 lines · Usama Arif
mm/migrate.c +7
-0+/* +* Record whether the source folio is on the deferred split queue +* before move_to_new_folio(), which unqueues it via +* __folio_migrate_mapping(). +*/ +if (folio_test_large (src) &&folio_test_large_rmappable (src) && + !data_race (list_empty (&src->_deferred_list)))Why do you need
data_race()here?list_empty()contains aREAD_ONCE()so shouldn't be necessary?
Ah mainly because we dont acquire split_queue_lock before accessing, similar to what we do in folio_unqueue_deferred_split().
Quoted 15 lines
Quoted 2 lines
+
old_page_state|=PAGE_WAS_ON_DEFERRED_SPLIT;You've done a great job of the naming. So much so that the comment seems entirely unnecessary?
mm/migrate.c static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · line 1414
Quoted 8 lines
mm/migrate.c +8
-0+/* +* Requeue the destination folio on the deferred split queue if +* the source was on the queue. The source is unqueued in +* __folio_migrate_mapping(), so we record and check the state +* from before move_to_new_folio(). +*/ +if (old_page_state &PAGE_WAS_ON_DEFERRED_SPLIT ) +deferred_split_folio (dst, false);
Quoted 1 line
Again, I'm not sure the comment says anything that the code doesn't?
Yeah there is a much simpler version in reply to Zi's review in [1]. I found the whole migrate_folio_move() function quite verbose already, was keeping up with the theme of that function lol. I will see if I can cut down on the comment in [1] as well. Thanks!
David Hildenbrand (Arm)
maintainerQuoted 23 lines · Usama Arif
During folio migration,
__folio_migrate_mapping()removes the source folio from the deferred split queue, but the destination folio is never re-queued. This causes underutilized THPs to escape the shrinker after NUMA migration, since they silently drop off the deferred split list.Fix this by calling
deferred_split_folio()on the destination folio after a successful migration, for large rmappable folios.Reported-by Johannes Weiner <[email protected]>Fixes dafff3f4c850 ("mm: split underused THPs")Signed-off-by Usama Arif <[email protected]>--- mm/migrate.c | 11 +++++++++++ 1 file changed, 11 insertions(+)mm/migrate.c +0
-0@@ -1393,6 +1393,17 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,if (old_page_state &PAGE_WAS_MAPPED )remove_migration_ptes (src, dst,0 );
mm/migrate.c static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · line 1405
Quoted 10 lines
mm/migrate.c +10
-0+/* +* Requeue the destination folio on the deferred split queue if +* the source was a large folio that was on the queue. Without +* this, NUMA migration causes underutilized THPs to escape +* the shrinker since the source is unqueued in +* __folio_migrate_mapping() and the destination is never +* re-queued. +*/ +if (folio_test_large (dst) &&folio_test_large_rmappable (dst)) +deferred_split_folio (dst, false);
Also, should you be checking for anon and non-device folios?
Signature · 3 lines
Cheers, David