Skip to contentSkip to conversation

mm: migrate: requeue destination folio on deferred split queue

PATCH Superseded Usama Arif 4 participants 10 messages Mar 6

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Usama Arif

author
#0 lore raw

PATCH 1/1 mm: migrate: requeue destination folio on deferred split queue

During folio migration, __folio_migrate_mapping() removes the source folio from the deferred split queue, but the destination folio is never re-queued. This causes underutilized THPs to escape the shrinker after NUMA migration, since they silently drop off the deferred split list.

Fix this by calling deferred_split_folio() on the destination folio after a successful migration, for large rmappable folios.

Fixes dafff3f4c850 ("mm: split underused THPs")
---
 mm/migrate.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
mm/migrate.c +11 -0
index ece77ccb2ec0..98d0a594f7b7 100644
@@ -1393,6 +1393,17 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 	if (old_page_state & PAGE_WAS_MAPPED)
 		remove_migration_ptes(src, dst, 0);
 
+	/*
+	 * Requeue the destination folio on the deferred split queue if
+	 * the source was a large folio that was on the queue. Without
+	 * this, NUMA migration causes underutilized THPs to escape
+	 * the shrinker since the source is unqueued in
+	 * __folio_migrate_mapping() and the destination is never
+	 * re-queued.
+	 */
+	if (folio_test_large(dst) && folio_test_large_rmappable(dst))
+		deferred_split_folio(dst, false);
+
 out_unlock_both:
 	folio_unlock(dst);
 	folio_set_owner_migrate_reason(dst, reason);
Signature · 1 line
2.47.3
#1 lore raw
Quoted 23 lines · Usama Arif

During folio migration, __folio_migrate_mapping() removes the source folio from the deferred split queue, but the destination folio is never re-queued. This causes underutilized THPs to escape the shrinker after NUMA migration, since they silently drop off the deferred split list.

Fix this by calling deferred_split_folio() on the destination folio after a successful migration, for large rmappable folios.

Fixes dafff3f4c850 ("mm: split underused THPs")
---
 mm/migrate.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
mm/migrate.c +0 -0
index ece77ccb2ec0..98d0a594f7b7 100644
@@ -1393,6 +1393,17 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 	if (old_page_state & PAGE_WAS_MAPPED)
 		remove_migration_ptes(src, dst, 0);
 

mm/migrate.c static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · line 1405

Quoted 10 lines
mm/migrate.c +10 -0
+	/*
+	 * Requeue the destination folio on the deferred split queue if
+	 * the source was a large folio that was on the queue. Without
+	 * this, NUMA migration causes underutilized THPs to escape
+	 * the shrinker since the source is unqueued in
+	 * __folio_migrate_mapping() and the destination is never
+	 * re-queued.
+	 */
+	if (folio_test_large(dst) && folio_test_large_rmappable(dst))
+		deferred_split_folio(dst, false);

Doesn't that mean that you will readd any large folios, even if already previously taken off the list after scanning?

So I am not sure if your "if the source was a large folio that was on the queue." comment is accurate?

Signature · 3 lines
Cheers,

David
author
#2 lore raw
Quoted 38 lines · David Hildenbrand (Arm)
Quoted 23 lines · Usama Arif

During folio migration, __folio_migrate_mapping() removes the source folio from the deferred split queue, but the destination folio is never re-queued. This causes underutilized THPs to escape the shrinker after NUMA migration, since they silently drop off the deferred split list.

Fix this by calling deferred_split_folio() on the destination folio after a successful migration, for large rmappable folios.

Fixes dafff3f4c850 ("mm: split underused THPs")
---
 mm/migrate.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
mm/migrate.c +0 -0
index ece77ccb2ec0..98d0a594f7b7 100644
@@ -1393,6 +1393,17 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 	if (old_page_state & PAGE_WAS_MAPPED)
 		remove_migration_ptes(src, dst, 0);
 

mm/migrate.c static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · line 1405

Quoted 10 lines
mm/migrate.c +10 -0
+	/*
+	 * Requeue the destination folio on the deferred split queue if
+	 * the source was a large folio that was on the queue. Without
+	 * this, NUMA migration causes underutilized THPs to escape
+	 * the shrinker since the source is unqueued in
+	 * __folio_migrate_mapping() and the destination is never
+	 * re-queued.
+	 */
+	if (folio_test_large(dst) && folio_test_large_rmappable(dst))
+		deferred_split_folio(dst, false);

Doesn't that mean that you will readd any large folios, even if already previously taken off the list after scanning?

Quoted 3 lines

So I am not sure if your "if the source was a large folio that was on the queue." comment is accurate?

Yes you are right. How about something like below? We also won't need to check for anon and non-device folios with this as we only set the the flag if it was already on deferred_split list.

mm/migrate.c +21 -1
index ece77ccb2ec0..9e0780d380e4 100644
@@ -1137,7 +1137,9 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
 enum {
        PAGE_WAS_MAPPED = BIT(0),
        PAGE_WAS_MLOCKED = BIT(1),
-       PAGE_OLD_STATES = PAGE_WAS_MAPPED | PAGE_WAS_MLOCKED,
+       PAGE_WAS_ON_DEFERRED_SPLIT = BIT(2),
+       PAGE_OLD_STATES = PAGE_WAS_MAPPED | PAGE_WAS_MLOCKED |
+                         PAGE_WAS_ON_DEFERRED_SPLIT,
 };
 
 static void __migrate_folio_record(struct folio *dst,
@@ -1373,6 +1375,15 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
                goto out_unlock_both;
        }
 
+       /*
+        * Record whether the source folio is on the deferred split queue
+        * before move_to_new_folio(), which unqueues it via
+        * __folio_migrate_mapping().
+        */
+       if (folio_test_large(src) && folio_test_large_rmappable(src) &&
+           !data_race(list_empty(&src->_deferred_list)))
+               old_page_state |= PAGE_WAS_ON_DEFERRED_SPLIT;
+
        rc = move_to_new_folio(dst, src, mode);
        if (rc)
                goto out;
@@ -1393,6 +1404,15 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
        if (old_page_state & PAGE_WAS_MAPPED)
                remove_migration_ptes(src, dst, 0);
 
+       /*
+        * Requeue the destination folio on the deferred split queue if
+        * the source was on the queue.  The source is unqueued in
+        * __folio_migrate_mapping(), so we record and check the state
+        * from before move_to_new_folio().
+        */
+       if (old_page_state & PAGE_WAS_ON_DEFERRED_SPLIT)
+               deferred_split_folio(dst, false);
+
 out_unlock_both:
        folio_unlock(dst);
        folio_set_owner_migrate_reason(dst, reason);
reviewer
#3 lore raw
Quoted 39 lines · Usama Arif
Quoted 38 lines · David Hildenbrand (Arm)
Quoted 23 lines · Usama Arif

During folio migration, __folio_migrate_mapping() removes the source folio from the deferred split queue, but the destination folio is never re-queued. This causes underutilized THPs to escape the shrinker after NUMA migration, since they silently drop off the deferred split list.

Fix this by calling deferred_split_folio() on the destination folio after a successful migration, for large rmappable folios.

Fixes dafff3f4c850 ("mm: split underused THPs")
---
 mm/migrate.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
mm/migrate.c +0 -0
index ece77ccb2ec0..98d0a594f7b7 100644
@@ -1393,6 +1393,17 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 	if (old_page_state & PAGE_WAS_MAPPED)
 		remove_migration_ptes(src, dst, 0);
 

mm/migrate.c static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · line 1405

Quoted 10 lines
mm/migrate.c +10 -0
+	/*
+	 * Requeue the destination folio on the deferred split queue if
+	 * the source was a large folio that was on the queue. Without
+	 * this, NUMA migration causes underutilized THPs to escape
+	 * the shrinker since the source is unqueued in
+	 * __folio_migrate_mapping() and the destination is never
+	 * re-queued.
+	 */
+	if (folio_test_large(dst) && folio_test_large_rmappable(dst))
+		deferred_split_folio(dst, false);

Doesn't that mean that you will readd any large folios, even if already previously taken off the list after scanning?

Quoted 7 lines
Quoted 3 lines

So I am not sure if your "if the source was a large folio that was on the queue." comment is accurate?

Yes you are right. How about something like below? We also won't need to check for anon and non-device folios with this as we only set the the flag if it was already on deferred_split list.

BTW, migrate_pages() tries to split partially mapped folios before migration[1], so what remains in the deferred_list would be:

1. partially mapped but with a pin,
2. fully mapped but potentially underused.

I wonder if you want to do an underused scan before migration and try to split underused THPs. Or to avoid this additional scan, find a way of detecting zero pages at page copy time and split it after migration.

Anyway, it seems that all large folios are in this deferred_list. Maybe, like David suggested in his LSFMM proposal, we should scan large folios on LRU lists at reclaim time instead, since there is not much difference between deferred_list and LRU lists right now.

[1] https://elixir.bootlin.com/linux/v6.19.3/source/mm/migrate.c#L1840

Quoted 39 lines
mm/migrate.c +14 -1
index ece77ccb2ec0..9e0780d380e4 100644
@@ -1137,7 +1137,9 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
 enum {
        PAGE_WAS_MAPPED = BIT(0),
        PAGE_WAS_MLOCKED = BIT(1),
-       PAGE_OLD_STATES = PAGE_WAS_MAPPED | PAGE_WAS_MLOCKED,
+       PAGE_WAS_ON_DEFERRED_SPLIT = BIT(2),
+       PAGE_OLD_STATES = PAGE_WAS_MAPPED | PAGE_WAS_MLOCKED |
+                         PAGE_WAS_ON_DEFERRED_SPLIT,
 };
 
 static void __migrate_folio_record(struct folio *dst,
@@ -1373,6 +1375,15 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
                goto out_unlock_both;
        }
 
+       /*
+        * Record whether the source folio is on the deferred split queue
+        * before move_to_new_folio(), which unqueues it via
+        * __folio_migrate_mapping().
+        */
+       if (folio_test_large(src) && folio_test_large_rmappable(src) &&
+           !data_race(list_empty(&src->_deferred_list)))
+               old_page_state |= PAGE_WAS_ON_DEFERRED_SPLIT;
+
        rc = move_to_new_folio(dst, src, mode);
        if (rc)
                goto out;
@@ -1393,6 +1404,15 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
        if (old_page_state & PAGE_WAS_MAPPED)
                remove_migration_ptes(src, dst, 0);
 
+       /*
+        * Requeue the destination folio on the deferred split queue if

mm/migrate.c static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · line 1418

Quoted 10 lines
mm/migrate.c +7 -0
+        * the source was on the queue.  The source is unqueued in
+        * __folio_migrate_mapping(), so we record and check the state
+        * from before move_to_new_folio().
+        */
+       if (old_page_state & PAGE_WAS_ON_DEFERRED_SPLIT)
+               deferred_split_folio(dst, false);
+
 out_unlock_both:
        folio_unlock(dst);
        folio_set_owner_migrate_reason(dst, reason);

Best Regards,
Yan, Zi

author
#4 lore raw
Quoted 49 lines · Zi Yan
Quoted 39 lines · Usama Arif
Quoted 38 lines · David Hildenbrand (Arm)
Quoted 23 lines · Usama Arif

During folio migration, __folio_migrate_mapping() removes the source folio from the deferred split queue, but the destination folio is never re-queued. This causes underutilized THPs to escape the shrinker after NUMA migration, since they silently drop off the deferred split list.

Fix this by calling deferred_split_folio() on the destination folio after a successful migration, for large rmappable folios.

Fixes dafff3f4c850 ("mm: split underused THPs")
---
 mm/migrate.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
mm/migrate.c +0 -0
index ece77ccb2ec0..98d0a594f7b7 100644
@@ -1393,6 +1393,17 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 	if (old_page_state & PAGE_WAS_MAPPED)
 		remove_migration_ptes(src, dst, 0);
 

mm/migrate.c static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · line 1405

Quoted 10 lines
mm/migrate.c +10 -0
+	/*
+	 * Requeue the destination folio on the deferred split queue if
+	 * the source was a large folio that was on the queue. Without
+	 * this, NUMA migration causes underutilized THPs to escape
+	 * the shrinker since the source is unqueued in
+	 * __folio_migrate_mapping() and the destination is never
+	 * re-queued.
+	 */
+	if (folio_test_large(dst) && folio_test_large_rmappable(dst))
+		deferred_split_folio(dst, false);

Doesn't that mean that you will readd any large folios, even if already previously taken off the list after scanning?

Quoted 7 lines
Quoted 3 lines

So I am not sure if your "if the source was a large folio that was on the queue." comment is accurate?

Yes you are right. How about something like below? We also won't need to check for anon and non-device folios with this as we only set the the flag if it was already on deferred_split list.

Quoted 6 lines

BTW, migrate_pages() tries to split partially mapped folios before migration[1], so what remains in the deferred_list would be:

1. partially mapped but with a pin,
2. fully mapped but potentially underused.

Yes, thats right.

Quoted 2 lines

I wonder if you want to do an underused scan before migration and try to split underused THPs.

hmm, I think we should keep THPs as is if there is no memory pressure (proactive or otherwise). Scanning THPs for zeros has a cost and we would also lose the benefit of THPs when we dont need memory.

Quoted 3 lines

Or to avoid this additional scan, find a way of detecting zero pages at page copy time and split it after migration.

Yeah but I think we lose the benefits of THPs after migration when we dont need additional memory?

Quoted 5 lines

Anyway, it seems that all large folios are in this deferred_list. Maybe, like David suggested in his LSFMM proposal, we should scan large folios on LRU lists at reclaim time instead, since there is not much difference between deferred_list and LRU lists right now.

Yeah the THP shrinker is a very basic implementation and there are a lot of

Quoted 3 lines

[1] https://elixir.bootlin.com/linux/v6.19.3/source/mm/migrate.c#L1840

Also Johannes pointed out its not great storing this information in page flags, we can just keep it as local variable. This is what the patch would look like:

mm/migrate.c +14 -0
index ece77ccb2ec0..48a972f158ab 100644
@@ -1360,6 +1360,7 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
        int rc;
        int old_page_state = 0;
        struct anon_vma *anon_vma = NULL;
+       bool src_deferred_split = false;
        struct list_head *prev;
 
        __migrate_folio_extract(dst, &old_page_state, &anon_vma);
@@ -1373,6 +1374,10 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
                goto out_unlock_both;
        }
 
+       if (folio_test_large(src) && folio_test_large_rmappable(src) &&
+           !data_race(list_empty(&src->_deferred_list)))
+               src_deferred_split = true;
+
        rc = move_to_new_folio(dst, src, mode);
        if (rc)
                goto out;
@@ -1393,6 +1398,15 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
        if (old_page_state & PAGE_WAS_MAPPED)
                remove_migration_ptes(src, dst, 0);
 
+       /*
+        * Requeue the destination folio on the deferred split queue if
+        * the source was on the queue.  The source is unqueued in
+        * __folio_migrate_mapping(), so we recorded the state from
+        * before move_to_new_folio().
+        */
+       if (src_deferred_split)
+               deferred_split_folio(dst, false);
+
 out_unlock_both:
        folio_unlock(dst);
        folio_set_owner_migrate_reason(dst, reason);
#5 lore raw
Quoted 39 lines · Usama Arif
Quoted 7 lines · Zi Yan
Quoted 4 lines · Usama Arif

Yes you are right. How about something like below? We also won't need to check for anon and non-device folios with this as we only set the the flag if it was already on deferred_split list.

Quoted 6 lines

BTW, migrate_pages() tries to split partially mapped folios before migration[1], so what remains in the deferred_list would be:

1. partially mapped but with a pin,
2. fully mapped but potentially underused.

Yes, thats right.

Quoted 2 lines

I wonder if you want to do an underused scan before migration and try to split underused THPs.

hmm, I think we should keep THPs as is if there is no memory pressure (proactive or otherwise). Scanning THPs for zeros has a cost and we would also lose the benefit of THPs when we dont need memory.

Quoted 3 lines

Or to avoid this additional scan, find a way of detecting zero pages at page copy time and split it after migration.

Yeah but I think we lose the benefits of THPs after migration when we dont need additional memory?

Quoted 5 lines

Anyway, it seems that all large folios are in this deferred_list. Maybe, like David suggested in his LSFMM proposal, we should scan large folios on LRU lists at reclaim time instead, since there is not much difference between deferred_list and LRU lists right now.

Quoted 8 lines

Yeah the THP shrinker is a very basic implementation and there are a lot of

Quoted 3 lines

[1] https://elixir.bootlin.com/linux/v6.19.3/source/mm/migrate.c#L1840

Also Johannes pointed out its not great storing this information in page flags, we can just keep it as local variable. This is what the patch would look like:

Much cleaner.

Signature · 3 lines
Cheers,

David
reviewer
#6 lore raw
In reply to Usama Arif
Quoted 57 lines · Usama Arif
Quoted 49 lines · Zi Yan
Quoted 39 lines · Usama Arif
Quoted 38 lines · David Hildenbrand (Arm)
Quoted 23 lines · Usama Arif

During folio migration, __folio_migrate_mapping() removes the source folio from the deferred split queue, but the destination folio is never re-queued. This causes underutilized THPs to escape the shrinker after NUMA migration, since they silently drop off the deferred split list.

Fix this by calling deferred_split_folio() on the destination folio after a successful migration, for large rmappable folios.

Fixes dafff3f4c850 ("mm: split underused THPs")
---
 mm/migrate.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
mm/migrate.c +0 -0
index ece77ccb2ec0..98d0a594f7b7 100644
@@ -1393,6 +1393,17 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 	if (old_page_state & PAGE_WAS_MAPPED)
 		remove_migration_ptes(src, dst, 0);
 

mm/migrate.c static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · line 1405

Quoted 10 lines
mm/migrate.c +10 -0
+	/*
+	 * Requeue the destination folio on the deferred split queue if
+	 * the source was a large folio that was on the queue. Without
+	 * this, NUMA migration causes underutilized THPs to escape
+	 * the shrinker since the source is unqueued in
+	 * __folio_migrate_mapping() and the destination is never
+	 * re-queued.
+	 */
+	if (folio_test_large(dst) && folio_test_large_rmappable(dst))
+		deferred_split_folio(dst, false);

Doesn't that mean that you will readd any large folios, even if already previously taken off the list after scanning?

Quoted 7 lines
Quoted 3 lines

So I am not sure if your "if the source was a large folio that was on the queue." comment is accurate?

Yes you are right. How about something like below? We also won't need to check for anon and non-device folios with this as we only set the the flag if it was already on deferred_split list.

Quoted 6 lines

BTW, migrate_pages() tries to split partially mapped folios before migration[1], so what remains in the deferred_list would be:

1. partially mapped but with a pin,
2. fully mapped but potentially underused.

Quoted 8 lines

Yes, thats right.

Quoted 2 lines

I wonder if you want to do an underused scan before migration and try to split underused THPs.

hmm, I think we should keep THPs as is if there is no memory pressure (proactive or otherwise). Scanning THPs for zeros has a cost and we would also lose the benefit of THPs when we dont need memory.

Makes sense.

Quoted 7 lines
Quoted 4 lines

Or to avoid this additional scan, find a way of detecting zero pages at page copy time and split it after migration.

Yeah but I think we lose the benefits of THPs after migration when we dont need additional memory?

Right.

Quoted 46 lines
Quoted 6 lines

Anyway, it seems that all large folios are in this deferred_list. Maybe, like David suggested in his LSFMM proposal, we should scan large folios on LRU lists at reclaim time instead, since there is not much difference between deferred_list and LRU lists right now.

Yeah the THP shrinker is a very basic implementation and there are a lot of

Quoted 3 lines

[1] https://elixir.bootlin.com/linux/v6.19.3/source/mm/migrate.c#L1840

Also Johannes pointed out its not great storing this information in page flags, we can just keep it as local variable. This is what the patch would look like:

mm/migrate.c +7 -0
index ece77ccb2ec0..48a972f158ab 100644
@@ -1360,6 +1360,7 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
        int rc;
        int old_page_state = 0;
        struct anon_vma *anon_vma = NULL;
+       bool src_deferred_split = false;
        struct list_head *prev;
 
        __migrate_folio_extract(dst, &old_page_state, &anon_vma);
@@ -1373,6 +1374,10 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
                goto out_unlock_both;
        }
 
+       if (folio_test_large(src) && folio_test_large_rmappable(src) &&
+           !data_race(list_empty(&src->_deferred_list)))
+               src_deferred_split = true;
+
        rc = move_to_new_folio(dst, src, mode);
        if (rc)
                goto out;
@@ -1393,6 +1398,15 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
        if (old_page_state & PAGE_WAS_MAPPED)
                remove_migration_ptes(src, dst, 0);
 
+       /*
+        * Requeue the destination folio on the deferred split queue if

mm/migrate.c static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · line 1412

Quoted 10 lines
mm/migrate.c +7 -0
+        * the source was on the queue.  The source is unqueued in
+        * __folio_migrate_mapping(), so we recorded the state from
+        * before move_to_new_folio().
+        */
+       if (src_deferred_split)
+               deferred_split_folio(dst, false);
+
 out_unlock_both:
        folio_unlock(dst);
        folio_set_owner_migrate_reason(dst, reason);

LGTM. Thanks for improving it.

Best Regards,
Yan, Zi

#7 lore raw
In reply to Usama Arif

mm/migrate.c static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · line 1384

Quoted 7 lines · Usama Arif
mm/migrate.c +7 -0
+       /*
+        * Record whether the source folio is on the deferred split queue
+        * before move_to_new_folio(), which unqueues it via
+        * __folio_migrate_mapping().
+        */
+       if (folio_test_large(src) && folio_test_large_rmappable(src) &&
+           !data_race(list_empty(&src->_deferred_list)))

Why do you need data_race() here? list_empty() contains a READ_ONCE() so shouldn't be necessary?

Quoted 1 line

+ old_page_state |= PAGE_WAS_ON_DEFERRED_SPLIT;

You've done a great job of the naming. So much so that the comment seems entirely unnecessary?

mm/migrate.c static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · line 1414

Quoted 8 lines
mm/migrate.c +8 -0
+       /*
+        * Requeue the destination folio on the deferred split queue if
+        * the source was on the queue.  The source is unqueued in
+        * __folio_migrate_mapping(), so we record and check the state
+        * from before move_to_new_folio().
+        */
+       if (old_page_state & PAGE_WAS_ON_DEFERRED_SPLIT)
+               deferred_split_folio(dst, false);

Again, I'm not sure the comment says anything that the code doesn't?

author
#8 lore raw
Quoted 11 lines · Matthew Wilcox

mm/migrate.c static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · line 1384

Quoted 7 lines · Usama Arif
mm/migrate.c +7 -0
+       /*
+        * Record whether the source folio is on the deferred split queue
+        * before move_to_new_folio(), which unqueues it via
+        * __folio_migrate_mapping().
+        */
+       if (folio_test_large(src) && folio_test_large_rmappable(src) &&
+           !data_race(list_empty(&src->_deferred_list)))

Why do you need data_race() here? list_empty() contains a READ_ONCE() so shouldn't be necessary?

Ah mainly because we dont acquire split_queue_lock before accessing, similar to what we do in folio_unqueue_deferred_split().

Quoted 15 lines
Quoted 2 lines

+ old_page_state |= PAGE_WAS_ON_DEFERRED_SPLIT;

You've done a great job of the naming. So much so that the comment seems entirely unnecessary?

mm/migrate.c static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · line 1414

Quoted 8 lines
mm/migrate.c +8 -0
+       /*
+        * Requeue the destination folio on the deferred split queue if
+        * the source was on the queue.  The source is unqueued in
+        * __folio_migrate_mapping(), so we record and check the state
+        * from before move_to_new_folio().
+        */
+       if (old_page_state & PAGE_WAS_ON_DEFERRED_SPLIT)
+               deferred_split_folio(dst, false);
Quoted 1 line

Again, I'm not sure the comment says anything that the code doesn't?

Yeah there is a much simpler version in reply to Zi's review in [1]. I found the whole migrate_folio_move() function quite verbose already, was keeping up with the theme of that function lol. I will see if I can cut down on the comment in [1] as well. Thanks!

[1] https://lore.kernel.org/all/[email protected]/

#9 lore raw
In reply to Usama Arif
Quoted 23 lines · Usama Arif

During folio migration, __folio_migrate_mapping() removes the source folio from the deferred split queue, but the destination folio is never re-queued. This causes underutilized THPs to escape the shrinker after NUMA migration, since they silently drop off the deferred split list.

Fix this by calling deferred_split_folio() on the destination folio after a successful migration, for large rmappable folios.

Fixes dafff3f4c850 ("mm: split underused THPs")
---
 mm/migrate.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
mm/migrate.c +0 -0
index ece77ccb2ec0..98d0a594f7b7 100644
@@ -1393,6 +1393,17 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 	if (old_page_state & PAGE_WAS_MAPPED)
 		remove_migration_ptes(src, dst, 0);
 

mm/migrate.c static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · line 1405

Quoted 10 lines
mm/migrate.c +10 -0
+	/*
+	 * Requeue the destination folio on the deferred split queue if
+	 * the source was a large folio that was on the queue. Without
+	 * this, NUMA migration causes underutilized THPs to escape
+	 * the shrinker since the source is unqueued in
+	 * __folio_migrate_mapping() and the destination is never
+	 * re-queued.
+	 */
+	if (folio_test_large(dst) && folio_test_large_rmappable(dst))
+		deferred_split_folio(dst, false);

Also, should you be checking for anon and non-device folios?

Signature · 3 lines
Cheers,

David