mm: vmscan: put rotation-missed folios at the LRU tail
Subsystems (3)
PATCH 1/1 mm: vmscan: put rotation-missed folios at the LRU tail
From: Ridong Chen <[email protected]>
The page reclaim isolates a batch of folios from the tail of an LRU list and works on them one by one. For a suitable swap-backed folio on an async swap device, it queues the folio for writeback and, after finishing the batch, puts the folio back to the head of the original LRU list.
Meanwhile the page writeback flushes the queued folios in its own, independent batches. For each folio it writes back it calls folio_rotate_reclaimable(), which tries to rotate the folio to the LRU tail. But folio_rotate_reclaimable() only takes effect once the folio has been put back by reclaim. If the async swap device is fast enough, the writeback can complete a folio while reclaim is still working on the rest of the batch that contains it. In that case the folio stays near the head and reclaim will not revisit it before wrapping around, causing a cold/hot
inversion: a clean, written-back folio that should be a prime reclaim candidate is kept ahead of hotter folios.
commit 359a5e1416ca ("mm: multi-gen LRU: retry folios written back while isolated") addressed this for MGLRU only. The traditional active/inactive LRU has the same problem, reported at [1]. A reproducer is available at [2].
Rather than re-reclaiming those folios (which would drop the swap cache that may still be useful for a future hit [4]), restore the rotation that was missed: when move_folios_to_lru() puts a folio back, add it to the LRU tail if it looks like it missed folio_rotate_reclaimable() (inactive, not mapped, not dirty and not under writeback). A new do_rotate parameter gates this so it only applies on the reclaim put-back path (shrink_inactive_list()), not on shrink_active_list() where the list order is already deliberate. This approach was suggested by Barry Song [3].
Only the traditional LRU is handled here. MGLRU already retries such folios via its own clean-list retry pass in evict_folios(), so it is left unchanged. The same do_rotate scheme could later replace that retry pass to unify both LRUs, which is left for a follow-up.
[1] https://lore.kernel.org/linux-kernel/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/CAGsJ_4zwP3_+EYY5Ug9EJ+yD1UdxsBSGr25u8s1K3u_i7LH3Zg@mail.gmail.com/
[4] https://lore.kernel.org/linux-mm/[email protected]/
--- mm/vmscan.c | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-)
mm/vmscan.c +15 -6
@@ -1971,7 +1971,7 @@ static bool too_many_isolated(struct pglist_data *pgdat, int file,* * Note: The caller must not hold any lruvec lock. */ -static unsigned int move_folios_to_lru (struct list_head *list) +static unsigned int move_folios_to_lru (struct list_head *list,bool do_rotate) {int nr_pages, nr_moved =0 ;struct lruvec *lruvec =NULL ;
@@ -2018,7 +2018,16 @@ static unsigned int move_folios_to_lru(struct list_head *list)continue ; } -lruvec_add_folio (lruvec, folio); +/* +* Put folios that may have missed folio_rotate_reclaimable() +* at the tail to avoid cold/hot inversion. +*/ +if (do_rotate && !folio_test_active (folio) && !folio_mapped (folio) && + !folio_test_dirty (folio) && !folio_test_writeback (folio)) +lruvec_add_folio_tail (lruvec, folio); +else +lruvec_add_folio (lruvec, folio); + nr_pages =folio_nr_pages (folio); nr_moved += nr_pages;if (folio_test_active (folio))
@@ -2135,7 +2144,7 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan, nr_reclaimed =shrink_folio_list (&folio_list, pgdat, sc, &stat, false,lruvec_memcg (lruvec)); -move_folios_to_lru (&folio_list); +move_folios_to_lru (&folio_list, true);mod_lruvec_state (lruvec,PGDEMOTE_KSWAPD +reclaimer_offset (sc), stat.nr_demoted);
@@ -2246,8 +2255,8 @@ static void shrink_active_list(unsigned long nr_to_scan,/* * Move folios back to the lru list. */ - nr_activate =move_folios_to_lru (&l_active); - nr_deactivate =move_folios_to_lru (&l_inactive); + nr_activate =move_folios_to_lru (&l_active, false); + nr_deactivate =move_folios_to_lru (&l_inactive, false);count_vm_events (PGDEACTIVATE , nr_deactivate);count_memcg_events (lruvec_memcg (lruvec),PGDEACTIVATE , nr_deactivate);
@@ -5115,7 +5124,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,folio_set_active (folio); } -move_folios_to_lru (&list); +move_folios_to_lru (&list, false); walk = current->reclaim_state->mm_walk;if (walk && walk->batched) {
Signature · 1 line
2.34.1
Quoted 23 lines · Ridong Chen
From: Ridong Chen <[email protected]>
The page reclaim isolates a batch of folios from the tail of an LRU list and works on them one by one. For a suitable swap-backed folio on an async swap device, it queues the folio for writeback and, after finishing the batch, puts the folio back to the head of the original LRU list.
Meanwhile the page writeback flushes the queued folios in its own, independent batches. For each folio it writes back it calls
folio_rotate_reclaimable(), which tries to rotate the folio to the LRU tail. Butfolio_rotate_reclaimable()only takes effect once the folio has been put back by reclaim. If the async swap device is fast enough, the writeback can complete a folio while reclaim is still working on the rest of the batch that contains it. In that case the folio stays near the head and reclaim will not revisit it before wrapping around, causing a cold/hot
inversion: a clean, written-back folio that should be a prime reclaim candidate is kept ahead of hotter folios.commit
359a5e1416ca("mm: multi-gen LRU: retry folios written back while isolated") addressed this for MGLRU only. The traditional active/inactive LRU has the same problem, reported at [1]. A reproducer is available at [2].
Quoted 9 lines
Rather than re-reclaiming those folios (which would drop the swap cache that may still be useful for a future hit [4]), restore the rotation that was missed: when
move_folios_to_lru()puts a folio back, add it to the LRU tail if it looks like it missedfolio_rotate_reclaimable()(inactive, not mapped, not dirty and not under writeback). A newdo_rotateparameter gates this so it only applies on the reclaim put-back path (shrink_inactive_list()), not onshrink_active_list()where the list order is already deliberate. This approach was suggested by Barry Song [3].
Add test result:
```
Without patch:
cat memory.usage_in_bytes
1073700864
cat memory.memsw.usage_in_bytes
1413124096
free -h
total used free
Mem: 1.6Gi 1.2Gi 299Mi
Swap: 1.0Gi 678Mi 346Mi
With patch:
cat memory.usage_in_bytes
1071140864 bytes
cat memory.memsw.usage_in_bytes
1413423104 bytes
free -h
total used free
Mem: 1.6Gi 1.2Gi 322Mi
Swap: 1.0Gi 328Mi 695Mi
```After applying the patch, the difference between memory.memsw.usage_in_bytes and memory.usage_in_bytes is close to the swap "used" value reported by 'free -h;.
This yields the same result as the 'retry' approach in v9 [1].
[1] https://lore.kernel.org/lkml/[email protected]/
Quoted 76 lines
Only the traditional LRU is handled here. MGLRU already retries such folios via its own clean-list retry pass in
evict_folios(), so it is left unchanged. The samedo_rotatescheme could later replace that retry pass to unify both LRUs, which is left for a follow-up.[1] https://lore.kernel.org/linux-kernel/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/CAGsJ_4zwP3_+EYY5Ug9EJ+yD1UdxsBSGr25u8s1K3u_i7LH3Zg@mail.gmail.com/
[4] https://lore.kernel.org/linux-mm/[email protected]/Suggested-by Barry Song <[email protected]>Signed-off-by Ridong Chen <[email protected]>--- mm/vmscan.c | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-)mm/vmscan.c +15
-6@@ -1971,7 +1971,7 @@ static bool too_many_isolated(struct pglist_data *pgdat, int file,* * Note: The caller must not hold any lruvec lock. */ -static unsigned int move_folios_to_lru (struct list_head *list) +static unsigned int move_folios_to_lru (struct list_head *list,bool do_rotate) {int nr_pages, nr_moved =0 ;struct lruvec *lruvec =NULL ;@@ -2018,7 +2018,16 @@ static unsigned int move_folios_to_lru(struct list_head *list)continue ; } -lruvec_add_folio (lruvec, folio); +/* +* Put folios that may have missed folio_rotate_reclaimable() +* at the tail to avoid cold/hot inversion. +*/ +if (do_rotate && !folio_test_active (folio) && !folio_mapped (folio) && + !folio_test_dirty (folio) && !folio_test_writeback (folio)) +lruvec_add_folio_tail (lruvec, folio); +else +lruvec_add_folio (lruvec, folio); + nr_pages =folio_nr_pages (folio); nr_moved += nr_pages;if (folio_test_active (folio))@@ -2135,7 +2144,7 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan, nr_reclaimed =shrink_folio_list (&folio_list, pgdat, sc, &stat, false,lruvec_memcg (lruvec)); -move_folios_to_lru (&folio_list); +move_folios_to_lru (&folio_list, true);mod_lruvec_state (lruvec,PGDEMOTE_KSWAPD +reclaimer_offset (sc), stat.nr_demoted);@@ -2246,8 +2255,8 @@ static void shrink_active_list(unsigned long nr_to_scan,/* * Move folios back to the lru list. */ - nr_activate =move_folios_to_lru (&l_active); - nr_deactivate =move_folios_to_lru (&l_inactive); + nr_activate =move_folios_to_lru (&l_active, false); + nr_deactivate =move_folios_to_lru (&l_inactive, false);count_vm_events (PGDEACTIVATE , nr_deactivate);count_memcg_events (lruvec_memcg (lruvec),PGDEACTIVATE , nr_deactivate);@@ -5115,7 +5124,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,folio_set_active (folio); } -move_folios_to_lru (&list); +move_folios_to_lru (&list, false); walk = current->reclaim_state->mm_walk;if (walk && walk->batched) {
Signature · 2 lines
Best regards Ridong
Barry Song
reviewerQuoted 64 lines · Ridong Chen
From: Ridong Chen <[email protected]>
The page reclaim isolates a batch of folios from the tail of an LRU list and works on them one by one. For a suitable swap-backed folio on an async swap device, it queues the folio for writeback and, after finishing the batch, puts the folio back to the head of the original LRU list.
Meanwhile the page writeback flushes the queued folios in its own, independent batches. For each folio it writes back it calls
folio_rotate_reclaimable(), which tries to rotate the folio to the LRU tail. Butfolio_rotate_reclaimable()only takes effect once the folio has been put back by reclaim. If the async swap device is fast enough, the writeback can complete a folio while reclaim is still working on the rest of the batch that contains it. In that case the folio stays near the head and reclaim will not revisit it before wrapping around, causing a cold/hot
inversion: a clean, written-back folio that should be a prime reclaim candidate is kept ahead of hotter folios.commit
359a5e1416ca("mm: multi-gen LRU: retry folios written back while isolated") addressed this for MGLRU only. The traditional active/inactive LRU has the same problem, reported at [1]. A reproducer is available at [2].Rather than re-reclaiming those folios (which would drop the swap cache that may still be useful for a future hit [4]), restore the rotation that was missed: when
move_folios_to_lru()puts a folio back, add it to the LRU tail if it looks like it missedfolio_rotate_reclaimable()(inactive, not mapped, not dirty and not under writeback). A newdo_rotateparameter gates this so it only applies on the reclaim put-back path (shrink_inactive_list()), not onshrink_active_list()where the list order is already deliberate. This approach was suggested by Barry Song [3].Only the traditional LRU is handled here. MGLRU already retries such folios via its own clean-list retry pass in
evict_folios(), so it is left unchanged. The samedo_rotatescheme could later replace that retry pass to unify both LRUs, which is left for a follow-up.[1] https://lore.kernel.org/linux-kernel/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/CAGsJ_4zwP3_+EYY5Ug9EJ+yD1UdxsBSGr25u8s1K3u_i7LH3Zg@mail.gmail.com/
[4] https://lore.kernel.org/linux-mm/[email protected]/Suggested-by Barry Song <[email protected]>Signed-off-by Ridong Chen <[email protected]>--- mm/vmscan.c | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-)mm/vmscan.c +1
-1@@ -1971,7 +1971,7 @@ static bool too_many_isolated(struct pglist_data *pgdat, int file,* * Note: The caller must not hold any lruvec lock. */ -static unsigned int move_folios_to_lru (struct list_head *list) +static unsigned int move_folios_to_lru (struct list_head *list,bool do_rotate) {int nr_pages, nr_moved =0 ;struct lruvec *lruvec =NULL ;@@ -2018,7 +2018,16 @@ static unsigned int move_folios_to_lru(struct list_head *list)continue ;
mm/vmscan.c static unsigned int move_folios_to_lru(struct list_head *list) · line 2027
Quoted 10 lines
mm/vmscan.c +7
-1} -lruvec_add_folio (lruvec, folio); +/* +* Put folios that may have missed folio_rotate_reclaimable() +* at the tail to avoid cold/hot inversion. +*/ +if (do_rotate && !folio_test_active (folio) && !folio_mapped (folio) && + !folio_test_dirty (folio) && !folio_test_writeback (folio)) +lruvec_add_folio_tail (lruvec, folio);
sashiko says:
"Could this heuristic indiscriminately place un-reclaimable clean folios at the LRU tail, ensuring they are immediately re-scanned in an infinite loop? If shrink_inactive_list() isolates a clean, unmapped file folio with an elevated refcount (such as from a concurrent GUP or speculative lookup), it will fail __remove_mapping() and be placed on ret_folios. Because this pinned folio matches the !active, !mapped, !dirty, and !writeback checks, it is placed at the tail of the inactive LRU. The very next isolate_lru_folios() pull from the tail will immediately isolate this exact same folio again, creating an infinite loop that prevents any other folios from being reclaimed and causes kswapd to spin at 100% CPU."
This seems to be a valid concern. For a clean and unmapped folio, we may still fail in __remove_mapping() if it is pinned somewhere. So perhaps we can add a refcount check when fixing the rotation.
Best Regards
Barry
Quoted 89 lines · Barry Song
Quoted 64 lines · Ridong Chen
From: Ridong Chen <[email protected]>
The page reclaim isolates a batch of folios from the tail of an LRU list and works on them one by one. For a suitable swap-backed folio on an async swap device, it queues the folio for writeback and, after finishing the batch, puts the folio back to the head of the original LRU list.
Meanwhile the page writeback flushes the queued folios in its own, independent batches. For each folio it writes back it calls
folio_rotate_reclaimable(), which tries to rotate the folio to the LRU tail. Butfolio_rotate_reclaimable()only takes effect once the folio has been put back by reclaim. If the async swap device is fast enough, the writeback can complete a folio while reclaim is still working on the rest of the batch that contains it. In that case the folio stays near the head and reclaim will not revisit it before wrapping around, causing a cold/hot
inversion: a clean, written-back folio that should be a prime reclaim candidate is kept ahead of hotter folios.commit
359a5e1416ca("mm: multi-gen LRU: retry folios written back while isolated") addressed this for MGLRU only. The traditional active/inactive LRU has the same problem, reported at [1]. A reproducer is available at [2].Rather than re-reclaiming those folios (which would drop the swap cache that may still be useful for a future hit [4]), restore the rotation that was missed: when
move_folios_to_lru()puts a folio back, add it to the LRU tail if it looks like it missedfolio_rotate_reclaimable()(inactive, not mapped, not dirty and not under writeback). A newdo_rotateparameter gates this so it only applies on the reclaim put-back path (shrink_inactive_list()), not onshrink_active_list()where the list order is already deliberate. This approach was suggested by Barry Song [3].Only the traditional LRU is handled here. MGLRU already retries such folios via its own clean-list retry pass in
evict_folios(), so it is left unchanged. The samedo_rotatescheme could later replace that retry pass to unify both LRUs, which is left for a follow-up.[1] https://lore.kernel.org/linux-kernel/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/CAGsJ_4zwP3_+EYY5Ug9EJ+yD1UdxsBSGr25u8s1K3u_i7LH3Zg@mail.gmail.com/
[4] https://lore.kernel.org/linux-mm/[email protected]/Suggested-by Barry Song <[email protected]>Signed-off-by Ridong Chen <[email protected]>--- mm/vmscan.c | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-)mm/vmscan.c +1
-1@@ -1971,7 +1971,7 @@ static bool too_many_isolated(struct pglist_data *pgdat, int file,* * Note: The caller must not hold any lruvec lock. */ -static unsigned int move_folios_to_lru (struct list_head *list) +static unsigned int move_folios_to_lru (struct list_head *list,bool do_rotate) {int nr_pages, nr_moved =0 ;struct lruvec *lruvec =NULL ;@@ -2018,7 +2018,16 @@ static unsigned int move_folios_to_lru(struct list_head *list)continue ;mm/vmscan.c static unsigned int move_folios_to_lru(struct list_head *list) · line 2027
Quoted 10 lines
mm/vmscan.c +7
-1} -lruvec_add_folio (lruvec, folio); +/* +* Put folios that may have missed folio_rotate_reclaimable() +* at the tail to avoid cold/hot inversion. +*/ +if (do_rotate && !folio_test_active (folio) && !folio_mapped (folio) && + !folio_test_dirty (folio) && !folio_test_writeback (folio)) +lruvec_add_folio_tail (lruvec, folio);sashiko says:
"Could this heuristic indiscriminately place un-reclaimable clean folios at the LRU tail, ensuring they are immediately re-scanned in an infinite loop? If
shrink_inactive_list()isolates a clean, unmapped file folio with an elevated refcount (such as from a concurrent GUP or speculative lookup), it will fail__remove_mapping()and be placed onret_folios. Because this pinned folio matches the !active, !mapped, !dirty, and !writeback checks, it is placed at the tail of the inactive LRU. The very nextisolate_lru_folios()pull from the tail will immediately isolate this exact same folio again, creating an infinite loop that prevents any other folios from being reclaimed and causes kswapd to spin at 100% CPU."
Quoted 4 lines
This seems to be a valid concern. For a clean and unmapped folio, we may still fail in
__remove_mapping()if it is pinned somewhere. So perhaps we can add a refcount check when fixing the rotation.
Thanks, I will update.
Signature · 2 lines
Best regards Ridong