Skip to contentSkip to conversation

mm/mglru: Avoid reclaiming kept folios during retrying missed rotated folios

RFC PATCH In review Barry Song (Xiaomi) 3 participants 6 messages Oct 3 – Oct 14

Subsystems (4)
lore
Share & tools
lore

Barry Song (Xiaomi)

author reviewer
#0 lore raw

RFC PATCH 1/1 mm/mglru: Avoid reclaiming kept folios during retrying missed rotated folios

Since commit 4d5d14a01e2c ("mm/mglru: rework workingset protection"), folio_test_referenced() was accidentally removed when checking for missed rotated folios. As a result, kept folios without an active set could be added to the retry list and eventually reclaimed.

The impact should be very small, as we have the folio_mapped(folio) check before retrying. Kept folios won't have try_to_unmap() called, so this is really only applicable to folios that are unmapped by users after folio_referenced() has been called.

I'm not sure if anyone else has seen any issues with this. Sending an RFC to check whether it is worth fixing.

Fixes 4d5d14a01e2c ("mm/mglru: rework workingset protection")
Cc Yu Zhao <[email protected]>
---
 mm/vmscan.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
mm/vmscan.c +5 -2
index 91295070ca33..6a931b576b7d 100644
@@ -994,8 +994,10 @@ static enum folio_references folio_check_references(struct folio *folio,
 		return FOLIOREF_KEEP;
 
 	if (lru_gen_enabled() && !lru_gen_switching()) {
-		if (!referenced_ptes)
+		if (!referenced_ptes) {
+			folio_clear_referenced(folio);
 			return FOLIOREF_RECLAIM;
+		}
 
 		return lru_gen_set_refs(folio, &vma_flags) ? FOLIOREF_ACTIVATE : FOLIOREF_KEEP;
 	}
@@ -5110,7 +5112,8 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 			continue;
 
 		/* retry folios that may have missed folio_rotate_reclaimable() */
-		if (!skip_retry && !folio_test_active(folio) && !folio_mapped(folio) &&
+		if (!skip_retry && !folio_test_active(folio) &&
+		    !folio_test_referenced(folio) && !folio_mapped(folio) &&
 		    !folio_test_dirty(folio) && !folio_test_writeback(folio)) {
 			list_move(&folio->lru, &clean);
 			continue;
Signature · 1 line
2.39.3 (Apple Git-146)
reviewer
#1 lore raw

Hi Barry,

Quoted 26 lines · Barry Song (Xiaomi)

Since commit 4d5d14a01e2c ("mm/mglru: rework workingset protection"), folio_test_referenced() was accidentally removed when checking for missed rotated folios. As a result, kept folios without an active set could be added to the retry list and eventually reclaimed.

The impact should be very small, as we have the folio_mapped(folio) check before retrying. Kept folios won't have try_to_unmap() called, so this is really only applicable to folios that are unmapped by users after folio_referenced() has been called.

I'm not sure if anyone else has seen any issues with this. Sending an RFC to check whether it is worth fixing.

Fixes 4d5d14a01e2c ("mm/mglru: rework workingset protection")
Cc Yu Zhao <[email protected]>
---
  mm/vmscan.c | 7 +++++--
  1 file changed, 5 insertions(+), 2 deletions(-)
mm/vmscan.c +0 -0
index 91295070ca33..6a931b576b7d 100644

mm/vmscan.c static enum folio_references folio_check_references(struct folio *folio, · line 1000

Quoted 8 lines
mm/vmscan.c +3 -1
@@ -994,8 +994,10 @@ static enum folio_references folio_check_references(struct folio *folio,
  		return FOLIOREF_KEEP;
 
  	if (lru_gen_enabled() && !lru_gen_switching()) {
-		if (!referenced_ptes)
+		if (!referenced_ptes) {
+			folio_clear_referenced(folio);
  			return FOLIOREF_RECLAIM;
+		}

I've also been investigating the issue of referenced folios being reclaimed recently. The background is that classical LRU will return FOLIOREF_KEEP for folios accessed once via page table, giving the accessed folio another chance.

However, for MGLRU, if the folio's pagetable access has already been consumed by aging or lru_gen_look_around(), referenced_ptes may be 0 at that point, leading to reclaiming folios that were accessed via page table, which could cause more refaults (I don't have data yet).

But it's not easy to maintain logic consistent with classical LRU, because we cannot distinguish whether this one access count comes from page table or file descriptors (for anonymous pages, it can be distinguished). I'm not sure if Kairui's MGLRU-FG patchset helps with this case (I haven't looked at it carefully yet).

Perhaps at least for anonymous pages, if the referenced flag is set, we should return FOLIOREF_KEEP to give it another scan chance.

mm/vmscan.c static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · line 5116

Quoted 9 lines
mm/vmscan.c +2 -1
  		return lru_gen_set_refs(folio, &vma_flags) ? FOLIOREF_ACTIVATE : FOLIOREF_KEEP;
  	}
@@ -5110,7 +5112,8 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
  			continue;
 
  		/* retry folios that may have missed folio_rotate_reclaimable() */
-		if (!skip_retry && !folio_test_active(folio) && !folio_mapped(folio) &&
+		if (!skip_retry && !folio_test_active(folio) &&
+		    !folio_test_referenced(folio) && !folio_mapped(folio) &&

This part makes sense to me.

Quoted 3 lines
mm/vmscan.c +0 -0
  		    !folio_test_dirty(folio) && !folio_test_writeback(folio)) {
  			list_move(&folio->lru, &clean);
  			continue;
#2 lore raw
Quoted 33 lines · Baolin Wang

Hi Barry,

Quoted 30 lines · Barry Song (Xiaomi)

Since commit 4d5d14a01e2c ("mm/mglru: rework workingset protection"), folio_test_referenced() was accidentally removed when checking for missed rotated folios. As a result, kept folios without an active set could be added to the retry list and eventually reclaimed.

The impact should be very small, as we have the folio_mapped(folio) check before retrying. Kept folios won't have try_to_unmap() called, so this is really only applicable to folios that are unmapped by users after folio_referenced() has been called.

I'm not sure if anyone else has seen any issues with this. Sending an RFC to check whether it is worth fixing.

Fixes 4d5d14a01e2c ("mm/mglru: rework workingset protection")
Cc Yu Zhao <[email protected]>
---
  mm/vmscan.c | 7 +++++--
  1 file changed, 5 insertions(+), 2 deletions(-)
mm/vmscan.c +1 -1
index 91295070ca33..6a931b576b7d 100644
@@ -994,8 +994,10 @@ static enum folio_references folio_check_references(struct folio *folio,
  		return FOLIOREF_KEEP;
  	if (lru_gen_enabled() && !lru_gen_switching()) {
-		if (!referenced_ptes)
+		if (!referenced_ptes) {
Quoted 8 lines

mm/vmscan.c static enum folio_references folio_check_references(struct folio *folio, · line 1000

Quoted 3 lines
mm/vmscan.c +2 -0
+			folio_clear_referenced(folio);
  			return FOLIOREF_RECLAIM;
+		}

I've also been investigating the issue of referenced folios being reclaimed recently. The background is that classical LRU will return FOLIOREF_KEEP for folios accessed once via page table, giving the accessed folio another chance.

Hi, thanks for looking into this.

Quoted 5 lines

However, for MGLRU, if the folio's pagetable access has already been consumed by aging or lru_gen_look_around(), referenced_ptes may be 0 at that point, leading to reclaiming folios that were accessed via page table, which could cause more refaults (I don't have data yet).

If we go with MGLRU-FG, aging or lru_gen_look_around may move the folio to newer gen which is identical to the behavior of promoting at eviction time.

Quoted 5 lines

But it's not easy to maintain logic consistent with classical LRU, because we cannot distinguish whether this one access count comes from page table or file descriptors (for anonymous pages, it can be distinguished). I'm not sure if Kairui's MGLRU-FG patchset helps with this case (I haven't looked at it carefully yet).

It definatly helps, check this one:
https://lore.kernel.org/linux-mm/[email protected]/

folio_inc_lru_refs and variants will promote the folios in a consistent way. See "First access only defers eviction from the oldest gen" part, I believe that is the issue you worries about?

BTW promote then to newest gen directly does over protect them, I've see regression with that.

Quoted 3 lines

Perhaps at least for anonymous pages, if the referenced flag is set, we should return FOLIOREF_KEEP to give it another scan chance.

No I think we really shouldn't do that, it will over protect single used folios. Promoting them to second newest gen or second oldest gen seems a good idea and similiar to CLRU, and this is the behavior of MGLRU-FG.

#3 lore raw
Quoted 73 lines · Kairui Song
Quoted 33 lines · Baolin Wang

Hi Barry,

Quoted 30 lines · Barry Song (Xiaomi)

Since commit 4d5d14a01e2c ("mm/mglru: rework workingset protection"), folio_test_referenced() was accidentally removed when checking for missed rotated folios. As a result, kept folios without an active set could be added to the retry list and eventually reclaimed.

The impact should be very small, as we have the folio_mapped(folio) check before retrying. Kept folios won't have try_to_unmap() called, so this is really only applicable to folios that are unmapped by users after folio_referenced() has been called.

I'm not sure if anyone else has seen any issues with this. Sending an RFC to check whether it is worth fixing.

Fixes 4d5d14a01e2c ("mm/mglru: rework workingset protection")
Cc Yu Zhao <[email protected]>
---
  mm/vmscan.c | 7 +++++--
  1 file changed, 5 insertions(+), 2 deletions(-)
mm/vmscan.c +1 -1
index 91295070ca33..6a931b576b7d 100644
@@ -994,8 +994,10 @@ static enum folio_references folio_check_references(struct folio *folio,
            return FOLIOREF_KEEP;
    if (lru_gen_enabled() && !lru_gen_switching()) {
-           if (!referenced_ptes)
+           if (!referenced_ptes) {
Quoted 8 lines

mm/vmscan.c static enum folio_references folio_check_references(struct folio *folio, · line 1000

Quoted 3 lines
mm/vmscan.c +2 -0
+                   folio_clear_referenced(folio);
                    return FOLIOREF_RECLAIM;
+           }

I've also been investigating the issue of referenced folios being reclaimed recently. The background is that classical LRU will return FOLIOREF_KEEP for folios accessed once via page table, giving the accessed folio another chance.

Hi, thanks for looking into this.

Quoted 5 lines

However, for MGLRU, if the folio's pagetable access has already been consumed by aging or lru_gen_look_around(), referenced_ptes may be 0 at that point, leading to reclaiming folios that were accessed via page table, which could cause more refaults (I don't have data yet).

If we go with MGLRU-FG, aging or lru_gen_look_around may move the folio to newer gen which is identical to the behavior of promoting at eviction time.

Quoted 5 lines

But it's not easy to maintain logic consistent with classical LRU, because we cannot distinguish whether this one access count comes from page table or file descriptors (for anonymous pages, it can be distinguished). I'm not sure if Kairui's MGLRU-FG patchset helps with this case (I haven't looked at it carefully yet).

It definatly helps, check this one:
https://lore.kernel.org/linux-mm/[email protected]/

folio_inc_lru_refs and variants will promote the folios in a consistent way. See "First access only defers eviction from the oldest gen" part, I believe that is the issue you worries about?

BTW promote then to newest gen directly does over protect them, I've see regression with that.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

Perhaps at least for anonymous pages, if the referenced flag is set, we should return FOLIOREF_KEEP to give it another scan chance.

No I think we really shouldn't do that, it will over protect single used folios. Promoting them to second newest gen or second oldest gen seems a good idea and similiar to CLRU, and this is the behavior of MGLRU-FG.

Oh I also mean that protecting them by lookaround / walking, and not at eviction seems better? As the rmap here have higher overhead and can see stall hotness info.

#4 lore raw
Quoted 16 lines · Kairui Song
Quoted 14 lines · Kairui Song

It definatly helps, check this one:
https://lore.kernel.org/linux-mm/[email protected]/

folio_inc_lru_refs and variants will promote the folios in a consistent way. See "First access only defers eviction from the oldest gen" part, I believe that is the issue you worries about?

BTW promote then to newest gen directly does over protect them, I've see regression with that.

Quoted 3 lines

Perhaps at least for anonymous pages, if the referenced flag is set, we should return FOLIOREF_KEEP to give it another scan chance.

Quoted 7 lines
Quoted 3 lines

No I think we really shouldn't do that, it will over protect single used folios. Promoting them to second newest gen or second oldest gen seems a good idea and similiar to CLRU, and this is the behavior of MGLRU-FG.

Oh I also mean that protecting them by lookaround / walking, and not at eviction seems better? As the rmap here have higher overhead and can see stall hotness info.

I also found this seems to work much better too on top of MGLRU-FG v3:
Will send a V4 later, the tricky part here is that oldest gen could be very short so it should cover second oldest gen as well. And swap readaheads are mostly unreferenced.

include/linux/mm_inline.h +5 -0
index e651f60cb914..7fb9c7e85732 100644
@@ -311,6 +311,13 @@ static inline int folio_lru_gen(const struct folio *folio)
        return lru_get_gen_flags(READ_ONCE(*const_folio_flags(folio, 0)));
 }
 
+static inline bool lru_gen_is_old(unsigned long max_seq, int gen)
+{
+       VM_WARN_ON_ONCE(gen >= MAX_NR_GENS);
+       /* see the comment on MIN_NR_GENS */
+       return gen == lru_gen_from_seq(max_seq) || gen ==

lru_gen_from_seq(max_seq - 1);
+}
+
static inline void lru_gen_update_size(struct lruvec *lruvec, int type,
struct folio *folio, int old_gen, int new_gen)
{

mm/vmscan.c +0 -0
index 5892be56b617..8514182b92b2 100644
@@ -1011,7 +1011,7 @@ void folio_inc_lru_refs(struct folio *folio,

unsigned int flags)

                            flags & (LRU_REF_EXEC | LRU_REF_FORCE))
                                gen = max_gen;
                        /* First access only defers eviction from the
oldest gen */
-                       else if (old_gen == min_gen)
+                       else if (lru_gen_is_old(max_seq, gen))
                                gen = (old_gen + 1) % MAX_NR_GENS;
                        refs = min(refs, LRU_REFS_PROTECTED);
                } else if (refs > LRU_REFS_MAX) {
@@ -1132,7 +1132,7 @@ static int folio_inc_lru_refs_walk(struct folio
*folio, struct lruvec *lruvec,
                        if (refs > LRU_REFS_REFERENCED ||
is_exec_file_folio(folio, vma_flags))
                                *new_gen = max_gen;
                        /* First access only defers eviction from the
oldest gen */
-                       else if (old_gen == min_gen)
+                       else if (lru_gen_is_old(max_seq, gen))
                                *new_gen = (old_gen + 1) % MAX_NR_GENS;
                        else
                                *new_gen = old_gen;
reviewer
#5 lore raw
In reply to Kairui Song
Quoted 57 lines · Kairui Song
Quoted 56 lines · Kairui Song
Quoted 33 lines · Baolin Wang

Hi Barry,

Quoted 30 lines · Barry Song (Xiaomi)

Since commit 4d5d14a01e2c ("mm/mglru: rework workingset protection"), folio_test_referenced() was accidentally removed when checking for missed rotated folios. As a result, kept folios without an active set could be added to the retry list and eventually reclaimed.

The impact should be very small, as we have the folio_mapped(folio) check before retrying. Kept folios won't have try_to_unmap() called, so this is really only applicable to folios that are unmapped by users after folio_referenced() has been called.

I'm not sure if anyone else has seen any issues with this. Sending an RFC to check whether it is worth fixing.

Fixes 4d5d14a01e2c ("mm/mglru: rework workingset protection")
Cc Yu Zhao <[email protected]>
---
   mm/vmscan.c | 7 +++++--
   1 file changed, 5 insertions(+), 2 deletions(-)
mm/vmscan.c +1 -1
index 91295070ca33..6a931b576b7d 100644
@@ -994,8 +994,10 @@ static enum folio_references folio_check_references(struct folio *folio,
             return FOLIOREF_KEEP;
     if (lru_gen_enabled() && !lru_gen_switching()) {
-           if (!referenced_ptes)
+           if (!referenced_ptes) {
Quoted 8 lines

mm/vmscan.c static enum folio_references folio_check_references(struct folio *folio, · line 1000

Quoted 3 lines
mm/vmscan.c +2 -0
+                   folio_clear_referenced(folio);
                     return FOLIOREF_RECLAIM;
+           }

I've also been investigating the issue of referenced folios being reclaimed recently. The background is that classical LRU will return FOLIOREF_KEEP for folios accessed once via page table, giving the accessed folio another chance.

Hi, thanks for looking into this.

Quoted 5 lines

However, for MGLRU, if the folio's pagetable access has already been consumed by aging or lru_gen_look_around(), referenced_ptes may be 0 at that point, leading to reclaiming folios that were accessed via page table, which could cause more refaults (I don't have data yet).

If we go with MGLRU-FG, aging or lru_gen_look_around may move the folio to newer gen which is identical to the behavior of promoting at eviction time.

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

But it's not easy to maintain logic consistent with classical LRU, because we cannot distinguish whether this one access count comes from page table or file descriptors (for anonymous pages, it can be distinguished). I'm not sure if Kairui's MGLRU-FG patchset helps with this case (I haven't looked at it carefully yet).

It definatly helps, check this one:
https://lore.kernel.org/linux-mm/[email protected]/

OK. Great.

Quoted 3 lines
Quoted 3 lines

folio_inc_lru_refs and variants will promote the folios in a consistent way. See "First access only defers eviction from the oldest gen" part, I believe that is the issue you worries about?

Seems so. I'll look at your patchset closely.

Quoted 2 lines
Quoted 2 lines

BTW promote then to newest gen directly does over protect them, I've see regression with that.

Agree. A folio with only one access indeed should not enter the newest gen.

Quoted 10 lines
Quoted 6 lines
Quoted 2 lines

Perhaps at least for anonymous pages, if the referenced flag is set, we should return FOLIOREF_KEEP to give it another scan chance.

No I think we really shouldn't do that, it will over protect single used folios. Promoting them to second newest gen or second oldest gen seems a good idea and similiar to CLRU, and this is the behavior of MGLRU-FG.

Oh I also mean that protecting them by lookaround / walking, and not at eviction seems better? As the rmap here have higher overhead and can see stall hotness info.

Sounds good to me. Thanks.