Skip to contentSkip to conversation

mm/mglru: reset page lru tier bits when activating

PATCH In review Wei Xu 3 participants 6 messages Oct 14, 2024 – Oct 17, 2024

Subsystems (4)
lore
Share & tools
lore

Wei Xu

author reviewer
#0 lore raw

PATCH 2/2 mm/mglru: reset page lru tier bits when activating

folio_activate() calls lru_gen_add_folio() to move the folio to the youngest generation. But unlike folio_update_gen()/folio_inc_gen(), lru_gen_add_folio() doesn't reset the folio lru tier bits (LRU_REFS_MASK | LRU_REFS_FLAGS). Fix this inconsistency in lru_gen_add_folio() when activating a folio.

Fixes 018ee47f1489 ("mm: multi-gen LRU: exploit locality in rmap")
---
 include/linux/mm_inline.h | 5 ++++-
 include/linux/mmzone.h    | 2 ++
 mm/vmscan.c               | 2 --
 3 files changed, 6 insertions(+), 3 deletions(-)
include/linux/mm_inline.h +4 -1
index 6f801c7b36e2..87580e8363ef 100644
@@ -222,6 +222,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
 {
 	unsigned long seq;
 	unsigned long flags;
+	unsigned long mask;
 	int gen = folio_lru_gen(folio);
 	int type = folio_is_file_lru(folio);
 	int zone = folio_zonenum(folio);
@@ -257,7 +258,9 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
 	gen = lru_gen_from_seq(seq);
 	flags = (gen + 1UL) << LRU_GEN_PGOFF;
 	/* see the comment on MIN_NR_GENS about PG_active */
-	set_mask_bits(&folio->flags, LRU_GEN_MASK | BIT(PG_active), flags);
+	mask = LRU_GEN_MASK | BIT(PG_active);
+	mask |= folio_test_active(folio) ? (LRU_REFS_MASK | LRU_REFS_FLAGS) : 0;
+	set_mask_bits(&folio->flags, mask, flags);
 
 	lru_gen_update_size(lruvec, folio, -1, gen);
 	/* for folio_rotate_reclaimable() */
include/linux/mmzone.h +2 -0
index 17506e4a2835..96dea31fb211 100644
@@ -403,6 +403,8 @@ enum {
 	NR_LRU_GEN_CAPS
 };
 
+#define LRU_REFS_FLAGS		(BIT(PG_referenced) | BIT(PG_workingset))
+
 #define MIN_LRU_BATCH		BITS_PER_LONG
 #define MAX_LRU_BATCH		(MIN_LRU_BATCH * 64)
 
mm/vmscan.c +0 -2
index 9d1e1c4e383d..907262ebaef8 100644
@@ -2601,8 +2601,6 @@ static bool should_clear_pmd_young(void)
  *                          shorthand helpers
  ******************************************************************************/
 
-#define LRU_REFS_FLAGS	(BIT(PG_referenced) | BIT(PG_workingset))
-
 #define DEFINE_MAX_SEQ(lruvec)						\
 	unsigned long max_seq = READ_ONCE((lruvec)->lrugen.max_seq)
 
Signature · 1 line
2.47.0.rc1.288.g06298d1525-goog
maintainer
#1 lore raw
Quoted 5 lines · On Mon, 14 Oct 2024 22:12:31 +0000 Wei Xu <[email protected]> wrote:

folio_activate() calls lru_gen_add_folio() to move the folio to the youngest generation. But unlike folio_update_gen()/folio_inc_gen(), lru_gen_add_folio() doesn't reset the folio lru tier bits (LRU_REFS_MASK | LRU_REFS_FLAGS). Fix this inconsistency in lru_gen_add_folio() when activating a folio.

What are the runtime effects of this flaw?

author reviewer
#2 lore raw
Quoted 10 lines · Andrew Morton
Quoted 5 lines · On Mon, 14 Oct 2024 22:12:31 +0000 Wei Xu <[email protected]> wrote:

folio_activate() calls lru_gen_add_folio() to move the folio to the youngest generation. But unlike folio_update_gen()/folio_inc_gen(), lru_gen_add_folio() doesn't reset the folio lru tier bits (LRU_REFS_MASK | LRU_REFS_FLAGS). Fix this inconsistency in lru_gen_add_folio() when activating a folio.

What are the runtime effects of this flaw?

It can affect how pages get aged via the MGLRU PID controller, though no bad behaviors clearly related to this have been detected at runtime. The fix is to address this inconsistency identified via code inspection.

#3 lore raw
In reply to Wei Xu
Quoted 28 lines · Wei Xu

folio_activate() calls lru_gen_add_folio() to move the folio to the youngest generation. But unlike folio_update_gen()/folio_inc_gen(), lru_gen_add_folio() doesn't reset the folio lru tier bits (LRU_REFS_MASK | LRU_REFS_FLAGS). Fix this inconsistency in lru_gen_add_folio() when activating a folio.

Fixes 018ee47f1489 ("mm: multi-gen LRU: exploit locality in rmap")
---
 include/linux/mm_inline.h | 5 ++++-
 include/linux/mmzone.h    | 2 ++
 mm/vmscan.c               | 2 --
 3 files changed, 6 insertions(+), 3 deletions(-)
include/linux/mm_inline.h +1 -0
index 6f801c7b36e2..87580e8363ef 100644
@@ -222,6 +222,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
 {
        unsigned long seq;
        unsigned long flags;
+       unsigned long mask;
        int gen = folio_lru_gen(folio);
        int type = folio_is_file_lru(folio);
        int zone = folio_zonenum(folio);

include/linux/mm_inline.h static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, · line 262

Quoted 6 lines
include/linux/mm_inline.h +2 -1
@@ -257,7 +258,9 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
        gen = lru_gen_from_seq(seq);
        flags = (gen + 1UL) << LRU_GEN_PGOFF;
        /* see the comment on MIN_NR_GENS about PG_active */
-       set_mask_bits(&folio->flags, LRU_GEN_MASK | BIT(PG_active), flags);
+       mask = LRU_GEN_MASK | BIT(PG_active);
+       mask |= folio_test_active(folio) ? (LRU_REFS_MASK | LRU_REFS_FLAGS) : 0;

We shouldn't clear PG_workingset here because it can affect PSI accounting, if the activation is due to workingset refault.

Also, nit:
mask = LRU_GEN_MASK;
if (folio_test_active(folio))

    mask |= LRU_REFS_MASK | BIT(PG_active) | BIT(PG_referenced);
Quoted 34 lines
include/linux/mm_inline.h +1 -0
+       set_mask_bits(&folio->flags, mask, flags);
 
        lru_gen_update_size(lruvec, folio, -1, gen);
        /* for folio_rotate_reclaimable() */
include/linux/mmzone.h +2 -0
index 17506e4a2835..96dea31fb211 100644
@@ -403,6 +403,8 @@ enum {
        NR_LRU_GEN_CAPS
 };
 
+#define LRU_REFS_FLAGS         (BIT(PG_referenced) | BIT(PG_workingset))
+
 #define MIN_LRU_BATCH          BITS_PER_LONG
 #define MAX_LRU_BATCH          (MIN_LRU_BATCH * 64)
 
mm/vmscan.c +0 -3
index 9d1e1c4e383d..907262ebaef8 100644
@@ -2601,8 +2601,6 @@ static bool should_clear_pmd_young(void)
  *                          shorthand helpers
  ******************************************************************************/
 
-#define LRU_REFS_FLAGS (BIT(PG_referenced) | BIT(PG_workingset))
-
 #define DEFINE_MAX_SEQ(lruvec)                                         \
        unsigned long max_seq = READ_ONCE((lruvec)->lrugen.max_seq)
 
--

2.47.0.rc1.288.g06298d1525-goog

maintainer
#4 lore raw
Quoted 11 lines · On Tue, 15 Oct 2024 22:55:23 -0600 Yu Zhao <[email protected]> wrote:

include/linux/mm_inline.h static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, · line 262

Quoted 7 lines
include/linux/mm_inline.h +2 -1
@@ -257,7 +258,9 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
        gen = lru_gen_from_seq(seq);
        flags = (gen + 1UL) << LRU_GEN_PGOFF;
        /* see the comment on MIN_NR_GENS about PG_active */
-       set_mask_bits(&folio->flags, LRU_GEN_MASK | BIT(PG_active), flags);
+       mask = LRU_GEN_MASK | BIT(PG_active);
+       mask |= folio_test_active(folio) ? (LRU_REFS_MASK | LRU_REFS_FLAGS) : 0;

We shouldn't clear PG_workingset here because it can affect PSI accounting, if the activation is due to workingset refault.

Quoted 5 lines

Also, nit:
mask = LRU_GEN_MASK;
if (folio_test_active(folio))

    mask |= LRU_REFS_MASK | BIT(PG_active) | BIT(PG_referenced);

Thanks, I'll drop this version of this patch.

When resending, please include a full description of the userspace-visible effects of the original flaw, thanks.

author reviewer
#5 lore raw
Quoted 14 lines · Andrew Morton
Quoted 11 lines · On Tue, 15 Oct 2024 22:55:23 -0600 Yu Zhao <[email protected]> wrote:

include/linux/mm_inline.h static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, · line 262

Quoted 7 lines
include/linux/mm_inline.h +2 -1
@@ -257,7 +258,9 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
        gen = lru_gen_from_seq(seq);
        flags = (gen + 1UL) << LRU_GEN_PGOFF;
        /* see the comment on MIN_NR_GENS about PG_active */
-       set_mask_bits(&folio->flags, LRU_GEN_MASK | BIT(PG_active), flags);
+       mask = LRU_GEN_MASK | BIT(PG_active);
+       mask |= folio_test_active(folio) ? (LRU_REFS_MASK | LRU_REFS_FLAGS) : 0;

We shouldn't clear PG_workingset here because it can affect PSI accounting, if the activation is due to workingset refault.

Good point. I have addressed this in the v2 patch.

Quoted 10 lines
Quoted 5 lines

Also, nit:
mask = LRU_GEN_MASK;
if (folio_test_active(folio))

    mask |= LRU_REFS_MASK | BIT(PG_active) | BIT(PG_referenced);

Thanks, I'll drop this version of this patch.

When resending, please include a full description of the userspace-visible effects of the original flaw, thanks.

I have sent out a v2 patch, which includes a description as suggested.