mm/mglru: reset page lru tier bits when activating
Subsystems (4)
Wei Xu
reviewerPATCH 2/2 mm/mglru: reset page lru tier bits when activating
folio_activate() calls lru_gen_add_folio() to move the folio to the youngest generation. But unlike folio_update_gen()/folio_inc_gen(), lru_gen_add_folio() doesn't reset the folio lru tier bits (LRU_REFS_MASK | LRU_REFS_FLAGS). Fix this inconsistency in lru_gen_add_folio() when activating a folio.
--- include/linux/mm_inline.h | 5 ++++- include/linux/mmzone.h | 2 ++ mm/vmscan.c | 2 -- 3 files changed, 6 insertions(+), 3 deletions(-)
include/linux/mm_inline.h +4 -1
@@ -222,6 +222,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, {unsigned long seq;unsigned long flags; +unsigned long mask;int gen =folio_lru_gen (folio);int type =folio_is_file_lru (folio);int zone =folio_zonenum (folio);
@@ -257,7 +258,9 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, gen =lru_gen_from_seq (seq); flags = (gen +1UL ) <<LRU_GEN_PGOFF ;/* see the comment on MIN_NR_GENS about PG_active */ -set_mask_bits (&folio->flags,LRU_GEN_MASK |BIT (PG_active), flags); + mask =LRU_GEN_MASK |BIT (PG_active); + mask |=folio_test_active (folio) ? (LRU_REFS_MASK |LRU_REFS_FLAGS ) :0 ; +set_mask_bits (&folio->flags, mask, flags);lru_gen_update_size (lruvec, folio,-1 , gen);/* for folio_rotate_reclaimable() */
include/linux/mmzone.h +2 -0
@@ -403,6 +403,8 @@ enum { NR_LRU_GEN_CAPS }; +#define LRU_REFS_FLAGS (BIT(PG_referenced) | BIT(PG_workingset)) +#define MIN_LRU_BATCH BITS_PER_LONG#define MAX_LRU_BATCH (MIN_LRU_BATCH * 64)
mm/vmscan.c +0 -2
@@ -2601,8 +2601,6 @@ static bool should_clear_pmd_young(void)* shorthand helpers ******************************************************************************/ -#define LRU_REFS_FLAGS (BIT(PG_referenced) | BIT(PG_workingset)) -#define DEFINE_MAX_SEQ (lruvec) \ unsigned long max_seq = READ_ONCE((lruvec)->lrugen.max_seq)
Signature · 1 line
2.47.0.rc1.288.g06298d1525-goog
Andrew Morton
maintainerQuoted 5 lines · On Mon, 14 Oct 2024 22:12:31 +0000 Wei Xu <[email protected]> wrote:
folio_activate()callslru_gen_add_folio()to move the folio to the youngest generation. But unlike folio_update_gen()/folio_inc_gen(),lru_gen_add_folio()doesn't reset the folio lru tier bits (LRU_REFS_MASK|LRU_REFS_FLAGS). Fix this inconsistency inlru_gen_add_folio()when activating a folio.
What are the runtime effects of this flaw?
Wei Xu
reviewerQuoted 10 lines · Andrew Morton
Quoted 5 lines · On Mon, 14 Oct 2024 22:12:31 +0000 Wei Xu <[email protected]> wrote:
folio_activate()callslru_gen_add_folio()to move the folio to the youngest generation. But unlike folio_update_gen()/folio_inc_gen(),lru_gen_add_folio()doesn't reset the folio lru tier bits (LRU_REFS_MASK|LRU_REFS_FLAGS). Fix this inconsistency inlru_gen_add_folio()when activating a folio.What are the runtime effects of this flaw?
It can affect how pages get aged via the MGLRU PID controller, though no bad behaviors clearly related to this have been detected at runtime. The fix is to address this inconsistency identified via code inspection.
Quoted 28 lines · Wei Xu
folio_activate()callslru_gen_add_folio()to move the folio to the youngest generation. But unlike folio_update_gen()/folio_inc_gen(),lru_gen_add_folio()doesn't reset the folio lru tier bits (LRU_REFS_MASK|LRU_REFS_FLAGS). Fix this inconsistency inlru_gen_add_folio()when activating a folio.Fixes 018ee47f1489 ("mm: multi-gen LRU: exploit locality in rmap")Signed-off-by Wei Xu <[email protected]>--- include/linux/mm_inline.h | 5 ++++- include/linux/mmzone.h | 2 ++ mm/vmscan.c | 2 -- 3 files changed, 6 insertions(+), 3 deletions(-)include/linux/mm_inline.h +1
-0@@ -222,6 +222,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, {unsigned long seq;unsigned long flags; +unsigned long mask;int gen =folio_lru_gen (folio);int type =folio_is_file_lru (folio);int zone =folio_zonenum (folio);
include/linux/mm_inline.h static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, · line 262
Quoted 6 lines
include/linux/mm_inline.h +2
-1@@ -257,7 +258,9 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, gen =lru_gen_from_seq (seq); flags = (gen +1UL ) <<LRU_GEN_PGOFF ;/* see the comment on MIN_NR_GENS about PG_active */ -set_mask_bits (&folio->flags,LRU_GEN_MASK |BIT (PG_active), flags); + mask =LRU_GEN_MASK |BIT (PG_active); + mask |=folio_test_active (folio) ? (LRU_REFS_MASK |LRU_REFS_FLAGS ) :0 ;
We shouldn't clear PG_workingset here because it can affect PSI accounting, if the activation is due to workingset refault.
Also, nit:
mask = LRU_GEN_MASK;
if (folio_test_active(folio))
mask |= LRU_REFS_MASK | BIT(PG_active) | BIT(PG_referenced);
Quoted 34 lines
include/linux/mm_inline.h +1
-0+set_mask_bits (&folio->flags, mask, flags);lru_gen_update_size (lruvec, folio,-1 , gen);/* for folio_rotate_reclaimable() */ include/linux/mmzone.h +2
-0@@ -403,6 +403,8 @@ enum { NR_LRU_GEN_CAPS }; +#define LRU_REFS_FLAGS (BIT(PG_referenced) | BIT(PG_workingset)) +#define MIN_LRU_BATCH BITS_PER_LONG#define MAX_LRU_BATCH (MIN_LRU_BATCH * 64)mm/vmscan.c +0
-3@@ -2601,8 +2601,6 @@ static bool should_clear_pmd_young(void)* shorthand helpers ******************************************************************************/ -#define LRU_REFS_FLAGS (BIT(PG_referenced) | BIT(PG_workingset)) -#define DEFINE_MAX_SEQ (lruvec) \ unsigned long max_seq = READ_ONCE((lruvec)->lrugen.max_seq) --2.47.0.rc1.288.g06298d1525-goog
Andrew Morton
maintainerQuoted 11 lines · On Tue, 15 Oct 2024 22:55:23 -0600 Yu Zhao <[email protected]> wrote:
include/linux/mm_inline.h static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, · line 262
Quoted 7 lines
include/linux/mm_inline.h +2
-1@@ -257,7 +258,9 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, gen =lru_gen_from_seq (seq); flags = (gen +1UL ) <<LRU_GEN_PGOFF ;/* see the comment on MIN_NR_GENS about PG_active */ -set_mask_bits (&folio->flags,LRU_GEN_MASK |BIT (PG_active), flags); + mask =LRU_GEN_MASK |BIT (PG_active); + mask |=folio_test_active (folio) ? (LRU_REFS_MASK |LRU_REFS_FLAGS ) :0 ;We shouldn't clear
PG_workingsethere because it can affect PSI accounting, if the activation is due to workingset refault.
Quoted 5 lines
Also, nit:
mask =LRU_GEN_MASK;
if (folio_test_active(folio))mask |= LRU_REFS_MASK | BIT(PG_active) | BIT(PG_referenced);
Thanks, I'll drop this version of this patch.
When resending, please include a full description of the userspace-visible effects of the original flaw, thanks.
Wei Xu
reviewerQuoted 14 lines · Andrew Morton
Quoted 11 lines · On Tue, 15 Oct 2024 22:55:23 -0600 Yu Zhao <[email protected]> wrote:
include/linux/mm_inline.h static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, · line 262
Quoted 7 lines
include/linux/mm_inline.h +2
-1@@ -257,7 +258,9 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, gen =lru_gen_from_seq (seq); flags = (gen +1UL ) <<LRU_GEN_PGOFF ;/* see the comment on MIN_NR_GENS about PG_active */ -set_mask_bits (&folio->flags,LRU_GEN_MASK |BIT (PG_active), flags); + mask =LRU_GEN_MASK |BIT (PG_active); + mask |=folio_test_active (folio) ? (LRU_REFS_MASK |LRU_REFS_FLAGS ) :0 ;We shouldn't clear
PG_workingsethere because it can affect PSI accounting, if the activation is due to workingset refault.
Good point. I have addressed this in the v2 patch.
Quoted 10 lines
Quoted 5 lines
Also, nit:
mask =LRU_GEN_MASK;
if (folio_test_active(folio))mask |= LRU_REFS_MASK | BIT(PG_active) | BIT(PG_referenced);Thanks, I'll drop this version of this patch.
When resending, please include a full description of the userspace-visible effects of the original flaw, thanks.
I have sent out a v2 patch, which includes a description as suggested.