Skip to contentSkip to conversation

Support arch-specific page aging mechanism

RFC PATCH New Aneesh Kumar K.V 1 participant 8 messages Apr 2, 2023

Subsystems (12)
lore
Share & tools
lore
#0 lore raw

cover Support arch-specific page aging mechanism

Architectures like powerpc support page access count mechanism which can be used for better identification of hot/cold pages in the system. POWER10 supports a 32-bit page access count which is incremented based on page access and decremented based on time decay. The page access count is incremented based on physical address filtering and hence should count access via page table(mmap) and read/write syscall.

This patch series updates multi-gen LRU to use this page access count instead of the page table reference bit to classify a page into a generation. Pages are classified into generation during the sorting phase of reclaim. Currently sorting phase use generation details stored in page flags and with this change, we can avoid using page flags for storing generation. That will free the 3 bits in page flag used to store generation. Since the page access counting mechanism can also count access via read/write, we can look at avoiding using tier index in page flags. That should free the 2 bits in page flag used for REFS (this is not done in this patch).

I also added a patch that did the below
@@ -5243,7 +5243,8 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap
if (list_empty(&list))
return scanned;
retry:
- reclaimed = shrink_folio_list(&list, pgdat, sc, &stat, false);
+ reclaimed = shrink_folio_list(&list, pgdat, sc,
+ &stat, arch_supports_page_access_count());
sc->nr_reclaimed += reclaimed;

The performance did improve, but that did result in a large increase in the workingset_refault_anon. I think this is because it takes some minimal access to classify the pages to the younger generation and we can have high page refaults during that window.

PATCH 2 did result in some improvements on powerpc because it is removing all additional code that is not used in page classification.

memcached:

patch details                 Total Ops/sec:
mglru                         160821
PATCH 2                       164572

mongodb:

Patch details                 Throughput(Ops/sec)
mglru                         92987
PATCH 2                       93740

Enabling the architecture-supported page access count does impact workload performance since updating the access count involves some memory access overhead. Another challenge with page access count is in determining relative hotness between pages. I did try two methods density-based clustering and kmean clustering to classify pages to LRU generation based on sampled hotness. Doing more work during page classification is resulting in increased lock contention on lru_lock and hence hurts performance.

memcached:

patch details                       Total Ops/sec:
arch page access count              161940
avoid folio_check_reference         171631 (but refault count increase from 2606765 -> 7793482)

mongodb:

Patch details                      Throughput(Ops/sec)
arch page access count             92533
avoid folio_check_reference        91105 ( refault: 828951 -> 4592539)

The patch series does show that using page access count is not resulting in any regression and can keep the code simpler w.r.t different feedback loop used during multi-gen LRU reclaim. This also saves some bits in page->flags . It was also observed that overhead in counting page access is not that high and can be mitigated by further tuning of the page generation classification logic. This also enables us to start looking at using page access count in other parts of the linux kernel like page promotion. I haven't been able to measure the impact on page promotion yet due to hardware availability.

Aneesh Kumar K.V (7):
mm: Move some code around so that next patch is simpler
mm: Don't build multi-gen LRU page table walk code on architecture not
supported
mm: multi-gen LRU: avoid using generation stored in page flags for
generation
mm: multi-gen LRU: support different page aging mechanism powerpc/mm: Add page access count support
powerpc/mm: Clear page access count on allocation
mm: multi-gen LRU: Shrink folio list without checking for page table
reference

 arch/Kconfig                          |   3 +
 arch/arm64/Kconfig                    |   1 +
 arch/powerpc/Kconfig                  |  10 +
 arch/powerpc/include/asm/hca.h        |  49 ++++
 arch/powerpc/include/asm/page.h       |   5 +
 arch/powerpc/include/asm/page_aging.h |  35 +++
 arch/powerpc/mm/Makefile              |   1 +
 arch/powerpc/mm/hca.c                 | 288 ++++++++++++++++++++
 arch/x86/Kconfig                      |   1 +
 include/linux/memcontrol.h            |   2 +-
 include/linux/mm_inline.h             |  47 +---
 include/linux/mm_types.h              |   8 +-
 include/linux/mmzone.h                |  15 +-
 include/linux/page_aging.h            |  43 +++
 include/linux/swap.h                  |   2 +-
 kernel/fork.c                         |   2 +-
 mm/Kconfig                            |   4 +
 mm/memcontrol.c                       |   2 +-
 mm/rmap.c                             |   4 +-
 mm/vmscan.c                           | 372 ++++++++++++++++++++++----
 20 files changed, 780 insertions(+), 114 deletions(-)
 create mode 100644 arch/powerpc/include/asm/hca.h
 create mode 100644 arch/powerpc/include/asm/page_aging.h
 create mode 100644 arch/powerpc/mm/hca.c
 create mode 100644 include/linux/page_aging.h
Signature · 1 line
2.39.2
#1 lore raw

RFC PATCH 1/7 mm: Move some code around so that next patch is simpler

Move lrur_gen_add_folio to .c. We will support arch specific mapping of page access count to generation in a later patch and will use that when adding folio to lruvec. This move enables that.

No functional change in this patch.

---
 include/linux/mm_inline.h |  47 +-------------
 mm/vmscan.c               | 127 ++++++++++++++++++++++++++------------
 2 files changed, 88 insertions(+), 86 deletions(-)
include/linux/mm_inline.h +1 -46
index ff3f3f23f649..4dc2ab95d612 100644
@@ -217,52 +217,7 @@ static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *foli
 	VM_WARN_ON_ONCE(lru_gen_is_active(lruvec, old_gen) && !lru_gen_is_active(lruvec, new_gen));
 }
 
-static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming)
-{
-	unsigned long seq;
-	unsigned long flags;
-	int gen = folio_lru_gen(folio);
-	int type = folio_is_file_lru(folio);
-	int zone = folio_zonenum(folio);
-	struct lru_gen_struct *lrugen = &lruvec->lrugen;
-
-	VM_WARN_ON_ONCE_FOLIO(gen != -1, folio);
-
-	if (folio_test_unevictable(folio) || !lrugen->enabled)
-		return false;
-	/*
-	 * There are three common cases for this page:
-	 * 1. If it's hot, e.g., freshly faulted in or previously hot and
-	 *    migrated, add it to the youngest generation.
-	 * 2. If it's cold but can't be evicted immediately, i.e., an anon page
-	 *    not in swapcache or a dirty page pending writeback, add it to the
-	 *    second oldest generation.
-	 * 3. Everything else (clean, cold) is added to the oldest generation.
-	 */
-	if (folio_test_active(folio))
-		seq = lrugen->max_seq;
-	else if ((type == LRU_GEN_ANON && !folio_test_swapcache(folio)) ||
-		 (folio_test_reclaim(folio) &&
-		  (folio_test_dirty(folio) || folio_test_writeback(folio))))
-		seq = lrugen->min_seq[type] + 1;
-	else
-		seq = lrugen->min_seq[type];
-
-	gen = lru_gen_from_seq(seq);
-	flags = (gen + 1UL) << LRU_GEN_PGOFF;
-	/* see the comment on MIN_NR_GENS about PG_active */
-	set_mask_bits(&folio->flags, LRU_GEN_MASK | BIT(PG_active), flags);
-
-	lru_gen_update_size(lruvec, folio, -1, gen);
-	/* for folio_rotate_reclaimable() */
-	if (reclaiming)
-		list_add_tail(&folio->lru, &lrugen->lists[gen][type][zone]);
-	else
-		list_add(&folio->lru, &lrugen->lists[gen][type][zone]);
-
-	return true;
-}
-
+bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming);
 static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming)
 {
 	unsigned long flags;
mm/vmscan.c +87 -40

151 lines not on this page · raw message

Signature · 1 line
2.39.2
#2 lore raw

RFC PATCH 2/7 mm: Don't build multi-gen LRU page table walk code on architecture not supported

In reply to Aneesh Kumar K.V

Not all architecture supports hardware atomic updates of access bits. On such an arch, we don't use page table walk to classify pages into generations. Add a kernel config option and remove adding all the page table walk code on such architecture.

lru_gen_look_around() code is duplicated because lru_gen_mm_walk is not available always.

This patch did result in some improvements on powerpc because it is removing all additional code that is not used in page classification.

memcached:

patch details                 Total Ops/sec:
mglru                         160821
PATCH 2                       164572

mongodb:

Patch details                 Throughput(Ops/sec)
mglru                         92987
PATCH 2                       93740
---
 arch/Kconfig               |   3 +
 arch/arm64/Kconfig         |   1 +
 arch/x86/Kconfig           |   1 +
 include/linux/memcontrol.h |   2 +-
 include/linux/mm_types.h   |   8 +-
 include/linux/mmzone.h     |  10 +-
 include/linux/swap.h       |   2 +-
 kernel/fork.c              |   2 +-
 mm/memcontrol.c            |   2 +-
 mm/vmscan.c                | 221 ++++++++++++++++++++++++++++++++++---
 10 files changed, 230 insertions(+), 22 deletions(-)
arch/Kconfig +3 -0
index 12e3ddabac9d..61fc138bb91a 100644
@@ -1426,6 +1426,9 @@ config DYNAMIC_SIGFRAME
 config HAVE_ARCH_NODE_DEV_GROUP
 	bool
 
+config LRU_TASK_PAGE_AGING
+	bool
+
 config ARCH_HAS_NONLEAF_PMD_YOUNG
 	bool
 	help
arch/arm64/Kconfig +1 -0
index 27b2592698b0..b783b339ef59 100644
@@ -219,6 +219,7 @@ config ARM64
 	select IRQ_DOMAIN
 	select IRQ_FORCED_THREADING
 	select KASAN_VMALLOC if KASAN
+	select LRU_TASK_PAGE_AGING
 	select MODULES_USE_ELF_RELA
 	select NEED_DMA_MAP_STATE
 	select NEED_SG_DMA_LENGTH
arch/x86/Kconfig +1 -0
index a825bf031f49..805d3f6a1a58 100644
@@ -274,6 +274,7 @@ config X86
 	select HAVE_GENERIC_VDSO
 	select HOTPLUG_SMT			if SMP
 	select IRQ_FORCED_THREADING
+	select LRU_TASK_PAGE_AGING
 	select NEED_PER_CPU_EMBED_FIRST_CHUNK
 	select NEED_PER_CPU_PAGE_FIRST_CHUNK
 	select NEED_SG_DMA_LENGTH
include/linux/memcontrol.h +1 -1
index a7b5925a033e..6b48a30a0dae 100644
@@ -320,7 +320,7 @@ struct mem_cgroup {
 	struct deferred_split deferred_split_queue;
 #endif
 
-#ifdef CONFIG_LRU_GEN
+#ifdef CONFIG_LRU_TASK_PAGE_AGING
 	/* per-memcg mm_struct list */
 	struct lru_gen_mm_list mm_list;
 #endif
include/linux/mm_types.h +4 -4
index af8119776ab1..7bca8987a86b 100644
@@ -796,7 +796,7 @@ struct mm_struct {
 		 */
 		unsigned long ksm_rmap_items;
 #endif
-#ifdef CONFIG_LRU_GEN
+#ifdef CONFIG_LRU_TASK_PAGE_AGING
 		struct {
 			/* this mm_struct is on lru_gen_mm_list */
 			struct list_head list;
@@ -811,7 +811,7 @@ struct mm_struct {
 			struct mem_cgroup *memcg;
 #endif
 		} lru_gen;
-#endif /* CONFIG_LRU_GEN */
+#endif /* CONFIG_LRU_TASK_PAGE_AGING */
 	} __randomize_layout;
 
 	/*
@@ -839,7 +839,7 @@ static inline cpumask_t *mm_cpumask(struct mm_struct *mm)
 	return (struct cpumask *)&mm->cpu_bitmap;
 }
 
-#ifdef CONFIG_LRU_GEN
+#ifdef CONFIG_LRU_TASK_PAGE_AGING
 
 struct lru_gen_mm_list {
 	/* mm_struct list for page table walkers */
@@ -873,7 +873,7 @@ static inline void lru_gen_use_mm(struct mm_struct *mm)
 	WRITE_ONCE(mm->lru_gen.bitmap, -1);
 }
 
-#else /* !CONFIG_LRU_GEN */
+#else /* !CONFIG_LRU_TASK_PAGE_AGING */
 
 static inline void lru_gen_add_mm(struct mm_struct *mm)
 {
include/linux/mmzone.h +9 -1
index cd28a100d9e4..0bcc5d88239a 100644
@@ -428,6 +428,7 @@ struct lru_gen_struct {
 	bool enabled;
 };
 
+#ifdef CONFIG_LRU_TASK_PAGE_AGING
 enum {
 	MM_LEAF_TOTAL,		/* total leaf entries */
 	MM_LEAF_OLD,		/* old leaf entries */
@@ -474,6 +475,7 @@ struct lru_gen_mm_walk {
 	bool can_swap;
 	bool force_scan;
 };
+#endif
 
 void lru_gen_init_lruvec(struct lruvec *lruvec);
 void lru_gen_look_around(struct page_vma_mapped_walk *pvmw);
@@ -525,8 +527,14 @@ struct lruvec {
 #ifdef CONFIG_LRU_GEN
 	/* evictable pages divided into generations */
 	struct lru_gen_struct		lrugen;
+#ifdef CONFIG_LRU_TASK_PAGE_AGING
 	/* to concurrently iterate lru_gen_mm_list */
 	struct lru_gen_mm_state		mm_state;
+#else
+	/* for concurrent update of max_seq without holding lru_lock */
+	struct wait_queue_head seq_update_wait;
+	bool seq_update_progress;
+#endif
 #endif
 #ifdef CONFIG_MEMCG
 	struct pglist_data *pgdat;
@@ -1240,7 +1248,7 @@ typedef struct pglist_data {
 
 	unsigned long		flags;
 
-#ifdef CONFIG_LRU_GEN
+#ifdef CONFIG_LRU_TASK_PAGE_AGING
 	/* kswap mm walk data */
 	struct lru_gen_mm_walk	mm_walk;
 #endif
include/linux/swap.h +1 -1
index 0ceed49516ad..d79976635c42 100644
@@ -154,7 +154,7 @@ union swap_header {
  */
 struct reclaim_state {
 	unsigned long reclaimed_slab;
-#ifdef CONFIG_LRU_GEN
+#ifdef CONFIG_LRU_TASK_PAGE_AGING
 	/* per-thread mm walk data */
 	struct lru_gen_mm_walk *mm_walk;
 #endif
kernel/fork.c +1 -1
index 038b898dad52..804517394f55 100644
@@ -2708,7 +2708,7 @@ pid_t kernel_clone(struct kernel_clone_args *args)
 		get_task_struct(p);
 	}
 
-	if (IS_ENABLED(CONFIG_LRU_GEN) && !(clone_flags & CLONE_VM)) {
+	if (IS_ENABLED(CONFIG_LRU_TASK_PAGE_AGING) && !(clone_flags & CLONE_VM)) {
 		/* lock the task to synchronize with memcg migration */
 		task_lock(p);
 		lru_gen_add_mm(p->mm);
mm/memcontrol.c +1 -1
index 2ca843cc3aa6..1302f00bd5e7 100644
@@ -6305,7 +6305,7 @@ static void mem_cgroup_move_task(void)
 }
 #endif
 
-#ifdef CONFIG_LRU_GEN
+#ifdef CONFIG_LRU_TASK_PAGE_AGING
 static void mem_cgroup_attach(struct cgroup_taskset *tset)
 {
 	struct task_struct *task;
mm/vmscan.c +208 -13

365 lines not on this page · raw message

Signature · 1 line
2.39.2
#3 lore raw

RFC PATCH 3/7 mm: multi-gen LRU: avoid using generation stored in page flags for generation

In reply to Aneesh Kumar K.V

Some architectures can handle different methods for determining page access count. In such case, we may not really need to use page flags for tracking generation. We can possibly derive generation directly from arch-supported access count values. Hence avoid using page flags to store generation in that case.

---
 include/linux/page_aging.h | 26 ++++++++++++++++++++++++++
 mm/rmap.c                  |  4 +++-
 mm/vmscan.c                | 34 +++++++++++++++++++++++++---------
 3 files changed, 54 insertions(+), 10 deletions(-)
 create mode 100644 include/linux/page_aging.h
include/linux/page_aging.h new file +26 -0
new file mode 100644
index 000000000000..ab77f4578916
@@ -0,0 +1,26 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+
+#ifndef _LINUX_PAGE_AGING_H
+#define _LINUX_PAGE_AGING_H
+
+#ifndef arch_supports_page_access_count
+static inline bool arch_supports_page_access_count(void)
+{
+	return false;
+}
+#endif
+
+#ifdef CONFIG_LRU_GEN
+#ifndef arch_get_lru_gen_seq
+static inline unsigned long arch_get_lru_gen_seq(struct lruvec *lruvec, struct folio *folio)
+{
+	int type = folio_is_file_lru(folio);
+
+	return lruvec->lrugen.min_seq[type];
+}
+#endif
+#endif /* CONFIG_LRU_GEN */
+
+#endif
+
+
mm/rmap.c +3 -1
index b616870a09be..1ef3cb8119d5 100644
@@ -74,6 +74,7 @@
 #include <linux/memremap.h>
 #include <linux/userfaultfd_k.h>
 #include <linux/mm_inline.h>
+#include <linux/page_aging.h>
 
 #include <asm/tlbflush.h>
 
@@ -825,7 +826,8 @@ static bool folio_referenced_one(struct folio *folio,
 		if (pvmw.pte) {
 			if (lru_gen_enabled() && pte_young(*pvmw.pte) &&
 			    !(vma->vm_flags & (VM_SEQ_READ | VM_RAND_READ))) {
-				lru_gen_look_around(&pvmw);
+				if (!arch_supports_page_access_count())
+					lru_gen_look_around(&pvmw);
 				referenced++;
 			}
 
mm/vmscan.c +25 -9
index f92b689af2a5..518d1482f6ab 100644
@@ -62,6 +62,7 @@
 #include <linux/swapops.h>
 #include <linux/balloon_compaction.h>
 #include <linux/sched/sysctl.h>
+#include <linux/page_aging.h>
 
 #include "internal.h"
 #include "swap.h"
@@ -4934,7 +4935,7 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw)
 static bool sort_folio(struct lruvec *lruvec, struct folio *folio, int tier_idx)
 {
 	bool success;
-	int gen = folio_lru_gen(folio);
+	int gen;
 	int type = folio_is_file_lru(folio);
 	int zone = folio_zonenum(folio);
 	int delta = folio_nr_pages(folio);
@@ -4942,7 +4943,6 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, int tier_idx)
 	int tier = lru_tier_from_refs(refs);
 	struct lru_gen_struct *lrugen = &lruvec->lrugen;
 
-	VM_WARN_ON_ONCE_FOLIO(gen >= MAX_NR_GENS, folio);
 
 	/* unevictable */
 	if (!folio_evictable(folio)) {
@@ -4963,8 +4963,14 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, int tier_idx)
 		return true;
 	}
 
-	/* promoted */
+	if (!arch_supports_page_access_count()) {
+		gen = folio_lru_gen(folio);
+		VM_WARN_ON_ONCE_FOLIO(gen >= MAX_NR_GENS, folio);
+	} else
+		gen = lru_gen_from_seq(arch_get_lru_gen_seq(lruvec, folio));
+
 	if (gen != lru_gen_from_seq(lrugen->min_seq[type])) {
+		/* promote the folio */
 		list_move(&folio->lru, &lrugen->lists[gen][type][zone]);
 		return true;
 	}
@@ -5464,12 +5470,22 @@ bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, bool reclaimi
 	 */
 	if (folio_test_active(folio))
 		seq = lrugen->max_seq;
-	else if ((type == LRU_GEN_ANON && !folio_test_swapcache(folio)) ||
-		 (folio_test_reclaim(folio) &&
-		  (folio_test_dirty(folio) || folio_test_writeback(folio))))
-		seq = lrugen->min_seq[type] + 1;
-	else
-		seq = lrugen->min_seq[type];
+	else {
+		/*
+		 * For a non active folio use the arch based
+		 * aging details to derive the MGLRU generation.
+		 */
+		seq = arch_get_lru_gen_seq(lruvec, folio);
+
+		if (seq == lrugen->min_seq[type]) {
+			if ((type == LRU_GEN_ANON &&
+			     !folio_test_swapcache(folio)) ||
+			    (folio_test_reclaim(folio) &&
+			     (folio_test_dirty(folio) ||
+			      folio_test_writeback(folio))))
+				seq = lrugen->min_seq[type] + 1;
+		}
+	}
 
 	gen = lru_gen_from_seq(seq);
 	flags = (gen + 1UL) << LRU_GEN_PGOFF;
Signature · 1 line
2.39.2
#4 lore raw

RFC PATCH 4/7 mm: multi-gen LRU: support different page aging mechanism

In reply to Aneesh Kumar K.V

Some architectures can handle different methods for determining page access count. We may want to do architecture-specific tasks reclassifying generation temperature etc during aging for such architecture. Add an arch hook to support that.

---
 include/linux/page_aging.h | 14 +++++++++++++-
 mm/vmscan.c                |  8 +++++++-
 2 files changed, 20 insertions(+), 2 deletions(-)
include/linux/page_aging.h +13 -1
index ab77f4578916..d7c63ce0d824 100644
@@ -11,6 +11,10 @@ static inline bool arch_supports_page_access_count(void)
 #endif
 
 #ifdef CONFIG_LRU_GEN
+bool __try_to_inc_max_seq(struct lruvec *lruvec,
+			  unsigned long max_seq, int scan_priority,
+			  bool can_swap, bool force_scan);
+
 #ifndef arch_get_lru_gen_seq
 static inline unsigned long arch_get_lru_gen_seq(struct lruvec *lruvec, struct folio *folio)
 {
@@ -19,8 +23,16 @@ static inline unsigned long arch_get_lru_gen_seq(struct lruvec *lruvec, struct f
 	return lruvec->lrugen.min_seq[type];
 }
 #endif
-#endif /* CONFIG_LRU_GEN */
 
+#ifndef arch_try_to_inc_max_seq
+static inline bool arch_try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq,
+					   int scan_priority, bool can_swap, bool force_scan)
+{
+	return __try_to_inc_max_seq(lruvec, max_seq, scan_priority, can_swap, force_scan);
+}
+#endif
+
+#endif /* CONFIG_LRU_GEN */
 #endif
 
 
mm/vmscan.c +7 -1
index 518d1482f6ab..c8b98201f0b0 100644
@@ -4536,7 +4536,13 @@ bool __try_to_inc_max_seq(struct lruvec *lruvec,
 static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq,
 			       int scan_priority, bool can_swap, bool force_scan)
 {
-	return __try_to_inc_max_seq(lruvec, max_seq, scan_priority, can_swap, force_scan);
+	if (arch_supports_page_access_count())
+		return arch_try_to_inc_max_seq(lruvec, max_seq,
+					       scan_priority, can_swap,
+					       force_scan);
+	else
+		return __try_to_inc_max_seq(lruvec, max_seq,
+					    scan_priority, can_swap, force_scan);
 }
 #endif
 
Signature · 1 line
2.39.2
#5 lore raw

RFC PATCH 5/7 powerpc/mm: Add page access count support

In reply to Aneesh Kumar K.V

Hot Cold Affinity engine is a facility provided by POWER10 where each access to a page is counted and the access count value decreased/decayed if not accessed in a time window. There is a 32-bit counter for each page.

This patch uses the HCA engine to provide page access count on POWER10 and uses the same with multi-gen LRU to classify the page to correct LRU generation. This uses a simple page classification mechanism where pages are sampled from the youngest and oldest generation to find the max and min page hotness in the lruvec. This value is later used to sort every page to the right generation.

The max and min hotness range is established during aging when new generations are created.

---
 arch/powerpc/Kconfig                  |  10 +
 arch/powerpc/include/asm/hca.h        |  49 +++++
 arch/powerpc/include/asm/page_aging.h |  35 ++++
 arch/powerpc/mm/Makefile              |   1 +
 arch/powerpc/mm/hca.c                 | 275 ++++++++++++++++++++++++++
 include/linux/mmzone.h                |   5 +
 include/linux/page_aging.h            |   5 +
 mm/Kconfig                            |   4 +
 mm/vmscan.c                           |   5 +-
 9 files changed, 387 insertions(+), 2 deletions(-)
 create mode 100644 arch/powerpc/include/asm/hca.h
 create mode 100644 arch/powerpc/include/asm/page_aging.h
 create mode 100644 arch/powerpc/mm/hca.c
arch/powerpc/Kconfig +10 -0
index 7a5f8dbfbdd0..71e8f23d9a96 100644
@@ -1045,6 +1045,16 @@ config PPC_SECVAR_SYSFS
 	  read/write operations on these variables. Say Y if you have
 	  secure boot enabled and want to expose variables to userspace.
 
+config PPC_HCA_HOTNESS
+	prompt "PowerPC HCA engine based page hotness"
+	def_bool y
+	select ARCH_HAS_PAGE_AGING
+	depends on PPC_BOOK3S_64
+	help
+	  Use HCA engine to find page hotness
+
+	  If unsure, say N.
+
 endmenu
 
 config ISA_DMA_API
arch/powerpc/include/asm/hca.h new file +49 -0

49 lines not on this page · raw message

arch/powerpc/include/asm/page_aging.h new file +35 -0
new file mode 100644
index 000000000000..0d98cd877308
@@ -0,0 +1,35 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+
+#ifndef _ASM_POWERPC_PAGE_AGING_H_
+#define _ASM_POWERPC_PAGE_AGING_H_
+
+#ifdef CONFIG_LRU_GEN
+extern bool hca_lru_age;
+unsigned long hca_map_lru_seq(struct lruvec *lruvec, struct folio *folio);
+bool hca_try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq,
+			    int scan_priority, bool can_swap, bool force_scan);
+
+#define arch_supports_page_access_count arch_supports_page_access_count
+static inline bool arch_supports_page_access_count(void)
+{
+	return hca_lru_age;
+}
+
+#define arch_try_to_inc_max_seq	arch_try_to_inc_max_seq
+static inline bool arch_try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq,
+					   int scan_priority, bool can_swap,
+					   bool force_scan)
+{
+	return hca_try_to_inc_max_seq(lruvec, max_seq, scan_priority,
+				      can_swap, force_scan);
+
+}
+
+#define arch_get_lru_gen_seq	arch_get_lru_gen_seq
+static inline unsigned long arch_get_lru_gen_seq(struct lruvec *lruvec, struct folio *folio)
+{
+	return hca_map_lru_seq(lruvec, folio);
+}
+
+#endif /* CONFIG_LRU_GEN */
+#endif
arch/powerpc/mm/Makefile +1 -0
index 503a6e249940..30bd4ad4aff0 100644
@@ -19,3 +19,4 @@ obj-$(CONFIG_NOT_COHERENT_CACHE) += dma-noncoherent.o
 obj-$(CONFIG_PPC_COPRO_BASE)	+= copro_fault.o
 obj-$(CONFIG_PTDUMP_CORE)	+= ptdump/
 obj-$(CONFIG_KASAN)		+= kasan/
+obj-$(CONFIG_PPC_HCA_HOTNESS)	+= hca.o
arch/powerpc/mm/hca.c new file +275 -0

275 lines not on this page · raw message

include/linux/mmzone.h +5 -0
index 0bcc5d88239a..934ad587a558 100644
@@ -425,6 +425,11 @@ struct lru_gen_struct {
 	atomic_long_t evicted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS];
 	atomic_long_t refaulted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS];
 	/* whether the multi-gen LRU is enabled */
+#ifndef CONFIG_LRU_TASK_PAGE_AGING
+	unsigned long max_hotness;
+	unsigned long min_hotness;
+	u64 next_span_scan;
+#endif
 	bool enabled;
 };
 
include/linux/page_aging.h +5 -0
index d7c63ce0d824..074c876f17e1 100644
@@ -3,6 +3,10 @@
 #ifndef _LINUX_PAGE_AGING_H
 #define _LINUX_PAGE_AGING_H
 
+#ifdef CONFIG_ARCH_HAS_PAGE_AGING
+#include <asm/page_aging.h>
+#endif
+
 #ifndef arch_supports_page_access_count
 static inline bool arch_supports_page_access_count(void)
 {
@@ -14,6 +18,7 @@ static inline bool arch_supports_page_access_count(void)
 bool __try_to_inc_max_seq(struct lruvec *lruvec,
 			  unsigned long max_seq, int scan_priority,
 			  bool can_swap, bool force_scan);
+void inc_max_seq(struct lruvec *lruvec, bool can_swap, bool force_scan);
 
 #ifndef arch_get_lru_gen_seq
 static inline unsigned long arch_get_lru_gen_seq(struct lruvec *lruvec, struct folio *folio)
mm/Kconfig +4 -0
index ff7b209dec05..493709ac758e 100644
@@ -1181,6 +1181,10 @@ config LRU_GEN_STATS
 	  from evicted generations for debugging purpose.
 
 	  This option has a per-memcg and per-node memory overhead.
+
+config ARCH_HAS_PAGE_AGING
+	bool
+
 # }
 
 source "mm/damon/Kconfig"
mm/vmscan.c +3 -2
index c8b98201f0b0..a5f6238b3926 100644
@@ -4362,7 +4362,7 @@ static bool try_to_inc_min_seq(struct lruvec *lruvec, bool can_swap)
 	return success;
 }
 
-static void inc_max_seq(struct lruvec *lruvec, bool can_swap, bool force_scan)
+void inc_max_seq(struct lruvec *lruvec, bool can_swap, bool force_scan)
 {
 	int prev, next;
 	int type, zone;
@@ -4420,6 +4420,7 @@ static void inc_max_seq(struct lruvec *lruvec, bool can_swap, bool force_scan)
 #endif
 	spin_unlock_irq(&lruvec->lru_lock);
 }
+
 #ifdef CONFIG_LRU_TASK_PAGE_AGING
 static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq,
 			       int scan_priority, bool can_swap, bool force_scan)
@@ -5861,7 +5862,7 @@ static int lru_gen_seq_show(struct seq_file *m, void *v)
 		seq_printf(m, "memcg %5hu %s\n", mem_cgroup_id(memcg), path);
 	}
 
-	seq_printf(m, " node %5d\n", nid);
+	seq_printf(m, " node %5d max_hotness %ld min_hotness %ld\n", nid, lrugen->max_hotness, lrugen->min_hotness);
 
 	if (!full)
 		seq = min_seq[LRU_GEN_ANON];
Signature · 1 line
2.39.2
#6 lore raw

RFC PATCH 6/7 powerpc/mm: Clear page access count on allocation

In reply to Aneesh Kumar K.V

Clear the HCA access count value on allocation.

---
 arch/powerpc/include/asm/page.h |  5 +++++
 arch/powerpc/mm/hca.c           | 13 +++++++++++++
 2 files changed, 18 insertions(+)
arch/powerpc/include/asm/page.h +5 -0
index edf1dd1b0ca9..515423744193 100644
@@ -319,6 +319,11 @@ void arch_free_page(struct page *page, int order);
 #define HAVE_ARCH_FREE_PAGE
 #endif
 
+#ifdef CONFIG_PPC_HCA_HOTNESS
+void arch_alloc_page(struct page *page, int order);
+#define HAVE_ARCH_ALLOC_PAGE
+#endif
+
 struct vm_area_struct;
 
 extern unsigned long kernstart_virt_addr;
arch/powerpc/mm/hca.c +13 -0
index af6de4492ead..1e79ea89df1b 100644
@@ -261,6 +261,19 @@ static void hca_debugfs_init(void)
 	hca_backend_debugfs_init(hca_debugfs_root);
 }
 
+void arch_alloc_page(struct page *page, int order)
+{
+	int i;
+
+	if (!hca_clear_entry)
+		return;
+
+	/* zero the counter value when we allocate the page */
+	for (i = 0; i < (1 << order); i++)
+		hca_clear_entry(page_to_pfn(page + i));
+}
+EXPORT_SYMBOL(arch_alloc_page);
+
 static int __init hca_init(void)
 {
 	if (!hca_backend_debugfs_init) {
Signature · 1 line
2.39.2
#7 lore raw

RFC PATCH 7/7 mm: multi-gen LRU: Shrink folio list without checking for page table reference

In reply to Aneesh Kumar K.V

If arch supports page access count, we would have already sorted the lru before collecting reclaimable pages. So unconditionally reclaim them in shrink_folio_list

---
 mm/vmscan.c | 25 ++++++++-----------------
 1 file changed, 8 insertions(+), 17 deletions(-)
mm/vmscan.c +8 -17
index a5f6238b3926..d9eb6a4d2975 100644
@@ -5242,7 +5242,8 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap
 	if (list_empty(&list))
 		return scanned;
 retry:
-	reclaimed = shrink_folio_list(&list, pgdat, sc, &stat, false);
+	reclaimed = shrink_folio_list(&list, pgdat, sc,
+				      &stat, arch_supports_page_access_count());
 	sc->nr_reclaimed += reclaimed;
 
 	list_for_each_entry_safe_reverse(folio, next, &list, lru) {
@@ -5477,22 +5478,12 @@ bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, bool reclaimi
 	 */
 	if (folio_test_active(folio))
 		seq = lrugen->max_seq;
-	else {
-		/*
-		 * For a non active folio use the arch based
-		 * aging details to derive the MGLRU generation.
-		 */
-		seq = arch_get_lru_gen_seq(lruvec, folio);
-
-		if (seq == lrugen->min_seq[type]) {
-			if ((type == LRU_GEN_ANON &&
-			     !folio_test_swapcache(folio)) ||
-			    (folio_test_reclaim(folio) &&
-			     (folio_test_dirty(folio) ||
-			      folio_test_writeback(folio))))
-				seq = lrugen->min_seq[type] + 1;
-		}
-	}
+	else if ((type == LRU_GEN_ANON && !folio_test_swapcache(folio)) ||
+		 (folio_test_reclaim(folio) &&
+		  (folio_test_dirty(folio) || folio_test_writeback(folio))))
+		seq = lrugen->min_seq[type] + 1;
+	else
+		seq = lrugen->min_seq[type];
 
 	gen = lru_gen_from_seq(seq);
 	flags = (gen + 1UL) << LRU_GEN_PGOFF;
Signature · 1 line
2.39.2