Support arch-specific page aging mechanism
Subsystems (12)
cover Support arch-specific page aging mechanism
Architectures like powerpc support page access count mechanism which can be used for better identification of hot/cold pages in the system. POWER10 supports a 32-bit page access count which is incremented based on page access and decremented based on time decay. The page access count is incremented based on physical address filtering and hence should count access via page table(mmap) and read/write syscall.
This patch series updates multi-gen LRU to use this page access count instead of the page table reference bit to classify a page into a generation. Pages are classified into generation during the sorting phase of reclaim. Currently sorting phase use generation details stored in page flags and with this change, we can avoid using page flags for storing generation. That will free the 3 bits in page flag used to store generation. Since the page access counting mechanism can also count access via read/write, we can look at avoiding using tier index in page flags. That should free the 2 bits in page flag used for REFS (this is not done in this patch).
I also added a patch that did the below
@@ -5243,7 +5243,8 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap
if (list_empty(&list))
return scanned;
retry:
- reclaimed = shrink_folio_list(&list, pgdat, sc, &stat, false);
+ reclaimed = shrink_folio_list(&list, pgdat, sc,
+ &stat, arch_supports_page_access_count());
sc->nr_reclaimed += reclaimed;
The performance did improve, but that did result in a large increase in the workingset_refault_anon. I think this is because it takes some minimal access to classify the pages to the younger generation and we can have high page refaults during that window.
PATCH 2 did result in some improvements on powerpc because it is removing all additional code that is not used in page classification.
memcached:
patch details Total Ops/sec: mglru 160821 PATCH 2 164572
mongodb:
Patch details Throughput(Ops/sec) mglru 92987 PATCH 2 93740
Enabling the architecture-supported page access count does impact workload performance since updating the access count involves some memory access overhead. Another challenge with page access count is in determining relative hotness between pages. I did try two methods density-based clustering and kmean clustering to classify pages to LRU generation based on sampled hotness. Doing more work during page classification is resulting in increased lock contention on lru_lock and hence hurts performance.
memcached:
patch details Total Ops/sec: arch page access count 161940 avoid folio_check_reference 171631 (but refault count increase from 2606765 -> 7793482)
mongodb:
Patch details Throughput(Ops/sec) arch page access count 92533 avoid folio_check_reference 91105 ( refault: 828951 -> 4592539)
The patch series does show that using page access count is not resulting in any regression and can keep the code simpler w.r.t different feedback loop used during multi-gen LRU reclaim. This also saves some bits in page->flags . It was also observed that overhead in counting page access is not that high and can be mitigated by further tuning of the page generation classification logic. This also enables us to start looking at using page access count in other parts of the linux kernel like page promotion. I haven't been able to measure the impact on page promotion yet due to hardware availability.
Aneesh Kumar K.V (7):
mm: Move some code around so that next patch is simpler
mm: Don't build multi-gen LRU page table walk code on architecture not
supported
mm: multi-gen LRU: avoid using generation stored in page flags for
generation
mm: multi-gen LRU: support different page aging mechanism powerpc/mm: Add page access count support
powerpc/mm: Clear page access count on allocation
mm: multi-gen LRU: Shrink folio list without checking for page table
reference
arch/Kconfig | 3 + arch/arm64/Kconfig | 1 + arch/powerpc/Kconfig | 10 + arch/powerpc/include/asm/hca.h | 49 ++++ arch/powerpc/include/asm/page.h | 5 + arch/powerpc/include/asm/page_aging.h | 35 +++ arch/powerpc/mm/Makefile | 1 + arch/powerpc/mm/hca.c | 288 ++++++++++++++++++++ arch/x86/Kconfig | 1 + include/linux/memcontrol.h | 2 +- include/linux/mm_inline.h | 47 +--- include/linux/mm_types.h | 8 +- include/linux/mmzone.h | 15 +- include/linux/page_aging.h | 43 +++ include/linux/swap.h | 2 +- kernel/fork.c | 2 +- mm/Kconfig | 4 + mm/memcontrol.c | 2 +- mm/rmap.c | 4 +- mm/vmscan.c | 372 ++++++++++++++++++++++---- 20 files changed, 780 insertions(+), 114 deletions(-) create mode 100644 arch/powerpc/include/asm/hca.h create mode 100644 arch/powerpc/include/asm/page_aging.h create mode 100644 arch/powerpc/mm/hca.c create mode 100644 include/linux/page_aging.h
Signature · 1 line
2.39.2
RFC PATCH 1/7 mm: Move some code around so that next patch is simpler
Move lrur_gen_add_folio to .c. We will support arch specific mapping of page access count to generation in a later patch and will use that when adding folio to lruvec. This move enables that.
No functional change in this patch.
--- include/linux/mm_inline.h | 47 +------------- mm/vmscan.c | 127 ++++++++++++++++++++++++++------------ 2 files changed, 88 insertions(+), 86 deletions(-)
include/linux/mm_inline.h +1 -46
@@ -217,52 +217,7 @@ static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *foliVM_WARN_ON_ONCE (lru_gen_is_active (lruvec, old_gen) && !lru_gen_is_active (lruvec, new_gen)); } -static inline bool lru_gen_add_folio (struct lruvec *lruvec,struct folio *folio,bool reclaiming) -{ -unsigned long seq; -unsigned long flags; -int gen =folio_lru_gen (folio); -int type =folio_is_file_lru (folio); -int zone =folio_zonenum (folio); -struct lru_gen_struct *lrugen = &lruvec->lrugen; - -VM_WARN_ON_ONCE_FOLIO (gen !=-1 , folio); - -if (folio_test_unevictable (folio) || !lrugen->enabled) -return false; -/* -* There are three common cases for this page: -* 1. If it's hot, e.g., freshly faulted in or previously hot and -* migrated, add it to the youngest generation. -* 2. If it's cold but can't be evicted immediately, i.e., an anon page -* not in swapcache or a dirty page pending writeback, add it to the -* second oldest generation. -* 3. Everything else (clean, cold) is added to the oldest generation. -*/ -if (folio_test_active (folio)) - seq = lrugen->max_seq; -else if ((type ==LRU_GEN_ANON && !folio_test_swapcache (folio)) || - (folio_test_reclaim (folio) && - (folio_test_dirty (folio) ||folio_test_writeback (folio)))) - seq = lrugen->min_seq[type] +1 ; -else - seq = lrugen->min_seq[type]; - - gen =lru_gen_from_seq (seq); - flags = (gen +1UL ) <<LRU_GEN_PGOFF ; -/* see the comment on MIN_NR_GENS about PG_active */ -set_mask_bits (&folio->flags,LRU_GEN_MASK |BIT (PG_active), flags); - -lru_gen_update_size (lruvec, folio,-1 , gen); -/* for folio_rotate_reclaimable() */ -if (reclaiming) -list_add_tail (&folio->lru, &lrugen->lists[gen][type][zone]); -else -list_add (&folio->lru, &lrugen->lists[gen][type][zone]); - -return true; -} - +bool lru_gen_add_folio (struct lruvec *lruvec,struct folio *folio,bool reclaiming);static inline bool lru_gen_del_folio (struct lruvec *lruvec,struct folio *folio,bool reclaiming) {unsigned long flags;
mm/vmscan.c +87 -40
151 lines not on this page · raw message
Signature · 1 line
2.39.2
RFC PATCH 2/7 mm: Don't build multi-gen LRU page table walk code on architecture not supported
In reply to Aneesh Kumar K.VNot all architecture supports hardware atomic updates of access bits. On such an arch, we don't use page table walk to classify pages into generations. Add a kernel config option and remove adding all the page table walk code on such architecture.
lru_gen_look_around() code is duplicated because lru_gen_mm_walk is not available always.
This patch did result in some improvements on powerpc because it is removing all additional code that is not used in page classification.
memcached:
patch details Total Ops/sec: mglru 160821 PATCH 2 164572
mongodb:
Patch details Throughput(Ops/sec) mglru 92987 PATCH 2 93740
--- arch/Kconfig | 3 + arch/arm64/Kconfig | 1 + arch/x86/Kconfig | 1 + include/linux/memcontrol.h | 2 +- include/linux/mm_types.h | 8 +- include/linux/mmzone.h | 10 +- include/linux/swap.h | 2 +- kernel/fork.c | 2 +- mm/memcontrol.c | 2 +- mm/vmscan.c | 221 ++++++++++++++++++++++++++++++++++--- 10 files changed, 230 insertions(+), 22 deletions(-)
arch/Kconfig +3 -0
@@ -1426,6 +1426,9 @@ config DYNAMIC_SIGFRAME config HAVE_ARCH_NODE_DEV_GROUP bool +config LRU_TASK_PAGE_AGING + bool + config ARCH_HAS_NONLEAF_PMD_YOUNG bool help
arch/arm64/Kconfig +1 -0
@@ -219,6 +219,7 @@ config ARM64 select IRQ_DOMAIN select IRQ_FORCED_THREADING select KASAN_VMALLOC if KASAN + select LRU_TASK_PAGE_AGING select MODULES_USE_ELF_RELA select NEED_DMA_MAP_STATE select NEED_SG_DMA_LENGTH
arch/x86/Kconfig +1 -0
@@ -274,6 +274,7 @@ config X86 select HAVE_GENERIC_VDSO select HOTPLUG_SMT if SMP select IRQ_FORCED_THREADING + select LRU_TASK_PAGE_AGING select NEED_PER_CPU_EMBED_FIRST_CHUNK select NEED_PER_CPU_PAGE_FIRST_CHUNK select NEED_SG_DMA_LENGTH
include/linux/memcontrol.h +1 -1
@@ -320,7 +320,7 @@ struct mem_cgroup {struct deferred_split deferred_split_queue;#endif -#ifdef CONFIG_LRU_GEN +#ifdef CONFIG_LRU_TASK_PAGE_AGING /* per-memcg mm_struct list */ struct lru_gen_mm_list mm_list;#endif
include/linux/mm_types.h +4 -4
@@ -796,7 +796,7 @@ struct mm_struct {*/ unsigned long ksm_rmap_items;#endif -#ifdef CONFIG_LRU_GEN +#ifdef CONFIG_LRU_TASK_PAGE_AGING struct { /* this mm_struct is on lru_gen_mm_list */ struct list_head list;
@@ -811,7 +811,7 @@ struct mm_struct {struct mem_cgroup *memcg; #endif } lru_gen; -#endif /* CONFIG_LRU_GEN */ +#endif /* CONFIG_LRU_TASK_PAGE_AGING */ } __randomize_layout;/*
@@ -839,7 +839,7 @@ static inline cpumask_t *mm_cpumask(struct mm_struct *mm)return (struct cpumask *)&mm->cpu_bitmap; } -#ifdef CONFIG_LRU_GEN +#ifdef CONFIG_LRU_TASK_PAGE_AGING struct lru_gen_mm_list {/* mm_struct list for page table walkers */
@@ -873,7 +873,7 @@ static inline void lru_gen_use_mm(struct mm_struct *mm) WRITE_ONCE(mm->lru_gen.bitmap, -1); } -#else /* !CONFIG_LRU_GEN */ +#else /* !CONFIG_LRU_TASK_PAGE_AGING */ static inline void lru_gen_add_mm (struct mm_struct *mm) {
include/linux/mmzone.h +9 -1
@@ -428,6 +428,7 @@ struct lru_gen_struct {bool enabled; }; +#ifdef CONFIG_LRU_TASK_PAGE_AGING enum {MM_LEAF_TOTAL ,/* total leaf entries */ MM_LEAF_OLD, /* old leaf entries */
@@ -474,6 +475,7 @@ struct lru_gen_mm_walk {bool can_swap; bool force_scan; }; +#endif void lru_gen_init_lruvec (struct lruvec *lruvec);void lru_gen_look_around (struct page_vma_mapped_walk *pvmw);
@@ -525,8 +527,14 @@ struct lruvec {#ifdef CONFIG_LRU_GEN /* evictable pages divided into generations */ struct lru_gen_struct lrugen; +#ifdef CONFIG_LRU_TASK_PAGE_AGING /* to concurrently iterate lru_gen_mm_list */ struct lru_gen_mm_state mm_state; +#else +/* for concurrent update of max_seq without holding lru_lock */ +struct wait_queue_head seq_update_wait; +bool seq_update_progress; +#endif #endif #ifdef CONFIG_MEMCG struct pglist_data *pgdat;
@@ -1240,7 +1248,7 @@ typedef struct pglist_data {unsigned long flags; -#ifdef CONFIG_LRU_GEN +#ifdef CONFIG_LRU_TASK_PAGE_AGING /* kswap mm walk data */ struct lru_gen_mm_walk mm_walk;#endif
include/linux/swap.h +1 -1
@@ -154,7 +154,7 @@ union swap_header {*/ struct reclaim_state {unsigned long reclaimed_slab; -#ifdef CONFIG_LRU_GEN +#ifdef CONFIG_LRU_TASK_PAGE_AGING /* per-thread mm walk data */ struct lru_gen_mm_walk *mm_walk;#endif
kernel/fork.c +1 -1
@@ -2708,7 +2708,7 @@ pid_t kernel_clone(struct kernel_clone_args *args)get_task_struct (p); } -if (IS_ENABLED (CONFIG_LRU_GEN ) && !(clone_flags &CLONE_VM )) { +if (IS_ENABLED (CONFIG_LRU_TASK_PAGE_AGING ) && !(clone_flags &CLONE_VM )) {/* lock the task to synchronize with memcg migration */ task_lock (p);lru_gen_add_mm (p->mm);
mm/memcontrol.c +1 -1
@@ -6305,7 +6305,7 @@ static void mem_cgroup_move_task(void) }#endif -#ifdef CONFIG_LRU_GEN +#ifdef CONFIG_LRU_TASK_PAGE_AGING static void mem_cgroup_attach (struct cgroup_taskset *tset) {struct task_struct *task;
mm/vmscan.c +208 -13
365 lines not on this page · raw message
Signature · 1 line
2.39.2
RFC PATCH 3/7 mm: multi-gen LRU: avoid using generation stored in page flags for generation
In reply to Aneesh Kumar K.VSome architectures can handle different methods for determining page access count. In such case, we may not really need to use page flags for tracking generation. We can possibly derive generation directly from arch-supported access count values. Hence avoid using page flags to store generation in that case.
--- include/linux/page_aging.h | 26 ++++++++++++++++++++++++++ mm/rmap.c | 4 +++- mm/vmscan.c | 34 +++++++++++++++++++++++++--------- 3 files changed, 54 insertions(+), 10 deletions(-) create mode 100644 include/linux/page_aging.h
include/linux/page_aging.h new file +26 -0
@@ -0,0 +1,26 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ + +#ifndef _LINUX_PAGE_AGING_H +#define _LINUX_PAGE_AGING_H + +#ifndef arch_supports_page_access_count +static inline bool arch_supports_page_access_count (void ) +{ +return false; +} +#endif + +#ifdef CONFIG_LRU_GEN +#ifndef arch_get_lru_gen_seq +static inline unsigned long arch_get_lru_gen_seq (struct lruvec *lruvec,struct folio *folio) +{ +int type =folio_is_file_lru (folio); + +return lruvec->lrugen.min_seq[type]; +} +#endif +#endif /* CONFIG_LRU_GEN */ + +#endif + +
mm/rmap.c +3 -1
@@ -74,6 +74,7 @@#include <linux/memremap.h> #include <linux/userfaultfd_k.h> #include <linux/mm_inline.h> +#include <linux/page_aging.h> #include <asm/tlbflush.h>
@@ -825,7 +826,8 @@ static bool folio_referenced_one(struct folio *folio,if (pvmw.pte) {if (lru_gen_enabled () &&pte_young (*pvmw.pte) && !(vma->vm_flags & (VM_SEQ_READ |VM_RAND_READ ))) { -lru_gen_look_around (&pvmw); +if (!arch_supports_page_access_count ()) +lru_gen_look_around (&pvmw); referenced++; }
mm/vmscan.c +25 -9
@@ -62,6 +62,7 @@#include <linux/swapops.h> #include <linux/balloon_compaction.h> #include <linux/sched/sysctl.h> +#include <linux/page_aging.h> #include "internal.h" #include "swap.h"
@@ -4934,7 +4935,7 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw)static bool sort_folio (struct lruvec *lruvec,struct folio *folio,int tier_idx) {bool success; -int gen =folio_lru_gen (folio); +int gen;int type =folio_is_file_lru (folio);int zone =folio_zonenum (folio);int delta =folio_nr_pages (folio);
@@ -4942,7 +4943,6 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, int tier_idx)int tier =lru_tier_from_refs (refs);struct lru_gen_struct *lrugen = &lruvec->lrugen; -VM_WARN_ON_ONCE_FOLIO (gen >=MAX_NR_GENS , folio);/* unevictable */ if (!folio_evictable (folio)) {
@@ -4963,8 +4963,14 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, int tier_idx)return true; } -/* promoted */ +if (!arch_supports_page_access_count ()) { + gen =folio_lru_gen (folio); +VM_WARN_ON_ONCE_FOLIO (gen >=MAX_NR_GENS , folio); + }else + gen =lru_gen_from_seq (arch_get_lru_gen_seq (lruvec, folio)); +if (gen !=lru_gen_from_seq (lrugen->min_seq[type])) { +/* promote the folio */ list_move (&folio->lru, &lrugen->lists[gen][type][zone]);return true; }
@@ -5464,12 +5470,22 @@ bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, bool reclaimi*/ if (folio_test_active (folio)) seq = lrugen->max_seq; -else if ((type ==LRU_GEN_ANON && !folio_test_swapcache (folio)) || - (folio_test_reclaim (folio) && - (folio_test_dirty (folio) ||folio_test_writeback (folio)))) - seq = lrugen->min_seq[type] +1 ; -else - seq = lrugen->min_seq[type]; +else { +/* +* For a non active folio use the arch based +* aging details to derive the MGLRU generation. +*/ + seq =arch_get_lru_gen_seq (lruvec, folio); + +if (seq == lrugen->min_seq[type]) { +if ((type ==LRU_GEN_ANON && + !folio_test_swapcache (folio)) || + (folio_test_reclaim (folio) && + (folio_test_dirty (folio) || +folio_test_writeback (folio)))) + seq = lrugen->min_seq[type] +1 ; + } + } gen =lru_gen_from_seq (seq); flags = (gen +1UL ) <<LRU_GEN_PGOFF ;
Signature · 1 line
2.39.2
RFC PATCH 4/7 mm: multi-gen LRU: support different page aging mechanism
In reply to Aneesh Kumar K.VSome architectures can handle different methods for determining page access count. We may want to do architecture-specific tasks reclassifying generation temperature etc during aging for such architecture. Add an arch hook to support that.
--- include/linux/page_aging.h | 14 +++++++++++++- mm/vmscan.c | 8 +++++++- 2 files changed, 20 insertions(+), 2 deletions(-)
include/linux/page_aging.h +13 -1
@@ -11,6 +11,10 @@ static inline bool arch_supports_page_access_count(void)#endif #ifdef CONFIG_LRU_GEN +bool __try_to_inc_max_seq(struct lruvec *lruvec, + unsigned long max_seq, int scan_priority, + bool can_swap, bool force_scan); + #ifndef arch_get_lru_gen_seq static inline unsigned long arch_get_lru_gen_seq(struct lruvec *lruvec, struct folio *folio) {
@@ -19,8 +23,16 @@ static inline unsigned long arch_get_lru_gen_seq(struct lruvec *lruvec, struct freturn lruvec->lrugen.min_seq[type]; }#endif -#endif /* CONFIG_LRU_GEN */ +#ifndef arch_try_to_inc_max_seq +static inline bool arch_try_to_inc_max_seq (struct lruvec *lruvec,unsigned long max_seq, +int scan_priority,bool can_swap,bool force_scan) +{ +return __try_to_inc_max_seq (lruvec, max_seq, scan_priority, can_swap, force_scan); +} +#endif + +#endif /* CONFIG_LRU_GEN */ #endif
mm/vmscan.c +7 -1
@@ -4536,7 +4536,13 @@ bool __try_to_inc_max_seq(struct lruvec *lruvec,static bool try_to_inc_max_seq (struct lruvec *lruvec,unsigned long max_seq,int scan_priority,bool can_swap,bool force_scan) { -return __try_to_inc_max_seq (lruvec, max_seq, scan_priority, can_swap, force_scan); +if (arch_supports_page_access_count ()) +return arch_try_to_inc_max_seq (lruvec, max_seq, + scan_priority, can_swap, + force_scan); +else +return __try_to_inc_max_seq (lruvec, max_seq, + scan_priority, can_swap, force_scan); }#endif
Signature · 1 line
2.39.2
RFC PATCH 5/7 powerpc/mm: Add page access count support
In reply to Aneesh Kumar K.VHot Cold Affinity engine is a facility provided by POWER10 where each access to a page is counted and the access count value decreased/decayed if not accessed in a time window. There is a 32-bit counter for each page.
This patch uses the HCA engine to provide page access count on POWER10 and uses the same with multi-gen LRU to classify the page to correct LRU generation. This uses a simple page classification mechanism where pages are sampled from the youngest and oldest generation to find the max and min page hotness in the lruvec. This value is later used to sort every page to the right generation.
The max and min hotness range is established during aging when new generations are created.
--- arch/powerpc/Kconfig | 10 + arch/powerpc/include/asm/hca.h | 49 +++++ arch/powerpc/include/asm/page_aging.h | 35 ++++ arch/powerpc/mm/Makefile | 1 + arch/powerpc/mm/hca.c | 275 ++++++++++++++++++++++++++ include/linux/mmzone.h | 5 + include/linux/page_aging.h | 5 + mm/Kconfig | 4 + mm/vmscan.c | 5 +- 9 files changed, 387 insertions(+), 2 deletions(-) create mode 100644 arch/powerpc/include/asm/hca.h create mode 100644 arch/powerpc/include/asm/page_aging.h create mode 100644 arch/powerpc/mm/hca.c
arch/powerpc/Kconfig +10 -0
@@ -1045,6 +1045,16 @@ config PPC_SECVAR_SYSFS read/write operations on these variables. Say Y if you have secure boot enabled and want to expose variables to userspace. +config PPC_HCA_HOTNESS + prompt "PowerPC HCA engine based page hotness" + def_bool y + select ARCH_HAS_PAGE_AGING + depends on PPC_BOOK3S_64 + help + Use HCA engine to find page hotness + + If unsure, say N. + endmenu config ISA_DMA_API
arch/powerpc/include/asm/hca.h new file +49 -0
49 lines not on this page · raw message
arch/powerpc/include/asm/page_aging.h new file +35 -0
@@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ + +#ifndef _ASM_POWERPC_PAGE_AGING_H_ +#define _ASM_POWERPC_PAGE_AGING_H_ + +#ifdef CONFIG_LRU_GEN +extern bool hca_lru_age; +unsigned long hca_map_lru_seq (struct lruvec *lruvec,struct folio *folio); +bool hca_try_to_inc_max_seq (struct lruvec *lruvec,unsigned long max_seq, +int scan_priority,bool can_swap,bool force_scan); + +#define arch_supports_page_access_count arch_supports_page_access_count +static inline bool arch_supports_page_access_count (void ) +{ +return hca_lru_age; +} + +#define arch_try_to_inc_max_seq arch_try_to_inc_max_seq +static inline bool arch_try_to_inc_max_seq (struct lruvec *lruvec,unsigned long max_seq, +int scan_priority,bool can_swap, +bool force_scan) +{ +return hca_try_to_inc_max_seq (lruvec, max_seq, scan_priority, + can_swap, force_scan); + +} + +#define arch_get_lru_gen_seq arch_get_lru_gen_seq +static inline unsigned long arch_get_lru_gen_seq (struct lruvec *lruvec,struct folio *folio) +{ +return hca_map_lru_seq (lruvec, folio); +} + +#endif /* CONFIG_LRU_GEN */ +#endif
arch/powerpc/mm/Makefile +1 -0
@@ -19,3 +19,4 @@ obj-$(CONFIG_NOT_COHERENT_CACHE) += dma-noncoherent.o obj-$(CONFIG_PPC_COPRO_BASE) += copro_fault.o obj-$(CONFIG_PTDUMP_CORE) += ptdump/ obj-$(CONFIG_KASAN) += kasan/ +obj-$(CONFIG_PPC_HCA_HOTNESS) += hca.o
arch/powerpc/mm/hca.c new file +275 -0
275 lines not on this page · raw message
include/linux/mmzone.h +5 -0
@@ -425,6 +425,11 @@ struct lru_gen_struct {atomic_long_t evicted[NR_HIST_GENS ][ANON_AND_FILE ][MAX_NR_TIERS ];atomic_long_t refaulted[NR_HIST_GENS ][ANON_AND_FILE ][MAX_NR_TIERS ];/* whether the multi-gen LRU is enabled */ +#ifndef CONFIG_LRU_TASK_PAGE_AGING +unsigned long max_hotness; +unsigned long min_hotness; +u64 next_span_scan; +#endif bool enabled; };
include/linux/page_aging.h +5 -0
@@ -3,6 +3,10 @@ #ifndef _LINUX_PAGE_AGING_H #define _LINUX_PAGE_AGING_H +#ifdef CONFIG_ARCH_HAS_PAGE_AGING +#include <asm/page_aging.h> +#endif + #ifndef arch_supports_page_access_count static inline bool arch_supports_page_access_count(void) {
@@ -14,6 +18,7 @@ static inline bool arch_supports_page_access_count(void)bool __try_to_inc_max_seq (struct lruvec *lruvec,unsigned long max_seq,int scan_priority,bool can_swap,bool force_scan); +void inc_max_seq (struct lruvec *lruvec,bool can_swap,bool force_scan);#ifndef arch_get_lru_gen_seqstatic inline unsigned long arch_get_lru_gen_seq (struct lruvec *lruvec,struct folio *folio)
mm/Kconfig +4 -0
@@ -1181,6 +1181,10 @@ config LRU_GEN_STATS from evicted generations for debugging purpose. This option has a per-memcg and per-node memory overhead. + +config ARCH_HAS_PAGE_AGING + bool + # } source "mm/damon/Kconfig"
mm/vmscan.c +3 -2
@@ -4362,7 +4362,7 @@ static bool try_to_inc_min_seq(struct lruvec *lruvec, bool can_swap)return success; } -static void inc_max_seq (struct lruvec *lruvec,bool can_swap,bool force_scan) +void inc_max_seq (struct lruvec *lruvec,bool can_swap,bool force_scan) {int prev, next;int type, zone;
@@ -4420,6 +4420,7 @@ static void inc_max_seq(struct lruvec *lruvec, bool can_swap, bool force_scan)#endif spin_unlock_irq (&lruvec->lru_lock); } +#ifdef CONFIG_LRU_TASK_PAGE_AGING static bool try_to_inc_max_seq (struct lruvec *lruvec,unsigned long max_seq,int scan_priority,bool can_swap,bool force_scan)
@@ -5861,7 +5862,7 @@ static int lru_gen_seq_show(struct seq_file *m, void *v)seq_printf (m,"memcg %5hu %s\n" ,mem_cgroup_id (memcg), path); } -seq_printf (m," node %5d\n" , nid); +seq_printf (m," node %5d max_hotness %ld min_hotness %ld\n" , nid, lrugen->max_hotness, lrugen->min_hotness);if (!full) seq = min_seq[LRU_GEN_ANON ];
Signature · 1 line
2.39.2
RFC PATCH 6/7 powerpc/mm: Clear page access count on allocation
In reply to Aneesh Kumar K.VClear the HCA access count value on allocation.
--- arch/powerpc/include/asm/page.h | 5 +++++ arch/powerpc/mm/hca.c | 13 +++++++++++++ 2 files changed, 18 insertions(+)
arch/powerpc/include/asm/page.h +5 -0
@@ -319,6 +319,11 @@ void arch_free_page(struct page *page, int order);#define HAVE_ARCH_FREE_PAGE #endif +#ifdef CONFIG_PPC_HCA_HOTNESS +void arch_alloc_page (struct page *page,int order); +#define HAVE_ARCH_ALLOC_PAGE +#endif +struct vm_area_struct ;extern unsigned long kernstart_virt_addr;
arch/powerpc/mm/hca.c +13 -0
@@ -261,6 +261,19 @@ static void hca_debugfs_init(void)hca_backend_debugfs_init (hca_debugfs_root); } +void arch_alloc_page (struct page *page,int order) +{ +int i; + +if (!hca_clear_entry) +return ; + +/* zero the counter value when we allocate the page */ +for (i =0 ; i < (1 << order); i++) +hca_clear_entry (page_to_pfn (page + i)); +} +EXPORT_SYMBOL (arch_alloc_page); +static int __init hca_init (void ) {if (!hca_backend_debugfs_init) {
Signature · 1 line
2.39.2
RFC PATCH 7/7 mm: multi-gen LRU: Shrink folio list without checking for page table reference
In reply to Aneesh Kumar K.VIf arch supports page access count, we would have already sorted the lru before collecting reclaimable pages. So unconditionally reclaim them in shrink_folio_list
--- mm/vmscan.c | 25 ++++++++----------------- 1 file changed, 8 insertions(+), 17 deletions(-)
mm/vmscan.c +8 -17
@@ -5242,7 +5242,8 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swapif (list_empty (&list))return scanned; retry: - reclaimed =shrink_folio_list (&list, pgdat, sc, &stat, false); + reclaimed =shrink_folio_list (&list, pgdat, sc, + &stat,arch_supports_page_access_count ()); sc->nr_reclaimed += reclaimed;list_for_each_entry_safe_reverse (folio, next, &list, lru) {
@@ -5477,22 +5478,12 @@ bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, bool reclaimi*/ if (folio_test_active (folio)) seq = lrugen->max_seq; -else { -/* -* For a non active folio use the arch based -* aging details to derive the MGLRU generation. -*/ - seq =arch_get_lru_gen_seq (lruvec, folio); - -if (seq == lrugen->min_seq[type]) { -if ((type ==LRU_GEN_ANON && - !folio_test_swapcache (folio)) || - (folio_test_reclaim (folio) && - (folio_test_dirty (folio) || -folio_test_writeback (folio)))) - seq = lrugen->min_seq[type] +1 ; - } - } +else if ((type ==LRU_GEN_ANON && !folio_test_swapcache (folio)) || + (folio_test_reclaim (folio) && + (folio_test_dirty (folio) ||folio_test_writeback (folio)))) + seq = lrugen->min_seq[type] +1 ; +else + seq = lrugen->min_seq[type]; gen =lru_gen_from_seq (seq); flags = (gen +1UL ) <<LRU_GEN_PGOFF ;
Signature · 1 line
2.39.2