Multigenerational LRU Framework
Subsystems (22)
v3 cover Multigenerational LRU Framework
What's new in v3
================
1) Fixed a bug reported by the Arch Linux kernel team:
https://github.com/zen-kernel/zen-kernel/issues/207
2) Rebased to v5.13-rc2.
Highlights from v2
==================
Konstantin Kharlamov <[email protected]> reported:
My success story: I have Archlinux with 8G RAM + zswap + swap. While developing, I have lots of apps opened such as multiple LSP-servers for different langs, chats, two browsers, etc. Usually, my system gets quickly to a point of SWAP-storms, where I have to kill LSP-servers, restart browsers to free memory, etc, otherwise the system lags heavily and is barely usable.
1.5 day ago I migrated from 5.11.15 kernel to 5.12 + the LRU patchset, and I started up by opening lots of apps to create memory pressure, and worked for a day like this. Till now I had *not a single SWAP-storm*, and mind you I got 3.4G in SWAP. I was never getting to the point of 3G in SWAP before without a single SWAP-storm.
TLDR
====
The current page reclaim is too expensive in terms of CPU usage and often making poor choices about what to evict. We would like to offer an alternative framework that is performant, versatile and straightforward.
Repo
====
git fetch https://linux-mm.googlesource.com/page-reclaim refs/changes/53/1253/1
Problems
========
Notion of active/inactive
-------------------------
Data centers need to predict whether a job can successfully land on a machine without actually impacting the existing jobs. The granularity of the active/inactive is too coarse to be useful for job schedulers to make such decisions. In addition, data centers need to monitor their memory utilization for horizontal scaling. The active/inactive cannot give any insight into a pool of machines because aggregating them across multiple machines without a common frame of reference yields no meaningful results.
Phones and laptops need to make good choices about what to evict, since they are more sensitive to the major faults and the power consumption. Major faults can cause "janks" (slow UI renderings) and negatively impact user experience. The selection between anon and file types has been suboptimal because direct comparisons between them are infeasible based on the notion of active/inactive. On phones and laptops, executable pages are frequently evicted despite the fact that there are many less recently used anon pages. Conversely, on workstations building large projects, anon pages are occasionally swapped out while page cache contains many less recently used pages.
Fundamentally, the notion of active/inactive has very limited ability to measure temporal locality.
Incremental scans via rmap
--------------------------
Each incremental scan picks up at where the last scan left off and stops after it has found a handful of unreferenced pages. For workloads using a large amount of anon memory, incremental scans lose the advantage under sustained memory pressure due to high ratios of the number of scanned pages to the number of reclaimed pages. On top of this, the rmap has complex data structures. And the combined effects typically result in a high amount of CPU usage in the reclaim path.
Simply put, incremental scans via rmap have no regard for spatial locality.
Solutions
=========
Notion of generation numbers
----------------------------
The notion of generation numbers introduces a temporal dimension. Each generation is a dot on the timeline and it includes all pages that have been referenced since it was created.
Given an lruvec, scans of anon and file types and selections between them are all based on direct comparisons of generation numbers, which are simple and yet effective.
A larger number of pages can be spread out across a configurable number of generations, which are associated with timestamps and therefore aggregatable. This is specifically designed for data centers that require working set estimation and proactive reclaim.
Differential scans via page tables
----------------------------------
Each differential scan discovers all pages that have been referenced since the last scan. It walks the mm_struct list associated with an lruvec to scan page tables of processes that have been scheduled since the last scan. The cost of each differential scan is roughly proportional to the number of referenced pages it discovers. Page tables usually have good memory locality. The end result is generally a significant reduction in CPU usage, for workloads using a large amount of anon memory.
For workloads that have extremely sparse page tables, it is still possible to fall back to incremental scans via rmap.
Framework
=========
For each lruvec, evictable pages are divided into multiple generations. The youngest generation number is stored in lrugen->max_seq for both anon and file types as they are aged on an equal footing. The oldest generation numbers are stored in lrugen->min_seq[2] separately for anon and file types as clean file pages can be evicted regardless of may_swap or may_writepage. These three variables are monotonically increasing. Generation numbers are truncated into order_base_2(MAX_NR_GENS+1) bits in order to fit into page->flags. The sliding window technique is used to prevent truncated generation numbers from overlapping. Each truncated generation number is an index to lrugen->lists[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES]. Evictable pages are added to the per-zone lists indexed by lrugen->max_seq or lrugen->min_seq[2] (modulo MAX_NR_GENS), depending on their types.
Each generation is then divided into multiple tiers. Tiers represent levels of usage from file descriptors only. Pages accessed N times via file descriptors belong to tier order_base_2(N). Each generation contains at most MAX_NR_TIERS tiers, and they require additional MAX_NR_TIERS-2 bits in page->flags. In contrast to moving across generations which requires the lru lock for the list operations, moving across tiers only involves an atomic operation on page->flags and therefore has a negligible cost. A feedback loop modeled after the PID controller monitors the refault rates across all tiers and decides when to activate pages from which tiers in the reclaim path.
The framework comprises two conceptually independent components: the aging and the eviction, which can be invoked separately from user space for the purpose of working set estimation and proactive reclaim.
Aging
-----
The aging produces young generations. Given an lruvec, the aging scans page tables for referenced pages of this lruvec. Upon finding one, the aging updates its generation number to max_seq. After each round of scan, the aging increments max_seq.
The aging maintains either a system-wide mm_struct list or per-memcg mm_struct lists, and it only scans page tables of processes that have been scheduled since the last scan.
The aging is due when both of min_seq[2] reaches max_seq-1, assuming both anon and file types are reclaimable.
Eviction
--------
The eviction consumes old generations. Given an lruvec, the eviction scans the pages on the per-zone lists indexed by either of min_seq[2]. It first tries to select a type based on the values of min_seq[2]. When anon and file types are both available from the same generation, it selects the one that has a lower refault rate.
During a scan, the eviction sorts pages according to their new generation numbers, if the aging has found them referenced. It also moves pages from the tiers that have higher refault rates than tier 0 to the next generation.
When it finds all the per-zone lists of a selected type are empty, the eviction increments min_seq[2] indexed by this selected type.
Use cases
=========
High anon workloads
-------------------
Our real-world benchmark that browses popular websites in multiple Chrome tabs demonstrates 51% less CPU usage from kswapd and 52% (full) less PSI.
Without this patchset, the profile of kswapd looks like:
31.03% page_vma_mapped_walk 25.59% lzo1x_1_do_compress 4.63% do_raw_spin_lock 3.89% vma_interval_tree_iter_next 3.33% vma_interval_tree_subtree_search
With this patchset, it looks like:
49.36% lzo1x_1_do_compress 4.54% page_vma_mapped_walk 4.45% memset_erms 3.47% walk_pte_range 2.88% zram_bvec_rw
In addition, direct reclaim latency is reduced by 22% at 99th percentile and the number of refaults is reduced by 7%. Both metrics are important to phones and laptops as they are highly correlated to user experience.
High page cache workloads
-------------------------
Tiers are specifically designed to improve the performance of page cache under memory pressure. The fio/io_uring benchmark shows 14% increase in IOPS when randomly accessing in buffered I/O mode.
Without this patchset, the profile of fio/io_uring looks like:
Children Self Symbol
-----------------------------------
12.03% 0.03% __page_cache_alloc 6.53% 0.83% shrink_active_list 2.53% 0.44% mark_page_accessed
With this patchset, it looks like:
Children Self Symbol
-----------------------------------
9.45% 0.03% __page_cache_alloc 0.52% 0.46% mark_page_accessed
Working set estimation
----------------------
User space can invoke the aging by writing "+ memcg_id node_id gen [swappiness]" to /sys/kernel/debug/lru_gen. This debugfs interface also provides the birth time and the size of each generation.
For example, given a pool of machines, a job scheduler periodically invokes the aging to estimate the working set of each machine. And it ranks the machines based on the sizes of their working sets and selects the most ideal ones to land new jobs.
Proactive reclaim
-----------------
User space can invoke the eviction by writing "- memcg_id node_id gen [swappiness] [nr_to_reclaim]" to /sys/kernel/debug/lru_gen. Multiple command lines are supported, so does concatenation with delimiters.
For example, a job scheduler can invoke the eviction if it anticipates new jobs. The savings from proactive reclaim may provide certain SLA when new jobs actually land.
Yu Zhao (14):
include/linux/memcontrol.h: do not warn in page_memcg_rcu() if
!CONFIG_MEMCG
include/linux/nodemask.h: define next_memory_node() if !CONFIG_NUMA include/linux/cgroup.h: export cgroup_mutex
mm, x86: support the access bit on non-leaf PMD entries mm/vmscan.c: refactor shrink_node()
mm/workingset.c: refactor pack_shadow() and unpack_shadow()
mm: multigenerational lru: groundwork
mm: multigenerational lru: activation
mm: multigenerational lru: mm_struct list
mm: multigenerational lru: aging
mm: multigenerational lru: eviction
mm: multigenerational lru: user interface
mm: multigenerational lru: Kconfig
mm: multigenerational lru: documentation
Documentation/vm/index.rst | 1 + Documentation/vm/multigen_lru.rst | 143 ++ arch/Kconfig | 9 + arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 2 +- arch/x86/mm/pgtable.c | 5 +- fs/exec.c | 2 + fs/fuse/dev.c | 3 +- include/linux/cgroup.h | 15 +- include/linux/memcontrol.h | 7 +- include/linux/mm.h | 2 + include/linux/mm_inline.h | 234 +++ include/linux/mm_types.h | 107 ++ include/linux/mmzone.h | 117 ++ include/linux/nodemask.h | 1 + include/linux/page-flags-layout.h | 19 +- include/linux/page-flags.h | 4 +- include/linux/pgtable.h | 4 +- include/linux/swap.h | 4 +- kernel/bounds.c | 6 + kernel/events/uprobes.c | 2 +- kernel/exit.c | 1 + kernel/fork.c | 10 + kernel/kthread.c | 1 + kernel/sched/core.c | 2 + mm/Kconfig | 58 + mm/huge_memory.c | 5 +- mm/khugepaged.c | 2 +- mm/memcontrol.c | 28 + mm/memory.c | 10 +- mm/migrate.c | 2 +- mm/mm_init.c | 6 +- mm/mmzone.c | 2 + mm/rmap.c | 6 + mm/swap.c | 22 +- mm/swapfile.c | 6 +- mm/userfaultfd.c | 2 +- mm/vmscan.c | 2638 ++++++++++++++++++++++++++++- mm/workingset.c | 169 +- 39 files changed, 3498 insertions(+), 160 deletions(-) create mode 100644 Documentation/vm/multigen_lru.rst
Signature · 1 line
2.31.1.751.gd2f1c929bd-goog
PATCH v3 1/14 include/linux/memcontrol.h: do not warn in page_memcg_rcu() if !CONFIG_MEMCG
page_memcg_rcu() warns on !rcu_read_lock_held() regardless of CONFIG_MEMCG. The following legit code trips the warning when !CONFIG_MEMCG, since lock_page_memcg() and unlock_page_memcg() are empty for this config.
memcg = lock_page_memcg(page1)
(rcu_read_lock() if CONFIG_MEMCG=y)
do something to page1
if (page_memcg_rcu(page2) == memcg)
do something to page2 too as it cannot be migrated away from the memcg either.
unlock_page_memcg(page1)
(rcu_read_unlock() if CONFIG_MEMCG=y)
Locking/unlocking rcu consistently for both configs is rigorous but it also forces unnecessary locking upon users who have no interest in CONFIG_MEMCG.
This patch removes the assertion for !CONFIG_MEMCG, because page_memcg_rcu() has a few callers and there are no concerns regarding their correctness at the moment.
--- include/linux/memcontrol.h | 1 - 1 file changed, 1 deletion(-)
include/linux/memcontrol.h +0 -1
@@ -1131,7 +1131,6 @@ static inline struct mem_cgroup *page_memcg(struct page *page)static inline struct mem_cgroup *page_memcg_rcu (struct page *page) { -WARN_ON_ONCE (!rcu_read_lock_held ());return NULL ; }
Signature · 1 line
2.31.1.751.gd2f1c929bd-goog
PATCH v3 2/14 include/linux/nodemask.h: define next_memory_node() if !CONFIG_NUMA
In reply to Yu ZhaoCurrently next_memory_node only exists when CONFIG_NUMA=y. This patch adds the macro for !CONFIG_NUMA.
--- include/linux/nodemask.h | 1 + 1 file changed, 1 insertion(+)
include/linux/nodemask.h +1 -0
@@ -486,6 +486,7 @@ static inline int num_node_state(enum node_states state)#define first_online_node 0#define first_memory_node 0#define next_online_node (nid) (MAX_NUMNODES) +#define next_memory_node (nid) (MAX_NUMNODES)#define nr_node_ids 1U#define nr_online_nodes 1U
Signature · 1 line
2.31.1.751.gd2f1c929bd-goog
PATCH v3 3/14 include/linux/cgroup.h: export cgroup_mutex
In reply to Yu Zhaocgroup_mutex is needed to synchronize with memcg creations.
--- include/linux/cgroup.h | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)
include/linux/cgroup.h +14 -1
@@ -432,6 +432,18 @@ static inline void cgroup_put(struct cgroup *cgrp)css_put (&cgrp->self); } +extern struct mutex cgroup_mutex; + +static inline void cgroup_lock (void ) +{ +mutex_lock (&cgroup_mutex); +} + +static inline void cgroup_unlock (void ) +{ +mutex_unlock (&cgroup_mutex); +} +/** * task_css_set_check - obtain a task's css_set with extra access conditions * @task: the task to obtain css_set for
@@ -446,7 +458,6 @@ static inline void cgroup_put(struct cgroup *cgrp)* as locks used during the cgroup_subsys::attach() methods. */ #ifdef CONFIG_PROVE_RCU -extern struct mutex cgroup_mutex;extern spinlock_t css_set_lock;#define task_css_set_check (task, __c) \ rcu_dereference_check((task)->cgroups, \
@@ -704,6 +715,8 @@ struct cgroup;static inline u64 cgroup_id (const struct cgroup *cgrp) {return 1 ; }static inline void css_get (struct cgroup_subsys_state *css) {}static inline void css_put (struct cgroup_subsys_state *css) {} +static inline void cgroup_lock (void ) {} +static inline void cgroup_unlock (void ) {}static inline int cgroup_attach_task_all (struct task_struct *from,struct task_struct *t) {return 0 ; } static inline int cgroupstats_build(struct cgroupstats *stats,
Signature · 1 line
2.31.1.751.gd2f1c929bd-goog
PATCH v3 4/14 mm, x86: support the access bit on non-leaf PMD entries
In reply to Yu ZhaoSome architectures support the accessed bit on non-leaf PMD entries (parents) in addition to leaf PTE entries (children) where pages are mapped, e.g., x86_64 sets the accessed bit on a parent when using it as part of linear-address translation [1]. Page table walkers who are interested in the accessed bit on children can take advantage of this:
they do not need to search the children when the accessed bit is not set on a parent, given that they have previously cleared the accessed bit on this parent.
[1]: Intel 64 and IA-32 Architectures Software Developer's Manual
Volume 3 (October 2019), section 4.8
--- arch/Kconfig | 9 +++++++++ arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 2 +- arch/x86/mm/pgtable.c | 5 ++++- include/linux/pgtable.h | 4 ++-- 5 files changed, 17 insertions(+), 4 deletions(-)
arch/Kconfig +9 -0
@@ -826,6 +826,15 @@ config HAVE_ARCH_TRANSPARENT_HUGEPAGE config HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD bool +config HAVE_ARCH_PARENT_PMD_YOUNG + bool + depends on PGTABLE_LEVELS > 2 + help + Architectures that select this are able to set the accessed bit on + non-leaf PMD entries in addition to leaf PTE entries where pages are + mapped. For them, page table walkers that clear the accessed bit may + stop at non-leaf PMD entries if they do not see the accessed bit. + config HAVE_ARCH_HUGE_VMAP bool
arch/x86/Kconfig +1 -0
@@ -170,6 +170,7 @@ config X86 select HAVE_ARCH_TRACEHOOK select HAVE_ARCH_TRANSPARENT_HUGEPAGE select HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD if X86_64 + select HAVE_ARCH_PARENT_PMD_YOUNG if X86_64 select HAVE_ARCH_USERFAULTFD_WP if X86_64 && USERFAULTFD select HAVE_ARCH_USERFAULTFD_MINOR if X86_64 && USERFAULTFD select HAVE_ARCH_VMAP_STACK if X86_64
arch/x86/include/asm/pgtable.h +1 -1
@@ -846,7 +846,7 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)static inline int pmd_bad (pmd_t pmd) { -return (pmd_flags (pmd) & ~_PAGE_USER) != _KERNPG_TABLE; +return ((pmd_flags (pmd) | _PAGE_ACCESSED) & ~_PAGE_USER) != _KERNPG_TABLE; }static inline unsigned long pages_to_mb (unsigned long npg)
arch/x86/mm/pgtable.c +4 -1
@@ -550,7 +550,7 @@ int ptep_test_and_clear_young(struct vm_area_struct *vma,return ret; } -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#if defined(CONFIG_TRANSPARENT_HUGEPAGE ) || defined(CONFIG_HAVE_ARCH_PARENT_PMD_YOUNG )int pmdp_test_and_clear_young (struct vm_area_struct *vma,unsigned long addr,pmd_t *pmdp) {
@@ -562,6 +562,9 @@ int pmdp_test_and_clear_young(struct vm_area_struct *vma,return ret; } +#endif + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE int pudp_test_and_clear_young (struct vm_area_struct *vma,unsigned long addr,pud_t *pudp) {
include/linux/pgtable.h +2 -2
@@ -193,7 +193,7 @@ static inline int ptep_test_and_clear_young(struct vm_area_struct *vma,#endif #ifndef __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#if defined(CONFIG_TRANSPARENT_HUGEPAGE ) || defined(CONFIG_HAVE_ARCH_PARENT_PMD_YOUNG )static inline int pmdp_test_and_clear_young (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp)
@@ -214,7 +214,7 @@ static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma,BUILD_BUG ();return 0 ; } -#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +#endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_HAVE_ARCH_PARENT_PMD_YOUNG */ #endif #ifndef __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH
Signature · 1 line
2.31.1.751.gd2f1c929bd-goog
PATCH v3 5/14 mm/vmscan.c: refactor shrink_node()
In reply to Yu ZhaoHeuristics that determine scan balance between anon and file LRUs are rather independent. Move them into a separate function to improve readability.
--- mm/vmscan.c | 186 +++++++++++++++++++++++++++------------------------- 1 file changed, 98 insertions(+), 88 deletions(-)
mm/vmscan.c +98 -88
@@ -2421,6 +2421,103 @@ enum scan_balance { SCAN_FILE, }; +static void prepare_scan_count (pg_data_t *pgdat,struct scan_control *sc) +{ +unsigned long file; +struct lruvec *target_lruvec; + + target_lruvec =mem_cgroup_lruvec (sc->target_mem_cgroup, pgdat); + +/* +* Determine the scan balance between anon and file LRUs. +*/ +spin_lock_irq (&target_lruvec->lru_lock); + sc->anon_cost = target_lruvec->anon_cost; + sc->file_cost = target_lruvec->file_cost; +spin_unlock_irq (&target_lruvec->lru_lock); + +/* +* Target desirable inactive:active list ratios for the anon +* and file LRU lists. +*/ +if (!sc->force_deactivate) { +unsigned long refaults; + + refaults =lruvec_page_state (target_lruvec, +WORKINGSET_ACTIVATE_ANON ); +if (refaults != target_lruvec->refaults[0 ] || +inactive_is_low (target_lruvec,LRU_INACTIVE_ANON )) + sc->may_deactivate |=DEACTIVATE_ANON ; +else + sc->may_deactivate &= ~DEACTIVATE_ANON ; + +/* +* When refaults are being observed, it means a new +* workingset is being established. Deactivate to get +* rid of any stale active pages quickly. +*/ + refaults =lruvec_page_state (target_lruvec, +WORKINGSET_ACTIVATE_FILE ); +if (refaults != target_lruvec->refaults[1 ] || +inactive_is_low (target_lruvec,LRU_INACTIVE_FILE )) + sc->may_deactivate |=DEACTIVATE_FILE ; +else + sc->may_deactivate &= ~DEACTIVATE_FILE ; + }else + sc->may_deactivate =DEACTIVATE_ANON |DEACTIVATE_FILE ; + +/* +* If we have plenty of inactive file pages that aren't +* thrashing, try to reclaim those first before touching +* anonymous pages. +*/ + file =lruvec_page_state (target_lruvec,NR_INACTIVE_FILE ); +if (file >> sc->priority && !(sc->may_deactivate &DEACTIVATE_FILE )) + sc->cache_trim_mode =1 ; +else + sc->cache_trim_mode =0 ; + +/* +* Prevent the reclaimer from falling into the cache trap: as +* cache pages start out inactive, every cache fault will tip +* the scan balance towards the file LRU. And as the file LRU +* shrinks, so does the window for rotation from references. +* This means we have a runaway feedback loop where a tiny +* thrashing file LRU becomes infinitely more attractive than +* anon pages. Try to detect this based on file LRU size. +*/ +if (!cgroup_reclaim (sc)) { +unsigned long total_high_wmark =0 ; +unsigned long free, anon; +int z; + + free =sum_zone_node_page_state (pgdat->node_id,NR_FREE_PAGES ); + file =node_page_state (pgdat,NR_ACTIVE_FILE ) + +node_page_state (pgdat,NR_INACTIVE_FILE ); + +for (z =0 ; z <MAX_NR_ZONES ; z++) { +struct zone *zone = &pgdat->node_zones[z]; + +if (!managed_zone (zone)) +continue ; + + total_high_wmark +=high_wmark_pages (zone); + } + +/* +* Consider anon: if that's low too, this isn't a +* runaway file reclaim problem, but rather just +* extreme pressure. Reclaim as per usual then. +*/ + anon =node_page_state (pgdat,NR_INACTIVE_ANON ); + + sc->file_is_tiny = + file + free <= total_high_wmark && + !(sc->may_deactivate &DEACTIVATE_ANON ) && + anon >> sc->priority; + } +} +/* * Determine how aggressively the anon and file LRU lists should be * scanned. The relative value of each set of LRU lists is determined
@@ -2866,7 +2963,6 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc)unsigned long nr_reclaimed, nr_scanned;struct lruvec *target_lruvec;bool reclaimable = false; -unsigned long file; target_lruvec =mem_cgroup_lruvec (sc->target_mem_cgroup, pgdat);
@@ -2876,93 +2972,7 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc) nr_reclaimed = sc->nr_reclaimed; nr_scanned = sc->nr_scanned; -/* -* Determine the scan balance between anon and file LRUs. -*/ -spin_lock_irq (&target_lruvec->lru_lock); - sc->anon_cost = target_lruvec->anon_cost; - sc->file_cost = target_lruvec->file_cost; -spin_unlock_irq (&target_lruvec->lru_lock); - -/* -* Target desirable inactive:active list ratios for the anon -* and file LRU lists. -*/ -if (!sc->force_deactivate) { -unsigned long refaults; - - refaults =lruvec_page_state (target_lruvec, -WORKINGSET_ACTIVATE_ANON ); -if (refaults != target_lruvec->refaults[0 ] || -inactive_is_low (target_lruvec,LRU_INACTIVE_ANON )) - sc->may_deactivate |=DEACTIVATE_ANON ; -else - sc->may_deactivate &= ~DEACTIVATE_ANON ; - -/* -* When refaults are being observed, it means a new -* workingset is being established. Deactivate to get -* rid of any stale active pages quickly. -*/ - refaults =lruvec_page_state (target_lruvec, -WORKINGSET_ACTIVATE_FILE ); -if (refaults != target_lruvec->refaults[1 ] || -inactive_is_low (target_lruvec,LRU_INACTIVE_FILE )) - sc->may_deactivate |=DEACTIVATE_FILE ; -else - sc->may_deactivate &= ~DEACTIVATE_FILE ; - }else - sc->may_deactivate =DEACTIVATE_ANON |DEACTIVATE_FILE ; - -/* -* If we have plenty of inactive file pages that aren't -* thrashing, try to reclaim those first before touching -* anonymous pages. -*/ - file =lruvec_page_state (target_lruvec,NR_INACTIVE_FILE ); -if (file >> sc->priority && !(sc->may_deactivate &DEACTIVATE_FILE )) - sc->cache_trim_mode =1 ; -else - sc->cache_trim_mode =0 ; - -/* -* Prevent the reclaimer from falling into the cache trap: as -* cache pages start out inactive, every cache fault will tip -* the scan balance towards the file LRU. And as the file LRU -* shrinks, so does the window for rotation from references. -* This means we have a runaway feedback loop where a tiny -* thrashing file LRU becomes infinitely more attractive than -* anon pages. Try to detect this based on file LRU size. -*/ -if (!cgroup_reclaim (sc)) { -unsigned long total_high_wmark =0 ; -unsigned long free, anon; -int z; - - free =sum_zone_node_page_state (pgdat->node_id,NR_FREE_PAGES ); - file =node_page_state (pgdat,NR_ACTIVE_FILE ) + -node_page_state (pgdat,NR_INACTIVE_FILE ); - -for (z =0 ; z <MAX_NR_ZONES ; z++) { -struct zone *zone = &pgdat->node_zones[z]; -if (!managed_zone (zone)) -continue ; - - total_high_wmark +=high_wmark_pages (zone); - } - -/* -* Consider anon: if that's low too, this isn't a -* runaway file reclaim problem, but rather just -* extreme pressure. Reclaim as per usual then. -*/ - anon =node_page_state (pgdat,NR_INACTIVE_ANON ); - - sc->file_is_tiny = - file + free <= total_high_wmark && - !(sc->may_deactivate &DEACTIVATE_ANON ) && - anon >> sc->priority; - } +prepare_scan_count (pgdat, sc);shrink_node_memcgs (pgdat, sc);
Signature · 1 line
2.31.1.751.gd2f1c929bd-goog
PATCH v3 6/14 mm/workingset.c: refactor pack_shadow() and unpack_shadow()
In reply to Yu ZhaoThis patches moves the bucket order and PageWorkingset() out of pack_shadow() and unpack_shadow(). It has no merits on its own but makes the upcoming changes to mm/workingset.c less diffy.
--- mm/workingset.c | 53 ++++++++++++++++++++----------------------------- 1 file changed, 22 insertions(+), 31 deletions(-)
mm/workingset.c +22 -31
@@ -168,9 +168,9 @@* refault distance will immediately activate the refaulting page. */ -#define EVICTION_SHIFT ((BITS_PER_LONG - BITS_PER_XA_VALUE) + \ - 1 + NODES_SHIFT + MEM_CGROUP_ID_SHIFT) -#define EVICTION_MASK (~0UL >> EVICTION_SHIFT) +#define EVICTION_SHIFT (BITS_PER_XA_VALUE - MEM_CGROUP_ID_SHIFT - NODES_SHIFT) +#define EVICTION_MASK (BIT(EVICTION_SHIFT) - 1) +#define WORKINGSET_WIDTH 1/* * Eviction timestamps need to be able to cover the full range of
@@ -182,36 +182,23 @@*/ static unsigned int bucket_order__read_mostly ; -static void *pack_shadow (int memcgid,pg_data_t *pgdat,unsigned long eviction, -bool workingset) +static void *pack_shadow (int memcg_id,struct pglist_data *pgdat,unsigned long val) { - eviction >>= bucket_order; - eviction &=EVICTION_MASK ; - eviction = (eviction <<MEM_CGROUP_ID_SHIFT ) | memcgid; - eviction = (eviction <<NODES_SHIFT ) | pgdat->node_id; - eviction = (eviction <<1 ) | workingset; + val = (val <<MEM_CGROUP_ID_SHIFT ) | memcg_id; + val = (val <<NODES_SHIFT ) | pgdat->node_id; -return xa_mk_value (eviction); +return xa_mk_value (val); } -static void unpack_shadow (void *shadow,int *memcgidp,pg_data_t **pgdat, -unsigned long *evictionp,bool *workingsetp) +static unsigned long unpack_shadow (void *shadow,int *memcg_id,struct pglist_data **pgdat) { -unsigned long entry =xa_to_value (shadow); -int memcgid, nid; -bool workingset; +unsigned long val =xa_to_value (shadow); - workingset = entry &1 ; - entry >>=1 ; - nid = entry & ((1UL <<NODES_SHIFT ) -1 ); - entry >>=NODES_SHIFT ; - memcgid = entry & ((1UL <<MEM_CGROUP_ID_SHIFT ) -1 ); - entry >>=MEM_CGROUP_ID_SHIFT ; + *pgdat =NODE_DATA (val & (BIT (NODES_SHIFT ) -1 )); + val >>=NODES_SHIFT ; + *memcg_id = val & (BIT (MEM_CGROUP_ID_SHIFT ) -1 ); - *memcgidp = memcgid; - *pgdat =NODE_DATA (nid); - *evictionp = entry << bucket_order; - *workingsetp = workingset; +return val >>MEM_CGROUP_ID_SHIFT ; }/**
@@ -266,8 +253,10 @@ void *workingset_eviction(struct page *page, struct mem_cgroup *target_memcg)/* XXX: target_memcg can be NULL, go through lruvec */ memcgid =mem_cgroup_id (lruvec_memcg (lruvec)); eviction =atomic_long_read (&lruvec->nonresident_age); + eviction >>= bucket_order; + eviction = (eviction <<WORKINGSET_WIDTH ) |PageWorkingset (page);workingset_age_nonresident (lruvec,thp_nr_pages (page)); -return pack_shadow (memcgid, pgdat, eviction,PageWorkingset (page)); +return pack_shadow (memcgid, pgdat, eviction); }/**
@@ -294,7 +283,7 @@ void workingset_refault(struct page *page, void *shadow)bool workingset;int memcgid; -unpack_shadow (shadow, &memcgid, &pgdat, &eviction, &workingset); + eviction =unpack_shadow (shadow, &memcgid, &pgdat);rcu_read_lock ();/*
@@ -318,6 +307,8 @@ void workingset_refault(struct page *page, void *shadow) goto out; eviction_lruvec =mem_cgroup_lruvec (eviction_memcg, pgdat); refault =atomic_long_read (&eviction_lruvec->nonresident_age); + workingset = eviction & (BIT (WORKINGSET_WIDTH ) -1 ); + eviction = (eviction >>WORKINGSET_WIDTH ) << bucket_order;/* * Calculate the refault distance
@@ -335,7 +326,7 @@ void workingset_refault(struct page *page, void *shadow)* longest time, so the occasional inappropriate activation * leading to pressure on the active list is not a problem. */ - refault_distance = (refault - eviction) &EVICTION_MASK ; + refault_distance = (refault - eviction) & (EVICTION_MASK >>WORKINGSET_WIDTH );/* * The activation decision for this page is made at the level
@@ -593,7 +584,7 @@ static int __init workingset_init(void)unsigned int max_order;int ret; -BUILD_BUG_ON (BITS_PER_LONG <EVICTION_SHIFT ); +BUILD_BUG_ON (EVICTION_SHIFT <WORKINGSET_WIDTH );/* * Calculate the eviction bucket size to cover the longest * actionable refault distance, which is currently half of
@@ -601,7 +592,7 @@ static int __init workingset_init(void)* some more pages@runtime, so keep working with up to * double the initial memory by using totalram_pages as-is. */ - timestamp_bits =BITS_PER_LONG -EVICTION_SHIFT ; + timestamp_bits =EVICTION_SHIFT -WORKINGSET_WIDTH ; max_order =fls_long (totalram_pages () -1 );if (max_order > timestamp_bits) bucket_order = max_order - timestamp_bits;
Signature · 1 line
2.31.1.751.gd2f1c929bd-goog
PATCH v3 7/14 mm: multigenerational lru: groundwork
In reply to Yu ZhaoFor each lruvec, evictable pages are divided into multiple generations. The youngest generation number is stored in lrugen->max_seq for both anon and file types as they are aged on an equal footing. The oldest generation numbers are stored in lrugen->min_seq[2] separately for anon and file types as clean file pages can be evicted regardless of may_swap or may_writepage. These three variables are monotonically increasing. Generation numbers are truncated into order_base_2(MAX_NR_GENS+1) bits in order to fit into page->flags. The sliding window technique is used to prevent truncated generation numbers from overlapping. Each truncated generation number is an index to lrugen->lists[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES]. Evictable pages are added to the per-zone lists indexed by lrugen->max_seq or lrugen->min_seq[2] (modulo MAX_NR_GENS), depending on their types.
Each generation is then divided into multiple tiers. Tiers represent levels of usage from file descriptors only. Pages accessed N times via file descriptors belong to tier order_base_2(N). Each generation contains at most MAX_NR_TIERS tiers, and they require additional MAX_NR_TIERS-2 bits in page->flags. In contrast to moving across generations which requires the lru lock for the list operations, moving across tiers only involves an atomic operation on page->flags and therefore has a negligible cost. A feedback loop modeled after the PID controller monitors the refault rates across all tiers and decides when to activate pages from which tiers in the reclaim path.
The framework comprises two conceptually independent components: the aging and the eviction, which can be invoked separately from user space for the purpose of working set estimation and proactive reclaim.
The aging produces young generations. Given an lruvec, the aging scans page tables for referenced pages of this lruvec. Upon finding one, the aging updates its generation number to max_seq. After each round of scan, the aging increments max_seq. The aging is due when both of min_seq[2] reaches max_seq-1, assuming both anon and file types are reclaimable.
The eviction consumes old generations. Given an lruvec, the eviction scans the pages on the per-zone lists indexed by either of min_seq[2]. It tries to select a type based on the values of min_seq[2] and swappiness. During a scan, the eviction sorts pages according to their new generation numbers, if the aging has found them referenced. When it finds all the per-zone lists of a selected type are empty, the eviction increments min_seq[2] indexed by this selected type.
--- fs/fuse/dev.c | 3 +- include/linux/mm.h | 2 + include/linux/mm_inline.h | 194 +++++++++++++++++++ include/linux/mmzone.h | 106 +++++++++++ include/linux/page-flags-layout.h | 19 +- include/linux/page-flags.h | 4 +- kernel/bounds.c | 6 + mm/huge_memory.c | 3 +- mm/mm_init.c | 6 +- mm/mmzone.c | 2 + mm/swapfile.c | 4 + mm/vmscan.c | 306 ++++++++++++++++++++++++++++++ 12 files changed, 644 insertions(+), 11 deletions(-)
fs/fuse/dev.c +2 -1
@@ -784,7 +784,8 @@ static int fuse_check_page(struct page *page) 1 << PG_lru | 1 << PG_active | 1 << PG_reclaim | - 1 << PG_waiters))) { + 1 << PG_waiters | + LRU_GEN_MASK | LRU_USAGE_MASK))) {dump_page (page,"fuse: trying to steal weird page" );return 1 ; }
include/linux/mm.h +2 -0
@@ -1089,6 +1089,8 @@ vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf);#define ZONES_PGOFF (NODES_PGOFF - ZONES_WIDTH)#define LAST_CPUPID_PGOFF (ZONES_PGOFF - LAST_CPUPID_WIDTH)#define KASAN_TAG_PGOFF (LAST_CPUPID_PGOFF - KASAN_TAG_WIDTH) +#define LRU_GEN_PGOFF (KASAN_TAG_PGOFF - LRU_GEN_WIDTH) +#define LRU_USAGE_PGOFF (LRU_GEN_PGOFF - LRU_USAGE_WIDTH)/* * Define the bit shifts to access each section. For non-existent
include/linux/mm_inline.h +194 -0
217 lines not on this page · raw message
include/linux/mmzone.h +106 -0
118 lines not on this page · raw message
include/linux/page-flags-layout.h +14 -5
@@ -26,6 +26,14 @@#define ZONES_WIDTH ZONES_SHIFT +#ifdef CONFIG_LRU_GEN +/* LRU_GEN_WIDTH is generated from order_base_2(CONFIG_NR_LRU_GENS + 1). */ +#define LRU_USAGE_WIDTH (CONFIG_TIERS_PER_GEN - 2) +#else +#define LRU_GEN_WIDTH 0 +#define LRU_USAGE_WIDTH 0 +#endif +#ifdef CONFIG_SPARSEMEM #include <asm/sparsemem.h> #define SECTIONS_SHIFT (MAX_PHYSMEM_BITS - SECTION_SIZE_BITS)
@@ -55,7 +63,8 @@#define SECTIONS_WIDTH 0#endif -#if ZONES_WIDTH +SECTIONS_WIDTH +NODES_SHIFT <=BITS_PER_LONG -NR_PAGEFLAGS +#if ZONES_WIDTH +LRU_GEN_WIDTH +LRU_USAGE_WIDTH +SECTIONS_WIDTH +NODES_SHIFT \ + <=BITS_PER_LONG -NR_PAGEFLAGS #define NODES_WIDTH NODES_SHIFT#elif defined(CONFIG_SPARSEMEM_VMEMMAP )#error "Vmemmap: No space for nodes field in page flags"
@@ -89,8 +98,8 @@#define LAST_CPUPID_SHIFT 0#endif -#if ZONES_WIDTH +SECTIONS_WIDTH +NODES_WIDTH +KASAN_TAG_WIDTH +LAST_CPUPID_SHIFT \ - <=BITS_PER_LONG -NR_PAGEFLAGS +#if ZONES_WIDTH +LRU_GEN_WIDTH +LRU_USAGE_WIDTH +SECTIONS_WIDTH +NODES_WIDTH + \ +KASAN_TAG_WIDTH +LAST_CPUPID_SHIFT <=BITS_PER_LONG -NR_PAGEFLAGS #define LAST_CPUPID_WIDTH LAST_CPUPID_SHIFT#else #define LAST_CPUPID_WIDTH 0
@@ -100,8 +109,8 @@#define LAST_CPUPID_NOT_IN_PAGE_FLAGS #endif -#if ZONES_WIDTH +SECTIONS_WIDTH +NODES_WIDTH +KASAN_TAG_WIDTH +LAST_CPUPID_WIDTH \ - >BITS_PER_LONG -NR_PAGEFLAGS +#if ZONES_WIDTH +LRU_GEN_WIDTH +LRU_USAGE_WIDTH +SECTIONS_WIDTH +NODES_WIDTH + \ +KASAN_TAG_WIDTH +LAST_CPUPID_WIDTH >BITS_PER_LONG -NR_PAGEFLAGS #error "Not enough bits in page flags"#endif
include/linux/page-flags.h +2 -2
@@ -817,7 +817,7 @@ static inline void ClearPageSlabPfmemalloc(struct page *page)1UL << PG_private |1UL << PG_private_2 | \1UL << PG_writeback |1UL << PG_reserved | \1UL << PG_slab |1UL << PG_active | \ - 1UL << PG_unevictable | __PG_MLOCKED) + 1UL << PG_unevictable | __PG_MLOCKED | LRU_GEN_MASK)/* * Flags checked when a page is prepped for return by the page allocator.
@@ -828,7 +828,7 @@ static inline void ClearPageSlabPfmemalloc(struct page *page)* alloc-free cycle to prevent from reusing the page. */ #define PAGE_FLAGS_CHECK_AT_PREP \ - (((1UL << NR_PAGEFLAGS) - 1) & ~__PG_HWPOISON) + ((((1UL << NR_PAGEFLAGS) - 1) & ~__PG_HWPOISON) | LRU_GEN_MASK | LRU_USAGE_MASK)#define PAGE_FLAGS_PRIVATE \ (1UL << PG_private | 1UL << PG_private_2)
kernel/bounds.c +6 -0
@@ -22,6 +22,12 @@ int main(void)DEFINE (NR_CPUS_BITS ,ilog2 (CONFIG_NR_CPUS ));#endif DEFINE (SPINLOCK_SIZE ,sizeof (spinlock_t)); +#ifdef CONFIG_LRU_GEN +/* bits needed to represent internal values stored in page->flags */ +DEFINE (LRU_GEN_WIDTH ,order_base_2 (CONFIG_NR_LRU_GENS +1 )); +/* bits needed to represent normalized values for external uses */ +DEFINE (LRU_GEN_SHIFT ,order_base_2 (CONFIG_NR_LRU_GENS )); +#endif /* End of constants */ return 0 ;
mm/huge_memory.c +2 -1
@@ -2410,7 +2410,8 @@ static void __split_huge_page_tail(struct page *head, int tail,#ifdef CONFIG_64BIT (1L << PG_arch_2) | #endif - (1L << PG_dirty))); + (1L << PG_dirty) | + LRU_GEN_MASK | LRU_USAGE_MASK));/* ->mapping in first tail page is compound_mapcount */ VM_BUG_ON_PAGE(tail > 2 && page_tail->mapping != TAIL_MAPPING,
mm/mm_init.c +4 -2
@@ -65,14 +65,16 @@ void __init mminit_verify_pageflags_layout(void) shift =8 *sizeof (unsigned long ); width = shift -SECTIONS_WIDTH -NODES_WIDTH -ZONES_WIDTH - -LAST_CPUPID_SHIFT -KASAN_TAG_WIDTH ; + -LAST_CPUPID_SHIFT -KASAN_TAG_WIDTH -LRU_GEN_WIDTH -LRU_USAGE_WIDTH ;mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_widths" , -"Section %d Node %d Zone %d Lastcpupid %d Kasantag %d Flags %d\n" , +"Section %d Node %d Zone %d Lastcpupid %d Kasantag %d Gen %d Tier %d Flags %d\n" ,SECTIONS_WIDTH ,NODES_WIDTH ,ZONES_WIDTH ,LAST_CPUPID_WIDTH ,KASAN_TAG_WIDTH , +LRU_GEN_WIDTH , +LRU_USAGE_WIDTH ,NR_PAGEFLAGS ); mminit_dprintk(MMINIT_TRACE, "pageflags_layout_shifts", "Section %d Node %d Zone %d Lastcpupid %d Kasantag %d\n",
mm/mmzone.c +2 -0
@@ -81,6 +81,8 @@ void lruvec_init(struct lruvec *lruvec)for_each_lru (lru)INIT_LIST_HEAD (&lruvec->lists[lru]); + +lru_gen_init_lruvec (lruvec); }#if defined(CONFIG_NUMA_BALANCING ) && !defined(LAST_CPUPID_NOT_IN_PAGE_FLAGS )
mm/swapfile.c +4 -0
@@ -2702,6 +2702,8 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) err =0 ;atomic_inc (&proc_poll_event);wake_up_interruptible (&proc_poll_wait); +/* stop tracking anon if the multigenerational lru is turned off */ +lru_gen_set_state (false, false, true); out_dput:filp_close (victim,NULL );
@@ -3348,6 +3350,8 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)mutex_unlock (&swapon_mutex);atomic_inc (&proc_poll_event);wake_up_interruptible (&proc_poll_wait); +/* start tracking anon if the multigenerational lru is turned on */ +lru_gen_set_state (true, false, true); error =0 ; goto out;
mm/vmscan.c +306 -0
318 lines not on this page · raw message
Signature · 1 line
2.31.1.751.gd2f1c929bd-goog
PATCH v3 8/14 mm: multigenerational lru: activation
In reply to Yu ZhaoFor pages accessed multiple times via file descriptors, instead of activating them upon the second access, we activate them based on the refault rates of their tiers. Each generation contains at most MAX_NR_TIERS tiers, and they require additional MAX_NR_TIERS-2 bits in page->flags. Pages accessed N times via file descriptors belong to tier order_base_2(N). Tier 0 is the base tier and it contains pages read ahead, accessed once via file descriptors and accessed only via page tables. Pages from the base tier are evicted regardless of the refault rate. Pages from upper tiers that have higher refault rates than the base tier will be moved to the next generation. A feedback loop modeled after the PID controller monitors refault rates across all tiers and decides when to activate pages from which upper tiers in the reclaim path. The advantages of this model are:
1) It has a negligible cost in the buffered IO access path because activations are done optionally in the reclaim path.
2) It takes mapped pages into account and avoids overprotecting pages accessed multiple times via file descriptors.
3) More tiers offer better protection to pages accessed more than twice when workloads doing intensive buffered IO are under memory pressure.
For pages mapped upon page faults, the accessed bit is set during the initial faults. Ideally we add them to the per-zone lists index by max_seq, i.e., the youngest generation, so that eviction will not consider them before the aging has scanned them. For anon pages not in swap cache, this can be done easily in the page fault path: we rename lru_cache_add_inactive_or_unevictable() to lru_cache_add_page_vma() and add a new parameter, which is set to true for pages mapped upon page faults. For pages in page cache or swap cache, we cannot differentiate the page fault path from the read ahead path at the time we call lru_cache_add(). So we add them to the per-zone lists index by min_seq, i.e., the oldest generation, for now.
Finally, we need to make sure deactivation works when the multigenerational lru is enabled. We cannot use PageActive() because it is not set on pages from active generations, in order to spare the aging the trouble of clearing it when active generations become inactive. So we deactivate pages unconditionally since deactivation is not a hot code path worth additional optimizations.
--- include/linux/mm_inline.h | 40 ++++++++++++++ include/linux/swap.h | 4 +- kernel/events/uprobes.c | 2 +- mm/huge_memory.c | 2 +- mm/khugepaged.c | 2 +- mm/memory.c | 10 ++-- mm/migrate.c | 2 +- mm/swap.c | 22 +++++--- mm/swapfile.c | 2 +- mm/userfaultfd.c | 2 +- mm/vmscan.c | 91 ++++++++++++++++++++++++++++++- mm/workingset.c | 112 ++++++++++++++++++++++++++++++++++++++ 12 files changed, 269 insertions(+), 22 deletions(-)
include/linux/mm_inline.h +40 -0
@@ -103,6 +103,12 @@ static inline int lru_gen_from_seq(unsigned long seq)return seq %MAX_NR_GENS ; } +/* Convert the level of usage to a tier. See the comment on MAX_NR_TIERS. */ +static inline int lru_tier_from_usage (int usage) +{ +return order_base_2 (usage +1 ); +} +/* Return a proper index regardless whether we keep a full history of stats. */ static inline int hist_from_seq_or_gen (int seq_or_gen) {
@@ -245,6 +251,36 @@ static inline bool lru_gen_deletion(struct page *page, struct lruvec *lruvec) return true; } +/* Return the level of usage of a page. See the comment on MAX_NR_TIERS. */ +static inline int page_tier_usage (struct page *page) +{ +unsigned long flags =READ_ONCE (page->flags); + +return flags &BIT (PG_workingset) ? + ((flags &LRU_USAGE_MASK ) >>LRU_USAGE_PGOFF ) +1 :0 ; +} + +/* Increment the usage counter after a page is accessed via file descriptors. */ +static inline void page_inc_usage (struct page *page) +{ +unsigned long usage; +unsigned long old_flags, new_flags; + +do { + old_flags =READ_ONCE (page->flags); + +if (!(old_flags &BIT (PG_workingset))) { + new_flags = old_flags |BIT (PG_workingset); +continue ; + } + + usage = (old_flags &LRU_USAGE_MASK ) +BIT (LRU_USAGE_PGOFF ); + + new_flags = (old_flags & ~LRU_USAGE_MASK ) |min (usage,LRU_USAGE_MASK ); + }while (new_flags != old_flags && +cmpxchg (&page->flags, old_flags, new_flags) != old_flags); +} +#else /* CONFIG_LRU_GEN */ static inline bool lru_gen_enabled (void )
@@ -262,6 +298,10 @@ static inline bool lru_gen_deletion(struct page *page, struct lruvec *lruvec)return false; } +static inline void page_inc_usage (struct page *page) +{ +} +#endif /* CONFIG_LRU_GEN */ static __always_inline void add_page_to_lru_list(struct page *page,
include/linux/swap.h +2 -2
@@ -365,8 +365,8 @@ extern void deactivate_page(struct page *page);extern void mark_page_lazyfree (struct page *page);extern void swap_setup (void ); -extern void lru_cache_add_inactive_or_unevictable (struct page *page, -struct vm_area_struct *vma); +extern void lru_cache_add_page_vma (struct page *page,struct vm_area_struct *vma, +bool faulting);/* linux/mm/vmscan.c */ extern unsigned long zone_reclaimable_pages (struct zone *zone);
kernel/events/uprobes.c +1 -1
@@ -184,7 +184,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr,if (new_page) {get_page (new_page);page_add_new_anon_rmap (new_page, vma, addr, false); -lru_cache_add_inactive_or_unevictable (new_page, vma); +lru_cache_add_page_vma (new_page, vma, false); }else /* no new page, just dec_mm_counter for old_page */ dec_mm_counter (mm,MM_ANONPAGES );
mm/huge_memory.c +1 -1
@@ -636,7 +636,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, entry =mk_huge_pmd (page, vma->vm_page_prot); entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma);page_add_new_anon_rmap (page, vma, haddr, true); -lru_cache_add_inactive_or_unevictable (page, vma); +lru_cache_add_page_vma (page, vma, true);pgtable_trans_huge_deposit (vma->vm_mm, vmf->pmd, pgtable);set_pmd_at (vma->vm_mm, haddr, vmf->pmd, entry);update_mmu_cache_pmd (vma, vmf->address, vmf->pmd);
mm/khugepaged.c +1 -1
@@ -1198,7 +1198,7 @@ static void collapse_huge_page(struct mm_struct *mm,spin_lock (pmd_ptl);BUG_ON (!pmd_none (*pmd));page_add_new_anon_rmap (new_page, vma, address, true); -lru_cache_add_inactive_or_unevictable (new_page, vma); +lru_cache_add_page_vma (new_page, vma, true);pgtable_trans_huge_deposit (mm, pmd, pgtable);set_pmd_at (mm, address, pmd, _pmd);update_mmu_cache_pmd (vma, address, pmd);
mm/memory.c +5 -5
@@ -839,7 +839,7 @@ copy_present_page(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vmacopy_user_highpage (new_page, page, addr, src_vma);__SetPageUptodate (new_page);page_add_new_anon_rmap (new_page, dst_vma, addr, false); -lru_cache_add_inactive_or_unevictable (new_page, dst_vma); +lru_cache_add_page_vma (new_page, dst_vma, false); rss[mm_counter (new_page)]++;/* All done, just insert the new page copy in the child */
@@ -2950,7 +2950,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)*/ ptep_clear_flush_notify (vma, vmf->address, vmf->pte);page_add_new_anon_rmap (new_page, vma, vmf->address, false); -lru_cache_add_inactive_or_unevictable (new_page, vma); +lru_cache_add_page_vma (new_page, vma, true);/* * We call the notify macro here because, when using secondary * mmu page tables (such as kvm shadow page tables), we want the
@@ -3479,7 +3479,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)/* ksm created a completely new copy */ if (unlikely (page != swapcache && swapcache)) {page_add_new_anon_rmap (page, vma, vmf->address, false); -lru_cache_add_inactive_or_unevictable (page, vma); +lru_cache_add_page_vma (page, vma, true); }else {do_page_add_anon_rmap (page, vma, vmf->address, exclusive); }
@@ -3625,7 +3625,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)inc_mm_counter_fast (vma->vm_mm,MM_ANONPAGES );page_add_new_anon_rmap (page, vma, vmf->address, false); -lru_cache_add_inactive_or_unevictable (page, vma); +lru_cache_add_page_vma (page, vma, true); setpte:set_pte_at (vma->vm_mm, vmf->address, vmf->pte, entry);
@@ -3793,7 +3793,7 @@ void do_set_pte(struct vm_fault *vmf, struct page *page, unsigned long addr)if (write && !(vma->vm_flags &VM_SHARED )) {inc_mm_counter_fast (vma->vm_mm,MM_ANONPAGES );page_add_new_anon_rmap (page, vma, addr, false); -lru_cache_add_inactive_or_unevictable (page, vma); +lru_cache_add_page_vma (page, vma, true); }else {inc_mm_counter_fast (vma->vm_mm,mm_counter_file (page));page_add_file_rmap (page, false);
mm/migrate.c +1 -1
@@ -2967,7 +2967,7 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate,inc_mm_counter (mm,MM_ANONPAGES );page_add_new_anon_rmap (page, vma, addr, false);if (!is_zone_device_page (page)) -lru_cache_add_inactive_or_unevictable (page, vma); +lru_cache_add_page_vma (page, vma, false);get_page (page);if (flush) {
mm/swap.c +14 -8
@@ -433,6 +433,8 @@ void mark_page_accessed(struct page *page)* this list is never rotated or maintained, so marking an * evictable page accessed has no effect. */ + }else if (lru_gen_enabled ()) { +page_inc_usage (page); }else if (!PageActive (page)) {/* * If the page is on the LRU, queue it for activation via
@@ -478,15 +480,14 @@ void lru_cache_add(struct page *page)EXPORT_SYMBOL (lru_cache_add);/** -* lru_cache_add_inactive_or_unevictable +* lru_cache_add_page_vma * @page: the page to be added to LRU * @vma: vma in which page is mapped for determining reclaimability * -* Place @page on the inactive or unevictable LRU list, depending on its -* evictability. +* Place @page on an LRU list, depending on its evictability. */ -void lru_cache_add_inactive_or_unevictable (struct page *page, -struct vm_area_struct *vma) +void lru_cache_add_page_vma (struct page *page,struct vm_area_struct *vma, +bool faulting) {bool unevictable;
@@ -503,6 +504,11 @@ void lru_cache_add_inactive_or_unevictable(struct page *page,__mod_zone_page_state (page_zone (page),NR_MLOCK , nr_pages);count_vm_events (UNEVICTABLE_PGMLOCKED , nr_pages); } + +/* tell the multigenerational lru that the page is being faulted in */ +if (lru_gen_enabled () && !unevictable && faulting) +SetPageActive (page); +lru_cache_add (page); }
@@ -529,7 +535,7 @@ void lru_cache_add_inactive_or_unevictable(struct page *page,*/ static void lru_deactivate_file_fn (struct page *page,struct lruvec *lruvec) { -bool active =PageActive (page); +bool active =PageActive (page) ||lru_gen_enabled ();int nr_pages =thp_nr_pages (page);if (PageUnevictable (page))
@@ -569,7 +575,7 @@ static void lru_deactivate_file_fn(struct page *page, struct lruvec *lruvec)static void lru_deactivate_fn (struct page *page,struct lruvec *lruvec) { -if (PageActive (page) && !PageUnevictable (page)) { +if (!PageUnevictable (page) && (PageActive (page) ||lru_gen_enabled ())) {int nr_pages =thp_nr_pages (page);del_page_from_lru_list (page, lruvec);
@@ -684,7 +690,7 @@ void deactivate_file_page(struct page *page)*/ void deactivate_page (struct page *page) { -if (PageLRU (page) &&PageActive (page) && !PageUnevictable (page)) { +if (PageLRU (page) && !PageUnevictable (page) && (PageActive (page) ||lru_gen_enabled ())) {struct pagevec *pvec;local_lock (&lru_pvecs.lock);
mm/swapfile.c +1 -1
@@ -1936,7 +1936,7 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd,page_add_anon_rmap (page, vma, addr, false); }else {/* ksm created a completely new copy */ page_add_new_anon_rmap (page, vma, addr, false); -lru_cache_add_inactive_or_unevictable (page, vma); +lru_cache_add_page_vma (page, vma, false); }swap_free (entry); out:
mm/userfaultfd.c +1 -1
@@ -123,7 +123,7 @@ static int mcopy_atomic_pte(struct mm_struct *dst_mm,inc_mm_counter (dst_mm,MM_ANONPAGES );page_add_new_anon_rmap (page, dst_vma, dst_addr, false); -lru_cache_add_inactive_or_unevictable (page, dst_vma); +lru_cache_add_page_vma (page, dst_vma, true);set_pte_at (dst_mm, dst_addr, dst_pte, _dst_pte);
mm/vmscan.c +90 -1
105 lines not on this page · raw message
mm/workingset.c +112 -0
130 lines not on this page · raw message
Signature · 1 line
2.31.1.751.gd2f1c929bd-goog
PATCH v3 9/14 mm: multigenerational lru: mm_struct list
In reply to Yu ZhaoIn order to scan page tables, we add an infrastructure to maintain either a system-wide mm_struct list or per-memcg mm_struct lists, and track whether an mm_struct is being used or has been used since the last scan.
Multiple threads can concurrently work on the same mm_struct list, and each of them will be given a different mm_struct belonging to a process that has been scheduled since the last scan.
--- fs/exec.c | 2 + include/linux/memcontrol.h | 6 + include/linux/mm_types.h | 107 ++++++++++++ kernel/exit.c | 1 + kernel/fork.c | 10 ++ kernel/kthread.c | 1 + kernel/sched/core.c | 2 + mm/memcontrol.c | 28 ++++ mm/vmscan.c | 324 +++++++++++++++++++++++++++++++++++++ 9 files changed, 481 insertions(+)
fs/exec.c +2 -0
@@ -1008,6 +1008,7 @@ static int exec_mmap(struct mm_struct *mm) active_mm = tsk->active_mm; tsk->active_mm = mm; tsk->mm = mm; +lru_gen_add_mm (mm);/* * This prevents preemption while active_mm is being loaded and * it and mm are being updated, which could cause problems for
@@ -1018,6 +1019,7 @@ static int exec_mmap(struct mm_struct *mm)if (!IS_ENABLED (CONFIG_ARCH_WANT_IRQS_OFF_ACTIVATE_MM ))local_irq_enable ();activate_mm (active_mm, mm); +lru_gen_switch_mm (active_mm, mm);if (IS_ENABLED (CONFIG_ARCH_WANT_IRQS_OFF_ACTIVATE_MM ))local_irq_enable (); tsk->mm->vmacache_seqnum =0 ;
include/linux/memcontrol.h +6 -0
@@ -230,6 +230,8 @@ struct obj_cgroup { }; }; +struct lru_gen_mm_list ; +/* * The memory controller data structure. The memory controller controls both * page cache and RSS per cgroup. We would eventually like to provide
@@ -349,6 +351,10 @@ struct mem_cgroup {struct deferred_split deferred_split_queue;#endif +#ifdef CONFIG_LRU_GEN +struct lru_gen_mm_list *mm_list; +#endif +struct mem_cgroup_per_node *nodeinfo[0 ];/* WARNING: nodeinfo must be the last member here */ };
include/linux/mm_types.h +107 -0
125 lines not on this page · raw message
kernel/exit.c +1 -0
@@ -423,6 +423,7 @@ void mm_update_next_owner(struct mm_struct *mm) goto retry; }WRITE_ONCE (mm->owner, c); +lru_gen_migrate_mm (mm);task_unlock (c);put_task_struct (c); }
kernel/fork.c +10 -0
@@ -669,6 +669,7 @@ static void check_mm(struct mm_struct *mm)#if defined(CONFIG_TRANSPARENT_HUGEPAGE ) && !USE_SPLIT_PMD_PTLOCKS VM_BUG_ON_MM (mm->pmd_huge_pte, mm);#endif +VM_BUG_ON_MM (lru_gen_mm_is_active (mm), mm); }#define allocate_mm () (kmem_cache_alloc(mm_cachep, GFP_KERNEL))
@@ -1061,6 +1062,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, goto fail_nocontext; mm->user_ns =get_user_ns (user_ns); +lru_gen_init_mm (mm);return mm; fail_nocontext:
@@ -1103,6 +1105,7 @@ static inline void __mmput(struct mm_struct *mm) }if (mm->binfmt)module_put (mm->binfmt->module); +lru_gen_del_mm (mm);mmdrop (mm); }
@@ -2524,6 +2527,13 @@ pid_t kernel_clone(struct kernel_clone_args *args)get_task_struct (p); } +if (IS_ENABLED (CONFIG_LRU_GEN ) && !(clone_flags &CLONE_VM )) { +/* lock the task to synchronize with memcg migration */ +task_lock (p); +lru_gen_add_mm (p->mm); +task_unlock (p); + } +wake_up_new_task (p);/* forking complete and child started to run, tell ptracer */
kernel/kthread.c +1 -0
@@ -1325,6 +1325,7 @@ void kthread_use_mm(struct mm_struct *mm) tsk->mm = mm;membarrier_update_current_mm (mm);switch_mm_irqs_off (active_mm, mm, tsk); +lru_gen_switch_mm (active_mm, mm);local_irq_enable ();task_unlock (tsk);#ifdef finish_arch_post_lock_switch
kernel/sched/core.c +2 -0
@@ -4323,6 +4323,7 @@ context_switch(struct rq *rq, struct task_struct *prev,* finish_task_switch()'s mmdrop(). */ switch_mm_irqs_off (prev->active_mm, next->mm, next); +lru_gen_switch_mm (prev->active_mm, next->mm);if (!prev->mm) {// from kernel /* will mmdrop() in finish_task_switch(). */
@@ -7603,6 +7604,7 @@ void idle_task_exit(void)if (mm != &init_mm) {switch_mm (mm, &init_mm, current); +lru_gen_switch_mm (mm, &init_mm);finish_arch_post_lock_switch (); }
mm/memcontrol.c +28 -0
@@ -4981,6 +4981,7 @@ static void __mem_cgroup_free(struct mem_cgroup *memcg)for_each_node (node)free_mem_cgroup_per_node_info (memcg, node);free_percpu (memcg->vmstats_percpu); +lru_gen_free_mm_list (memcg);kfree (memcg); }
@@ -5030,6 +5031,9 @@ static struct mem_cgroup *mem_cgroup_alloc(void)if (alloc_mem_cgroup_per_node_info (memcg, node)) goto fail; +if (lru_gen_alloc_mm_list (memcg)) + goto fail; +if (memcg_wb_domain_init (memcg,GFP_KERNEL )) goto fail;
@@ -5991,6 +5995,29 @@ static void mem_cgroup_move_task(void) }#endif +#ifdef CONFIG_LRU_GEN +static void mem_cgroup_attach (struct cgroup_taskset *tset) +{ +struct cgroup_subsys_state *css; +struct task_struct *task =NULL ; + +cgroup_taskset_for_each_leader (task, css, tset) + ; + +if (!task) +return ; + +task_lock (task); +if (task->mm && task->mm->owner == task) +lru_gen_migrate_mm (task->mm); +task_unlock (task); +} +#else +static void mem_cgroup_attach (struct cgroup_taskset *tset) +{ +} +#endif +static int seq_puts_memcg_tunable (struct seq_file *m,unsigned long value) {if (value ==PAGE_COUNTER_MAX )
@@ -6332,6 +6359,7 @@ struct cgroup_subsys memory_cgrp_subsys = { .css_reset = mem_cgroup_css_reset, .css_rstat_flush = mem_cgroup_css_rstat_flush, .can_attach = mem_cgroup_can_attach, + .attach = mem_cgroup_attach, .cancel_attach = mem_cgroup_cancel_attach, .post_attach = mem_cgroup_move_task, .dfl_cftypes = memory_files,
mm/vmscan.c +324 -0
336 lines not on this page · raw message
Signature · 1 line
2.31.1.751.gd2f1c929bd-goog
PATCH v3 10/14 mm: multigenerational lru: aging
In reply to Yu ZhaoThe aging produces young generations. Given an lruvec, the aging scans page tables for referenced pages of this lruvec. Upon finding one, the aging updates its generation number to max_seq. After each round of scan, the aging increments max_seq. The aging is due when both of min_seq[2] reaches max_seq-1, assuming both anon and file types are reclaimable.
The aging uses the following optimizations when scanning page tables:
1) It will not scan page tables from processes that have been sleeping since the last scan.
2) It will not scan PTE tables under non-leaf PMD entries that do not have the accessed bit set, when CONFIG_HAVE_ARCH_PARENT_PMD_YOUNG=y.
3) It will not zigzag between the PGD table and the same PMD or PTE table spanning multiple VMAs. In other words, it finishes all the VMAs within the range of the same PMD or PTE table before it returns to the PGD table. This optimizes workloads that have large numbers of tiny VMAs, especially when CONFIG_PGTABLE_LEVELS=5.
The aging also takes advantage of the spatial locality: pages mapped around a referenced PTE may also have been referenced. If the rmap finds the PTE mapping a page under reclaim referenced, it will call a new function lru_gen_scan_around() to scan the vicinity of this PTE. And for each additional PTE found referenced, lru_gen_scan_around() will update the generation number of the page mapped by this PTE.
--- include/linux/mmzone.h | 6 + mm/rmap.c | 6 + mm/vmscan.c | 788 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 800 insertions(+)
include/linux/mmzone.h +6 -0
@@ -294,6 +294,7 @@ enum lruvec_flags { };struct lruvec ; +struct page_vma_mapped_walk ;#define LRU_GEN_MASK ((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF)#define LRU_USAGE_MASK ((BIT(LRU_USAGE_WIDTH) - 1) << LRU_USAGE_PGOFF)
@@ -382,6 +383,7 @@ struct lrugen {void lru_gen_init_lruvec (struct lruvec *lruvec);void lru_gen_set_state (bool enable,bool main,bool swap); +void lru_gen_scan_around (struct page_vma_mapped_walk *pvmw);#else /* CONFIG_LRU_GEN */
@@ -393,6 +395,10 @@ static inline void lru_gen_set_state(bool enable, bool main, bool swap) { } +static inline void lru_gen_scan_around (struct page_vma_mapped_walk *pvmw) +{ +} +#endif /* CONFIG_LRU_GEN */ struct lruvec {
mm/rmap.c +6 -0
@@ -72,6 +72,7 @@#include <linux/page_idle.h> #include <linux/memremap.h> #include <linux/userfaultfd_k.h> +#include <linux/mm_inline.h> #include <asm/tlbflush.h>
@@ -792,6 +793,11 @@ static bool page_referenced_one(struct page *page, struct vm_area_struct *vma, }if (pvmw.pte) { +/* the multigenerational lru exploits the spatial locality */ +if (lru_gen_enabled () &&pte_young (*pvmw.pte)) { +lru_gen_scan_around (&pvmw); + referenced++; + }if (ptep_clear_flush_young_notify (vma, address, pvmw.pte)) {/*
mm/vmscan.c +788 -0
806 lines not on this page · raw message
Signature · 1 line
2.31.1.751.gd2f1c929bd-goog
PATCH v3 11/14 mm: multigenerational lru: eviction
In reply to Yu ZhaoThe eviction consumes old generations. Given an lruvec, the eviction scans the pages on the per-zone lists indexed by either of min_seq[2]. It first tries to select a type based on the values of min_seq[2]. When anon and file types are both available from the same generation, it selects the one that has a lower refault rate.
During a scan, the eviction sorts pages according to their new generation numbers, if the aging has found them referenced. It also moves pages from the tiers that have higher refault rates than tier 0 to the next generation. When it finds all the per-zone lists of a selected type are empty, the eviction increments min_seq[2] indexed by this selected type.
With the aging and the eviction in place, we can build page reclaim in a straightforward manner:
1) In order to reduce the latency, direct reclaim only invokes the aging when both min_seq[2] reaches max_seq-1; otherwise it invokes the eviction.
2) In order to avoid the aging in the direct reclaim path, kswapd does the background aging. It invokes the aging when either of min_seq[2] reaches max_seq-1; otherwise it invokes the eviction.
--- include/linux/mmzone.h | 5 + mm/vmscan.c | 540 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 545 insertions(+)
include/linux/mmzone.h +5 -0
@@ -863,6 +863,8 @@ struct deferred_split { };#endif +struct mm_walk_args ; +/* * On NUMA machines, each NUMA node would have a pg_data_t to describe * it's memory layout. On UMA machines there is a single pglist_data which
@@ -968,6 +970,9 @@ typedef struct pglist_data {unsigned long flags; +#ifdef CONFIG_LRU_GEN +struct mm_walk_args *mm_walk_args; +#endif ZONE_PADDING (_pad2_)/* Per-node vmstats */
mm/vmscan.c +540 -0
600 lines not on this page · raw message
Signature · 1 line
2.31.1.751.gd2f1c929bd-goog
PATCH v3 12/14 mm: multigenerational lru: user interface
In reply to Yu ZhaoAdd a sysfs file /sys/kernel/mm/lru_gen/enabled to enable and disable the multigenerational lru at runtime.
Add a sysfs file /sys/kernel/mm/lru_gen/spread to optionally spread pages out across more than three generations. More generations make the background aging more aggressive.
Add a debugfs file /sys/kernel/debug/lru_gen to monitor the multigenerational lru and trigger the aging and the eviction. This file has the following output:
memcg memcg_id memcg_path
node node_id
min_gen birth_time anon_size file_size
...
max_gen birth_time anon_size file_size
Given a memcg and a node, "min_gen" is the oldest generation (number) and "max_gen" is the youngest. Birth time is in milliseconds. The sizes of anon and file types are in pages.
This file takes the following input:
+ memcg_id node_id gen [swappiness]
- memcg_id node_id gen [swappiness] [nr_to_reclaim]
The first command line accounts referenced pages to generation "max_gen" and creates the next generation "max_gen"+1. In this case, "gen" should be equal to "max_gen". A swap file and a non-zero "swappiness" are required to scan anon type. If swapping is not desired, set vm.swappiness to 0. The second command line evicts generations less than or equal to "gen". In this case, "gen" should be less than "max_gen"-1 as "max_gen" and "max_gen"-1 are active generations and therefore protected from the eviction. Use "nr_to_reclaim" to limit the number of pages to evict. Multiple command lines are supported, so does concatenation with delimiters "," and ";".
--- mm/vmscan.c | 403 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 403 insertions(+)
mm/vmscan.c +403 -0
@@ -52,6 +52,8 @@#include <linux/memory.h> #include <linux/pagewalk.h> #include <linux/shmem_fs.h> +#include <linux/ctype.h> +#include <linux/debugfs.h> #include <asm/tlbflush.h> #include <asm/div64.h>
@@ -4678,6 +4680,401 @@ static void lru_gen_stop_kswapd(int nid)kvfree (pgdat->mm_walk_args); } +/****************************************************************************** +* sysfs interface +******************************************************************************/ + +static ssize_t show_lru_gen_spread (struct kobject *kobj,struct kobj_attribute *attr, +char *buf) +{ +return sprintf (buf,"%d\n" ,READ_ONCE (lru_gen_spread)); +} + +static ssize_t store_lru_gen_spread (struct kobject *kobj,struct kobj_attribute *attr, +const char *buf,size_t len) +{ +int spread; + +if (kstrtoint (buf,10 , &spread) || spread >=MAX_NR_GENS ) +return -EINVAL ; + +WRITE_ONCE (lru_gen_spread, spread); + +return len; +} + +static struct kobj_attribute lru_gen_spread_attr =__ATTR ( + spread,0644 , show_lru_gen_spread, store_lru_gen_spread +); + +static ssize_t show_lru_gen_enabled (struct kobject *kobj,struct kobj_attribute *attr, +char *buf) +{ +return snprintf (buf,PAGE_SIZE ,"%d\n" ,lru_gen_enabled ()); +} + +static ssize_t store_lru_gen_enabled (struct kobject *kobj,struct kobj_attribute *attr, +const char *buf,size_t len) +{ +int enable; + +if (kstrtoint (buf,10 , &enable)) +return -EINVAL ; + +lru_gen_set_state (enable, true, false); + +return len; +} + +static struct kobj_attribute lru_gen_enabled_attr =__ATTR ( + enabled,0644 , show_lru_gen_enabled, store_lru_gen_enabled +); + +static struct attribute *lru_gen_attrs[] = { + &lru_gen_spread_attr.attr, + &lru_gen_enabled_attr.attr, +NULL +}; + +static struct attribute_group lru_gen_attr_group = { + .name ="lru_gen" , + .attrs = lru_gen_attrs, +}; + +/****************************************************************************** +* debugfs interface +******************************************************************************/ + +static void *lru_gen_seq_start (struct seq_file *m,loff_t *pos) +{ +struct mem_cgroup *memcg; +loff_t nr_to_skip = *pos; + + m->private =kzalloc (PATH_MAX ,GFP_KERNEL ); +if (!m->private) +return ERR_PTR (-ENOMEM ); + + memcg =mem_cgroup_iter (NULL ,NULL ,NULL ); +do { +int nid; + +for_each_node_state (nid,N_MEMORY ) { +if (!nr_to_skip--) +return mem_cgroup_lruvec (memcg,NODE_DATA (nid)); + } + }while ((memcg =mem_cgroup_iter (NULL , memcg,NULL ))); + +return NULL ; +} + +static void lru_gen_seq_stop (struct seq_file *m,void *v) +{ +if (!IS_ERR_OR_NULL (v)) +mem_cgroup_iter_break (NULL ,lruvec_memcg (v)); + +kfree (m->private); + m->private =NULL ; +} + +static void *lru_gen_seq_next (struct seq_file *m,void *v,loff_t *pos) +{ +int nid =lruvec_pgdat (v)->node_id; +struct mem_cgroup *memcg =lruvec_memcg (v); + + ++*pos; + + nid =next_memory_node (nid); +if (nid ==MAX_NUMNODES ) { + memcg =mem_cgroup_iter (NULL , memcg,NULL ); +if (!memcg) +return NULL ; + + nid = first_memory_node; + } + +return mem_cgroup_lruvec (memcg,NODE_DATA (nid)); +} + +static void lru_gen_seq_show_full (struct seq_file *m,struct lruvec *lruvec, +unsigned long max_seq,unsigned long *min_seq, +unsigned long seq) +{ +int i; +int type, tier; +int hist =hist_from_seq_or_gen (seq); +struct lrugen *lrugen = &lruvec->evictable; +int nid =lruvec_pgdat (lruvec)->node_id; +struct mem_cgroup *memcg =lruvec_memcg (lruvec); +struct lru_gen_mm_list *mm_list =get_mm_list (memcg); + +for (tier =0 ; tier <MAX_NR_TIERS ; tier++) { +seq_printf (m," %10d" , tier); +for (type =0 ; type <ANON_AND_FILE ; type++) { +unsigned long n[3 ] = {}; + +if (seq == max_seq) { + n[0 ] =READ_ONCE (lrugen->avg_refaulted[type][tier]); + n[1 ] =READ_ONCE (lrugen->avg_total[type][tier]); + +seq_printf (m," %10luR %10luT %10lu " , n[0 ], n[1 ], n[2 ]); + }else if (seq == min_seq[type] ||NR_STAT_GENS >1 ) { + n[0 ] =atomic_long_read (&lrugen->refaulted[hist][type][tier]); + n[1 ] =atomic_long_read (&lrugen->evicted[hist][type][tier]); +if (tier) + n[2 ] =READ_ONCE (lrugen->activated[hist][type][tier -1 ]); + +seq_printf (m," %10lur %10lue %10lua" , n[0 ], n[1 ], n[2 ]); + }else +seq_puts (m," 0 0 0 " ); + } +seq_putc (m,'\n' ); + } + +seq_puts (m," " ); +for (i =0 ; i <NR_MM_STATS ; i++) { +if (seq == max_seq &&NR_STAT_GENS ==1 ) +seq_printf (m," %10lu%c" ,READ_ONCE (mm_list->nodes[nid].stats[hist][i]), +toupper (MM_STAT_CODES [i])); +else if (seq != max_seq &&NR_STAT_GENS >1 ) +seq_printf (m," %10lu%c" ,READ_ONCE (mm_list->nodes[nid].stats[hist][i]), +MM_STAT_CODES [i]); +else +seq_puts (m," 0 " ); + } +seq_putc (m,'\n' ); +} + +static int lru_gen_seq_show (struct seq_file *m,void *v) +{ +unsigned long seq; +bool full = !debugfs_real_fops (m->file)->write; +struct lruvec *lruvec = v; +struct lrugen *lrugen = &lruvec->evictable; +int nid =lruvec_pgdat (lruvec)->node_id; +struct mem_cgroup *memcg =lruvec_memcg (lruvec); +DEFINE_MAX_SEQ (); +DEFINE_MIN_SEQ (); + +if (nid == first_memory_node) { +#ifdef CONFIG_MEMCG +if (memcg) +cgroup_path (memcg->css.cgroup, m->private,PATH_MAX ); +#endif +seq_printf (m,"memcg %5hu %s\n" ,mem_cgroup_id (memcg), (char *)m->private); + } + +seq_printf (m," node %5d\n" , nid); + + seq = full ? (max_seq <MAX_NR_GENS ?0 : max_seq -MAX_NR_GENS +1 ) : +min (min_seq[0 ], min_seq[1 ]); + +for (; seq <= max_seq; seq++) { +int gen, type, zone; +unsigned int msecs; + + gen =lru_gen_from_seq (seq); + msecs =jiffies_to_msecs (jiffies -READ_ONCE (lrugen->timestamps[gen])); + +seq_printf (m," %10lu %10u" , seq, msecs); + +for (type =0 ; type <ANON_AND_FILE ; type++) { +long size =0 ; + +if (seq < min_seq[type]) { +seq_puts (m," -0 " ); +continue ; + } + +for (zone =0 ; zone <MAX_NR_ZONES ; zone++) + size +=READ_ONCE (lrugen->sizes[gen][type][zone]); + +seq_printf (m," %10lu " ,max (size,0L )); + } + +seq_putc (m,'\n' ); + +if (full) +lru_gen_seq_show_full (m, lruvec, max_seq, min_seq, seq); + } + +return 0 ; +} + +static const struct seq_operations lru_gen_seq_ops = { + .start = lru_gen_seq_start, + .stop = lru_gen_seq_stop, + .next = lru_gen_seq_next, + .show = lru_gen_seq_show, +}; + +static int advance_max_seq (struct lruvec *lruvec,unsigned long seq,int swappiness) +{ +struct scan_control sc = { + .target_mem_cgroup =lruvec_memcg (lruvec), + }; +DEFINE_MAX_SEQ (); + +if (seq == max_seq) +walk_mm_list (lruvec, max_seq, &sc, swappiness,NULL ); + +return seq > max_seq ? -EINVAL :0 ; +} + +static int advance_min_seq (struct lruvec *lruvec,unsigned long seq,int swappiness, +unsigned long nr_to_reclaim) +{ +struct blk_plug plug; +int err = -EINTR ; +long nr_to_scan =LONG_MAX ; +struct scan_control sc = { + .nr_to_reclaim = nr_to_reclaim, + .target_mem_cgroup =lruvec_memcg (lruvec), + .may_writepage =1 , + .may_unmap =1 , + .may_swap =1 , + .reclaim_idx =MAX_NR_ZONES -1 , + .gfp_mask =GFP_KERNEL , + }; +DEFINE_MAX_SEQ (); + +if (seq >= max_seq -1 ) +return -EINVAL ; + +blk_start_plug (&plug); + +while (!signal_pending (current)) { +DEFINE_MIN_SEQ (); + +if (seq <min (min_seq[!swappiness], min_seq[swappiness <200 ]) || + !evict_pages (lruvec, &sc, swappiness, &nr_to_scan)) { + err =0 ; +break ; + } + +cond_resched (); + } + +blk_finish_plug (&plug); + +return err; +} + +static int advance_seq (char cmd,int memcg_id,int nid,unsigned long seq, +int swappiness,unsigned long nr_to_reclaim) +{ +struct lruvec *lruvec; +int err = -EINVAL ; +struct mem_cgroup *memcg =NULL ; + +if (!mem_cgroup_disabled ()) { +rcu_read_lock (); + memcg =mem_cgroup_from_id (memcg_id); +#ifdef CONFIG_MEMCG +if (memcg && !css_tryget (&memcg->css)) + memcg =NULL ; +#endif +rcu_read_unlock (); + +if (!memcg) + goto done; + } +if (memcg_id !=mem_cgroup_id (memcg)) + goto done; + +if (nid <0 || nid >=MAX_NUMNODES || !node_state (nid,N_MEMORY )) + goto done; + + lruvec =mem_cgroup_lruvec (memcg,NODE_DATA (nid)); + +if (swappiness ==-1 ) + swappiness =get_swappiness (lruvec); +else if (swappiness >200U ) + goto done; + +switch (cmd) { +case '+' : + err =advance_max_seq (lruvec, seq, swappiness); +break ; +case '-' : + err =advance_min_seq (lruvec, seq, swappiness, nr_to_reclaim); +break ; + } +done: +mem_cgroup_put (memcg); + +return err; +} + +static ssize_t lru_gen_seq_write (struct file *file,const char __user *src, +size_t len,loff_t *pos) +{ +void *buf; +char *cur, *next; +int err =0 ; + + buf =kvmalloc (len +1 ,GFP_USER ); +if (!buf) +return -ENOMEM ; + +if (copy_from_user (buf, src, len)) { +kvfree (buf); +return -EFAULT ; + } + + next = buf; + next[len] ='\0' ; + +while ((cur =strsep (&next,",;\n" ))) { +int n; +int end; +char cmd; +unsigned int memcg_id; +unsigned int nid; +unsigned long seq; +unsigned int swappiness =-1 ; +unsigned long nr_to_reclaim =-1 ; + + cur =skip_spaces (cur); +if (!*cur) +continue ; + + n =sscanf (cur,"%c %u %u %lu %n %u %n %lu %n" , &cmd, &memcg_id, &nid, + &seq, &end, &swappiness, &end, &nr_to_reclaim, &end); +if (n <4 || cur[end]) { + err = -EINVAL ; +break ; + } + + err =advance_seq (cmd, memcg_id, nid, seq, swappiness, nr_to_reclaim); +if (err) +break ; + } + +kvfree (buf); + +return err ? : len; +} + +static int lru_gen_seq_open (struct inode *inode,struct file *file) +{ +return seq_open (file, &lru_gen_seq_ops); +} + +static const struct file_operations lru_gen_rw_fops = { + .open = lru_gen_seq_open, + .read = seq_read, + .write = lru_gen_seq_write, + .llseek = seq_lseek, + .release = seq_release, +}; + +static const struct file_operations lru_gen_ro_fops = { + .open = lru_gen_seq_open, + .read = seq_read, + .llseek = seq_lseek, + .release = seq_release, +}; +/****************************************************************************** * initialization ******************************************************************************/
@@ -4718,6 +5115,12 @@ static int __init init_lru_gen(void)if (hotplug_memory_notifier (lru_gen_online_mem,0 ))pr_err ("lru_gen: failed to subscribe hotplug notifications\n" ); +if (sysfs_create_group (mm_kobj, &lru_gen_attr_group)) +pr_err ("lru_gen: failed to create sysfs group\n" ); + +debugfs_create_file ("lru_gen" ,0644 ,NULL ,NULL , &lru_gen_rw_fops); +debugfs_create_file ("lru_gen_full" ,0444 ,NULL ,NULL , &lru_gen_ro_fops); +return 0 ; };/*
Signature · 1 line
2.31.1.751.gd2f1c929bd-goog
PATCH v3 13/14 mm: multigenerational lru: Kconfig
In reply to Yu ZhaoAdd configuration options for the multigenerational lru.
--- mm/Kconfig | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+)
mm/Kconfig +58 -0
@@ -901,4 +901,62 @@ config KMAP_LOCAL # struct io_mapping based helper. Selected by drivers that need them config IO_MAPPING bool + +# the multigenerational lru { +config LRU_GEN + bool "Multigenerational LRU" + depends on MMU + help + A high performance LRU implementation to heavily overcommit workloads + that are not IO bound. See Documentation/vm/multigen_lru.rst for + details. + + Warning: do not enable this option unless you plan to use it because + it introduces a small per-process and per-memcg and per-node memory + overhead. + +config LRU_GEN_ENABLED + bool "Turn on by default" + depends on LRU_GEN + help + The default value of /sys/kernel/mm/lru_gen/enabled is 0. This option + changes it to 1. + + Warning: the default value is the fast path. See + Documentation/static-keys.txt for details. + +config LRU_GEN_STATS + bool "Full stats for debugging" + depends on LRU_GEN + help + This option keeps full stats for each generation, which can be read + from /sys/kernel/debug/lru_gen_full. + + Warning: do not enable this option unless you plan to use it because + it introduces an additional small per-process and per-memcg and + per-node memory overhead. + +config NR_LRU_GENS + int "Max number of generations" + depends on LRU_GEN + range 4 31 + default 7 + help + This will use order_base_2(N+1) spare bits from page flags. + + Warning: do not use numbers larger than necessary because each + generation introduces a small per-node and per-memcg memory overhead. + +config TIERS_PER_GEN + int "Number of tiers per generation" + depends on LRU_GEN + range 2 5 + default 4 + help + This will use N-2 spare bits from page flags. + + Larger values generally offer better protection to active pages under + heavy buffered I/O workloads. +# } + endmenu
Signature · 1 line
2.31.1.751.gd2f1c929bd-goog
PATCH v3 14/14 mm: multigenerational lru: documentation
In reply to Yu ZhaoAdd Documentation/vm/multigen_lru.rst.
--- Documentation/vm/index.rst | 1 + Documentation/vm/multigen_lru.rst | 143 ++++++++++++++++++++++++++++++ 2 files changed, 144 insertions(+) create mode 100644 Documentation/vm/multigen_lru.rst
Documentation/vm/index.rst +1 -0
@@ -17,6 +17,7 @@ various features of the Linux memory management swap_numa zswap + multigen_lru Kernel developers MM documentation ==================================
Documentation/vm/multigen_lru.rst new file +143 -0
143 lines not on this page · raw message
Signature · 1 line
2.31.1.751.gd2f1c929bd-goog
Quoted 6 lines · On Thu, 20 May 2021 00:53:41 -0600 Yu Zhao wrote:
What's new in v3
================
1) Fixed a bug reported by the Arch Linux kernel team:https://github.com/zen-kernel/zen-kernel/issues/2072) Rebased to v5.13-rc2.
Given no reply within two months, feel free to spin, would-be Mr Kswapd.
Hillf
Quoted 10 lines · Hillf Danton
Quoted 6 lines · On Thu, 20 May 2021 00:53:41 -0600 Yu Zhao wrote:
What's new in v3
================
1) Fixed a bug reported by the Arch Linux kernel team:https://github.com/zen-kernel/zen-kernel/issues/2072) Rebased to v5.13-rc2.
Given no reply within two months, feel free to spin, would-be Mr Kswapd.
Thanks for the reminder. Will gladly get on it sometime this week.
Quoted 1 line
Given no reply within two months
We are discussing mgLRU outside the mailing lists.
Some issues have been found, some have been resolved[1], some are still pending[2].
[1] https://github.com/zen-kernel/zen-kernel/issues/216
[2] https://github.com/zen-kernel/zen-kernel/issues/223
Quoted 5 lines · On 7 Aug 2021 14:04 Alexey Avramov wrote:
Quoted 2 lines
Given no reply within two months
We are discussing mgLRU outside the mailing lists.
Some issues have been found, some have been resolved[1], some are still pending[2].
Thanks for sharing that.
What was preventing you from solving the issues at linux-mm?
Quoted 1 line
What was preventing you from solving the issues at linux-mm?
It was just more convenient for me.
But I agree that mailing lists are a better place.