Multi-Gen LRU Framework
Subsystems (20)
v11 cover Multi-Gen LRU Framework
TLDR
====
The current page reclaim is too expensive in terms of CPU usage and it often makes poor choices about what to evict. This patchset offers an alternative solution that is performant, versatile and straightforward.
Patchset overview
=================
The design and implementation overview is in patch 14:
https://lore.kernel.org/r/[email protected]/
01. mm: x86, arm64: add arch_has_hw_pte_young()
02. mm: x86: add CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG
Take advantage of hardware features when trying to clear the accessed bit in many PTEs.
03. mm/vmscan.c: refactor shrink_node()
04. Revert "include/linux/mm_inline.h: fold __update_lru_size() into its sole caller"
Minor refactors to improve readability for the following patches.
05. mm: multi-gen LRU: groundwork
Adds the basic data structure and the functions that insert pages to and remove pages from the multi-gen LRU (MGLRU) lists.
06. mm: multi-gen LRU: minimal implementation
A minimal implementation without optimizations.
07. mm: multi-gen LRU: exploit locality in rmap
Exploits spatial locality to improve efficiency when using the rmap.
08. mm: multi-gen LRU: support page table walks
Further exploits spatial locality by optionally scanning page tables.
09. mm: multi-gen LRU: optimize multiple memcgs Optimizes the overall performance for multiple memcgs running mixed types of workloads.
10. mm: multi-gen LRU: kill switch
Adds a kill switch to enable or disable MGLRU at runtime.
11. mm: multi-gen LRU: thrashing prevention
12. mm: multi-gen LRU: debugfs interface
Provide userspace with features like thrashing prevention, working set estimation and proactive reclaim.
13. mm: multi-gen LRU: admin guide
14. mm: multi-gen LRU: design doc
Add an admin guide and a design doc.
Benchmark results
=================
Independent lab results
-----------------------
Based on the popularity of searches [01] and the memory usage in Google's public cloud, the most popular open-source memory-hungry applications, in alphabetical order, are:
Apache Cassandra Memcached
Apache Hadoop MongoDB
Apache Spark PostgreSQL
MariaDB (MySQL) RedisAn independent lab evaluated MGLRU with the most widely used benchmark suites for the above applications. They posted 960 data points along with kernel metrics and perf profiles collected over more than 500 hours of total benchmark time. Their final reports show that, with 95% confidence intervals (CIs), the above applications all performed significantly better for at least part of their benchmark matrices.
On 5.14:
1. Apache Spark [02] took 95% CIs [9.28, 11.19]% and [12.20, 14.93]% less wall time to sort three billion random integers, respectively, under the medium- and the high-concurrency conditions, when overcommitting memory. There were no statistically significant changes in wall time for the rest of the benchmark matrix.
2. MariaDB [03] achieved 95% CIs [5.24, 10.71]% and [20.22, 25.97]% more transactions per minute (TPM), respectively, under the medium- and the high-concurrency conditions, when overcommitting memory. There were no statistically significant changes in TPM for the rest of the benchmark matrix.
3. Memcached [04] achieved 95% CIs [23.54, 32.25]%, [20.76, 41.61]% and [21.59, 30.02]% more operations per second (OPS), respectively, for sequential access, random access and Gaussian (distribution) access, when THP=always; 95% CIs [13.85, 15.97]% and [23.94, 29.92]% more OPS, respectively, for random access and Gaussian access, when THP=never. There were no statistically significant changes in OPS for the rest of the benchmark matrix.
4. MongoDB [05] achieved 95% CIs [2.23, 3.44]%, [6.97, 9.73]% and [2.16, 3.55]% more operations per second (OPS), respectively, for exponential (distribution) access, random access and Zipfian (distribution) access, when underutilizing memory; 95% CIs [8.83, 10.03]%, [21.12, 23.14]% and [5.53, 6.46]% more OPS, respectively, for exponential access, random access and Zipfian access, when overcommitting memory.
On 5.15:
5. Apache Cassandra [06] achieved 95% CIs [1.06, 4.10]%, [1.94, 5.43]% and [4.11, 7.50]% more operations per second (OPS), respectively, for exponential (distribution) access, random access and Zipfian (distribution) access, when swap was off; 95% CIs [0.50, 2.60]%, [6.51, 8.77]% and [3.29, 6.75]% more OPS, respectively, for exponential access, random access and Zipfian access, when swap was on.
6. Apache Hadoop [07] took 95% CIs [5.31, 9.69]% and [2.02, 7.86]% less average wall time to finish twelve parallel TeraSort jobs, respectively, under the medium- and the high-concurrency conditions, when swap was on. There were no statistically significant changes in average wall time for the rest of the benchmark matrix.
7. PostgreSQL [08] achieved 95% CI [1.75, 6.42]% more transactions per minute (TPM) under the high-concurrency condition, when swap was off; 95% CIs [12.82, 18.69]% and [22.70, 46.86]% more TPM, respectively, under the medium- and the high-concurrency conditions, when swap was on. There were no statistically significant changes in TPM for the rest of the benchmark matrix.
8. Redis [09] achieved 95% CIs [0.58, 5.94]%, [6.55, 14.58]% and [11.47, 19.36]% more total operations per second (OPS), respectively, for sequential access, random access and Gaussian (distribution) access, when THP=always; 95% CIs [1.27, 3.54]%, [10.11, 14.81]% and [8.75, 13.64]% more total OPS, respectively, for sequential access, random access and Gaussian access, when THP=never.
Our lab results
---------------
To supplement the above results, we ran the following benchmark suites on 5.16-rc7 and found no regressions [10].
fs_fio_bench_hdd_mq pft
fs_lmbench pgsql-hammerdb
fs_parallelio redis
fs_postmark stream
hackbench sysbenchthread
kernbench tpcc_spark
memcached unixbench
multichase vm-scalability
mutilate will-it-scalenginx
[01] https://trends.google.com
[02] https://lore.kernel.org/r/[email protected]/
[03] https://lore.kernel.org/r/[email protected]/
[04] https://lore.kernel.org/r/[email protected]/
[05] https://lore.kernel.org/r/[email protected]/
[06] https://lore.kernel.org/r/[email protected]/
[07] https://lore.kernel.org/r/[email protected]/
[08] https://lore.kernel.org/r/[email protected]/
[09] https://lore.kernel.org/r/[email protected]/
[10] https://lore.kernel.org/r/[email protected]/
Read-world applications
=======================
Third-party testimonials
------------------------
Konstantin reported [11]:
I have Archlinux with 8G RAM + zswap + swap. While developing, I have lots of apps opened such as multiple LSP-servers for different langs, chats, two browsers, etc... Usually, my system gets quickly to a point of SWAP-storms, where I have to kill LSP-servers, restart browsers to free memory, etc, otherwise the system lags heavily and is barely usable.
1.5 day ago I migrated from 5.11.15 kernel to 5.12 + the LRU patchset, and I started up by opening lots of apps to create memory pressure, and worked for a day like this. Till now I had not a single SWAP-storm, and mind you I got 3.4G in SWAP. I was never getting to the point of 3G in SWAP before without a single SWAP-storm.
Vaibhav from IBM reported [12]:
In a synthetic MongoDB Benchmark, seeing an average of ~19% throughput improvement on POWER10(Radix MMU + 64K Page Size) with MGLRU patches on top of v5.16 kernel for MongoDB + YCSB across three different request distributions, namely, Exponential, Uniform and Zipfan.
Shuang from U of Rochester reported [13]:
With the MGLRU, fio achieved 95% CIs [38.95, 40.26]%, [4.12, 6.64]% and [9.26, 10.36]% higher throughput, respectively, for random access, Zipfian (distribution) access and Gaussian (distribution) access, when the average number of jobs per CPU is 1; 95% CIs [42.32, 49.15]%, [9.44, 9.89]% and [20.99, 22.86]% higher throughput, respectively, for random access, Zipfian access and Gaussian access, when the average number of jobs per CPU is 2.
Daniel from Michigan Tech reported [14]:
With Memcached allocating ~100GB of byte-addressable Optante, performance improvement in terms of throughput (measured as queries per second) was about 10% for a series of workloads.
Large-scale deployments
-----------------------
The downstream kernels that have been using MGLRU include:
1. Android [15]
2. Arch Linux Zen [16]
3. Chrome OS [17]
4. Liquorix [18]
5. post-factum [19]
6. XanMod [20]
We've rolled out MGLRU to tens of millions of Chrome OS users and about a million Android users. Google's fleetwide profiling [21] shows an overall 40% decrease in kswapd CPU usage, in addition to improvements in other UX metrics, e.g., an 85% decrease in the number of low-memory kills at the 75th percentile and an 18% decrease in app launch time at the 50th percentile.
[11] https://lore.kernel.org/r/[email protected]/
[12] https://lore.kernel.org/r/[email protected]/
[13] https://lore.kernel.org/r/[email protected]/
[14] https://lore.kernel.org/r/CA+4-3vksGvKd18FgRinxhqHetBS1hQekJE2gwco8Ja-bJWKtFw@mail.gmail.com/
[15] https://android.com
[16] https://archlinux.org
[17] https://chromium.org
[18] https://liquorix.net
[19] https://gitlab.com/post-factum/pf-kernel/
[20] https://xanmod.org
[21] https://research.google/pubs/pub44271/
Summery
=======
The facts are:
1. The independent lab results and the real-world applications indicate substantial improvements; there are no known regressions.
2. Thrashing prevention, working set estimation and proactive reclaim work out of the box; there are no equivalent solutions.
3. There is a lot of new code; no one has demonstrated smaller changes with similar effects.
Our options, accordingly, are:
1. Given the amount of evidence, the reported improvements will likely materialize for a wide range of workloads.
2. Gauging the interest from the past discussions, the new features will likely be put to use for both personal computers and data centers.
3. Based on Google's track record, the new code will likely be well maintained in the long term. It'd be more difficult if not impossible to achieve similar effects with other alternatives.
Yu Zhao (14):
mm: x86, arm64: add arch_has_hw_pte_young()
mm: x86: add CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG mm/vmscan.c: refactor shrink_node()
Revert "include/linux/mm_inline.h: fold __update_lru_size() into its
sole caller"
mm: multi-gen LRU: groundwork
mm: multi-gen LRU: minimal implementation
mm: multi-gen LRU: exploit locality in rmap
mm: multi-gen LRU: support page table walks
mm: multi-gen LRU: optimize multiple memcgs
mm: multi-gen LRU: kill switch
mm: multi-gen LRU: thrashing prevention
mm: multi-gen LRU: debugfs interface
mm: multi-gen LRU: admin guide
mm: multi-gen LRU: design doc
Documentation/admin-guide/mm/index.rst | 1 + Documentation/admin-guide/mm/multigen_lru.rst | 156 + Documentation/vm/index.rst | 1 + Documentation/vm/multigen_lru.rst | 159 + arch/Kconfig | 8 + arch/arm64/include/asm/pgtable.h | 14 +- arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 9 +- arch/x86/mm/pgtable.c | 5 +- fs/exec.c | 2 + fs/fuse/dev.c | 3 +- include/linux/cgroup.h | 15 +- include/linux/memcontrol.h | 36 + include/linux/mm.h | 7 + include/linux/mm_inline.h | 233 +- include/linux/mm_types.h | 77 + include/linux/mmzone.h | 216 ++ include/linux/nodemask.h | 1 + include/linux/page-flags-layout.h | 16 +- include/linux/page-flags.h | 4 +- include/linux/pgtable.h | 17 +- include/linux/sched.h | 4 + include/linux/swap.h | 4 + kernel/bounds.c | 7 + kernel/cgroup/cgroup-internal.h | 1 - kernel/exit.c | 1 + kernel/fork.c | 9 + kernel/sched/core.c | 1 + mm/Kconfig | 26 + mm/huge_memory.c | 3 +- mm/internal.h | 1 + mm/memcontrol.c | 28 + mm/memory.c | 39 +- mm/mm_init.c | 6 +- mm/mmzone.c | 2 + mm/rmap.c | 7 + mm/swap.c | 52 +- mm/vmscan.c | 2888 ++++++++++++++++- mm/workingset.c | 110 +- 39 files changed, 4017 insertions(+), 153 deletions(-) create mode 100644 Documentation/admin-guide/mm/multigen_lru.rst create mode 100644 Documentation/vm/multigen_lru.rst
Signature · 1 line
2.36.0.550.gb090851708-goog
PATCH v11 1/14 mm: x86, arm64: add arch_has_hw_pte_young()
Some architectures automatically set the accessed bit in PTEs, e.g., x86 and arm64 v8.2. On architectures that do not have this capability, clearing the accessed bit in a PTE usually triggers a page fault following the TLB miss of this PTE (to emulate the accessed bit).
Being aware of this capability can help make better decisions, e.g., whether to spread the work out over a period of time to reduce bursty page faults when trying to clear the accessed bit in many PTEs.
Note that theoretically this capability can be unreliable, e.g., hotplugged CPUs might be different from builtin ones. Therefore it should not be used in architecture-independent code that involves correctness, e.g., to determine whether TLB flushes are required (in combination with the accessed bit).
--- arch/arm64/include/asm/pgtable.h | 14 ++------------ arch/x86/include/asm/pgtable.h | 6 +++--- include/linux/pgtable.h | 13 +++++++++++++ mm/memory.c | 14 +------------- 4 files changed, 19 insertions(+), 28 deletions(-)
arch/arm64/include/asm/pgtable.h +2 -12
@@ -999,23 +999,13 @@ static inline void update_mmu_cache(struct vm_area_struct *vma,* page after fork() + CoW for pfn mappings. We don't always have a * hardware-managed access flag on arm64. */ -static inline bool arch_faults_on_old_pte (void ) -{ -WARN_ON (preemptible ()); - -return !cpu_has_hw_af (); -} -#define arch_faults_on_old_pte arch_faults_on_old_pte +#define arch_has_hw_pte_young cpu_has_hw_af/* * Experimentally, it's cheap to set the access flag in hardware and we * benefit from prefaulting mappings as 'old' to start with. */ -static inline bool arch_wants_old_prefaulted_pte (void ) -{ -return !arch_faults_on_old_pte (); -} -#define arch_wants_old_prefaulted_pte arch_wants_old_prefaulted_pte +#define arch_wants_old_prefaulted_pte cpu_has_hw_afstatic inline bool pud_sect_supported (void ) {
arch/x86/include/asm/pgtable.h +3 -3
@@ -1424,10 +1424,10 @@ static inline bool arch_has_pfn_modify_check(void) return boot_cpu_has_bug(X86_BUG_L1TF); } -#define arch_faults_on_old_pte arch_faults_on_old_pte -static inline bool arch_faults_on_old_pte (void ) +#define arch_has_hw_pte_young arch_has_hw_pte_young +static inline bool arch_has_hw_pte_young (void ) { -return false; +return true; }#endif /* __ASSEMBLY__ */
include/linux/pgtable.h +13 -0
@@ -259,6 +259,19 @@ static inline int pmdp_clear_flush_young(struct vm_area_struct *vma,#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif +#ifndef arch_has_hw_pte_young +/* +* Return whether the accessed bit is supported on the local CPU. +* +* This stub assumes accessing through an old PTE triggers a page fault. +* Architectures that automatically set the access bit should overwrite it. +*/ +static inline bool arch_has_hw_pte_young (void ) +{ +return false; +} +#endif +#ifndef __HAVE_ARCH_PTEP_CLEARstatic inline void ptep_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep)
mm/memory.c +1 -13
@@ -122,18 +122,6 @@ int randomize_va_space __read_mostly =2 ;#endif -#ifndef arch_faults_on_old_pte -static inline bool arch_faults_on_old_pte (void ) -{ -/* -* Those arches which don't have hw access flag feature need to -* implement their own helper. By default, "true" means pagefault -* will be hit on old pte. -*/ -return true; -} -#endif -#ifndef arch_wants_old_prefaulted_ptestatic inline bool arch_wants_old_prefaulted_pte (void ) {
@@ -2784,7 +2772,7 @@ static inline bool cow_user_page(struct page *dst, struct page *src,* On architectures with software "accessed" bits, we would * take a double page fault, so mark it accessed here. */ -if (arch_faults_on_old_pte () && !pte_young (vmf->orig_pte)) { +if (!arch_has_hw_pte_young () && !pte_young (vmf->orig_pte)) {pte_t entry; vmf->pte =pte_offset_map_lock (mm, vmf->pmd, addr, &vmf->ptl);
Signature · 1 line
2.36.0.550.gb090851708-goog
PATCH v11 2/14 mm: x86: add CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG
In reply to Yu ZhaoSome architectures support the accessed bit in non-leaf PMD entries, e.g., x86 sets the accessed bit in a non-leaf PMD entry when using it as part of linear address translation [1]. Page table walkers that clear the accessed bit may use this capability to reduce their search space.
Note that:
1. Although an inline function is preferable, this capability is added as a configuration option for consistency with the existing macros.
2. Due to the little interest in other varieties, this capability was only tested on Intel and AMD CPUs.
Thanks to the following developers for their efforts [2][3].
Randy Dunlap <[email protected]>
Stephen Rothwell <[email protected]>
[1]: Intel 64 and IA-32 Architectures Software Developer's Manual
Volume 3 (June 2021), section 4.8
[2] https://lore.kernel.org/r/[email protected]/
[3] https://lore.kernel.org/r/[email protected]/
--- arch/Kconfig | 8 ++++++++ arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 3 ++- arch/x86/mm/pgtable.c | 5 ++++- include/linux/pgtable.h | 4 ++-- 5 files changed, 17 insertions(+), 4 deletions(-)
arch/Kconfig +8 -0
@@ -1376,6 +1376,14 @@ config DYNAMIC_SIGFRAME config HAVE_ARCH_NODE_DEV_GROUP bool +config ARCH_HAS_NONLEAF_PMD_YOUNG + bool + help + Architectures that select this option are capable of setting the + accessed bit in non-leaf PMD entries when using them as part of linear + address translations. Page table walkers that clear the accessed bit + may use this capability to reduce their search space. + source "kernel/gcov/Kconfig" source "scripts/gcc-plugins/Kconfig"
arch/x86/Kconfig +1 -0
@@ -86,6 +86,7 @@ config X86 select ARCH_HAS_PMEM_API if X86_64 select ARCH_HAS_PTE_DEVMAP if X86_64 select ARCH_HAS_PTE_SPECIAL + select ARCH_HAS_NONLEAF_PMD_YOUNG if PGTABLE_LEVELS > 2 select ARCH_HAS_UACCESS_FLUSHCACHE if X86_64 select ARCH_HAS_COPY_MC if X86_64 select ARCH_HAS_SET_MEMORY
arch/x86/include/asm/pgtable.h +2 -1
@@ -820,7 +820,8 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)static inline int pmd_bad (pmd_t pmd) { -return (pmd_flags (pmd) & ~_PAGE_USER) != _KERNPG_TABLE; +return (pmd_flags (pmd) & ~(_PAGE_USER | _PAGE_ACCESSED)) != + (_KERNPG_TABLE & ~_PAGE_ACCESSED); }static inline unsigned long pages_to_mb (unsigned long npg)
arch/x86/mm/pgtable.c +4 -1
@@ -550,7 +550,7 @@ int ptep_test_and_clear_young(struct vm_area_struct *vma,return ret; } -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#if defined(CONFIG_TRANSPARENT_HUGEPAGE ) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG )int pmdp_test_and_clear_young (struct vm_area_struct *vma,unsigned long addr,pmd_t *pmdp) {
@@ -562,6 +562,9 @@ int pmdp_test_and_clear_young(struct vm_area_struct *vma,return ret; } +#endif + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE int pudp_test_and_clear_young (struct vm_area_struct *vma,unsigned long addr,pud_t *pudp) {
include/linux/pgtable.h +2 -2
@@ -212,7 +212,7 @@ static inline int ptep_test_and_clear_young(struct vm_area_struct *vma,#endif #ifndef __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#if defined(CONFIG_TRANSPARENT_HUGEPAGE ) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG )static inline int pmdp_test_and_clear_young (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp)
@@ -233,7 +233,7 @@ static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma,BUILD_BUG ();return 0 ; } -#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +#endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG */ #endif #ifndef __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH
Signature · 1 line
2.36.0.550.gb090851708-goog
PATCH v11 3/14 mm/vmscan.c: refactor shrink_node()
In reply to Yu ZhaoThis patch refactors shrink_node() to improve readability for the upcoming changes to mm/vmscan.c.
--- mm/vmscan.c | 198 +++++++++++++++++++++++++++------------------------- 1 file changed, 104 insertions(+), 94 deletions(-)
mm/vmscan.c +104 -94
@@ -2644,6 +2644,109 @@ enum scan_balance { SCAN_FILE, }; +static void prepare_scan_count (pg_data_t *pgdat,struct scan_control *sc) +{ +unsigned long file; +struct lruvec *target_lruvec; + + target_lruvec =mem_cgroup_lruvec (sc->target_mem_cgroup, pgdat); + +/* +* Flush the memory cgroup stats, so that we read accurate per-memcg +* lruvec stats for heuristics. +*/ +mem_cgroup_flush_stats (); + +/* +* Determine the scan balance between anon and file LRUs. +*/ +spin_lock_irq (&target_lruvec->lru_lock); + sc->anon_cost = target_lruvec->anon_cost; + sc->file_cost = target_lruvec->file_cost; +spin_unlock_irq (&target_lruvec->lru_lock); + +/* +* Target desirable inactive:active list ratios for the anon +* and file LRU lists. +*/ +if (!sc->force_deactivate) { +unsigned long refaults; + + refaults =lruvec_page_state (target_lruvec, +WORKINGSET_ACTIVATE_ANON ); +if (refaults != target_lruvec->refaults[0 ] || +inactive_is_low (target_lruvec,LRU_INACTIVE_ANON )) + sc->may_deactivate |=DEACTIVATE_ANON ; +else + sc->may_deactivate &= ~DEACTIVATE_ANON ; + +/* +* When refaults are being observed, it means a new +* workingset is being established. Deactivate to get +* rid of any stale active pages quickly. +*/ + refaults =lruvec_page_state (target_lruvec, +WORKINGSET_ACTIVATE_FILE ); +if (refaults != target_lruvec->refaults[1 ] || +inactive_is_low (target_lruvec,LRU_INACTIVE_FILE )) + sc->may_deactivate |=DEACTIVATE_FILE ; +else + sc->may_deactivate &= ~DEACTIVATE_FILE ; + }else + sc->may_deactivate =DEACTIVATE_ANON |DEACTIVATE_FILE ; + +/* +* If we have plenty of inactive file pages that aren't +* thrashing, try to reclaim those first before touching +* anonymous pages. +*/ + file =lruvec_page_state (target_lruvec,NR_INACTIVE_FILE ); +if (file >> sc->priority && !(sc->may_deactivate &DEACTIVATE_FILE )) + sc->cache_trim_mode =1 ; +else + sc->cache_trim_mode =0 ; + +/* +* Prevent the reclaimer from falling into the cache trap: as +* cache pages start out inactive, every cache fault will tip +* the scan balance towards the file LRU. And as the file LRU +* shrinks, so does the window for rotation from references. +* This means we have a runaway feedback loop where a tiny +* thrashing file LRU becomes infinitely more attractive than +* anon pages. Try to detect this based on file LRU size. +*/ +if (!cgroup_reclaim (sc)) { +unsigned long total_high_wmark =0 ; +unsigned long free, anon; +int z; + + free =sum_zone_node_page_state (pgdat->node_id,NR_FREE_PAGES ); + file =node_page_state (pgdat,NR_ACTIVE_FILE ) + +node_page_state (pgdat,NR_INACTIVE_FILE ); + +for (z =0 ; z <MAX_NR_ZONES ; z++) { +struct zone *zone = &pgdat->node_zones[z]; + +if (!managed_zone (zone)) +continue ; + + total_high_wmark +=high_wmark_pages (zone); + } + +/* +* Consider anon: if that's low too, this isn't a +* runaway file reclaim problem, but rather just +* extreme pressure. Reclaim as per usual then. +*/ + anon =node_page_state (pgdat,NR_INACTIVE_ANON ); + + sc->file_is_tiny = + file + free <= total_high_wmark && + !(sc->may_deactivate &DEACTIVATE_ANON ) && + anon >> sc->priority; + } +} +/* * Determine how aggressively the anon and file LRU lists should be * scanned. The relative value of each set of LRU lists is determined
@@ -3114,109 +3217,16 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc)unsigned long nr_reclaimed, nr_scanned;struct lruvec *target_lruvec;bool reclaimable = false; -unsigned long file; target_lruvec =mem_cgroup_lruvec (sc->target_mem_cgroup, pgdat); again: -/* -* Flush the memory cgroup stats, so that we read accurate per-memcg -* lruvec stats for heuristics. -*/ -mem_cgroup_flush_stats (); -memset (&sc->nr,0 ,sizeof (sc->nr)); nr_reclaimed = sc->nr_reclaimed; nr_scanned = sc->nr_scanned; -/* -* Determine the scan balance between anon and file LRUs. -*/ -spin_lock_irq (&target_lruvec->lru_lock); - sc->anon_cost = target_lruvec->anon_cost; - sc->file_cost = target_lruvec->file_cost; -spin_unlock_irq (&target_lruvec->lru_lock); - -/* -* Target desirable inactive:active list ratios for the anon -* and file LRU lists. -*/ -if (!sc->force_deactivate) { -unsigned long refaults; - - refaults =lruvec_page_state (target_lruvec, -WORKINGSET_ACTIVATE_ANON ); -if (refaults != target_lruvec->refaults[0 ] || -inactive_is_low (target_lruvec,LRU_INACTIVE_ANON )) - sc->may_deactivate |=DEACTIVATE_ANON ; -else - sc->may_deactivate &= ~DEACTIVATE_ANON ; - -/* -* When refaults are being observed, it means a new -* workingset is being established. Deactivate to get -* rid of any stale active pages quickly. -*/ - refaults =lruvec_page_state (target_lruvec, -WORKINGSET_ACTIVATE_FILE ); -if (refaults != target_lruvec->refaults[1 ] || -inactive_is_low (target_lruvec,LRU_INACTIVE_FILE )) - sc->may_deactivate |=DEACTIVATE_FILE ; -else - sc->may_deactivate &= ~DEACTIVATE_FILE ; - }else - sc->may_deactivate =DEACTIVATE_ANON |DEACTIVATE_FILE ; - -/* -* If we have plenty of inactive file pages that aren't -* thrashing, try to reclaim those first before touching -* anonymous pages. -*/ - file =lruvec_page_state (target_lruvec,NR_INACTIVE_FILE ); -if (file >> sc->priority && !(sc->may_deactivate &DEACTIVATE_FILE )) - sc->cache_trim_mode =1 ; -else - sc->cache_trim_mode =0 ; - -/* -* Prevent the reclaimer from falling into the cache trap: as -* cache pages start out inactive, every cache fault will tip -* the scan balance towards the file LRU. And as the file LRU -* shrinks, so does the window for rotation from references. -* This means we have a runaway feedback loop where a tiny -* thrashing file LRU becomes infinitely more attractive than -* anon pages. Try to detect this based on file LRU size. -*/ -if (!cgroup_reclaim (sc)) { -unsigned long total_high_wmark =0 ; -unsigned long free, anon; -int z; - - free =sum_zone_node_page_state (pgdat->node_id,NR_FREE_PAGES ); - file =node_page_state (pgdat,NR_ACTIVE_FILE ) + -node_page_state (pgdat,NR_INACTIVE_FILE ); - -for (z =0 ; z <MAX_NR_ZONES ; z++) { -struct zone *zone = &pgdat->node_zones[z]; -if (!managed_zone (zone)) -continue ; - - total_high_wmark +=high_wmark_pages (zone); - } - -/* -* Consider anon: if that's low too, this isn't a -* runaway file reclaim problem, but rather just -* extreme pressure. Reclaim as per usual then. -*/ - anon =node_page_state (pgdat,NR_INACTIVE_ANON ); - - sc->file_is_tiny = - file + free <= total_high_wmark && - !(sc->may_deactivate &DEACTIVATE_ANON ) && - anon >> sc->priority; - } +prepare_scan_count (pgdat, sc);shrink_node_memcgs (pgdat, sc);
Signature · 1 line
2.36.0.550.gb090851708-goog
PATCH v11 4/14 Revert "include/linux/mm_inline.h: fold __update_lru_size() into its sole caller"
In reply to Yu ZhaoThis patch undoes the following refactor:
commit 289ccba18af4 ("include/linux/mm_inline.h: fold __update_lru_size() into its sole caller")
The upcoming changes to include/linux/mm_inline.h will reuse __update_lru_size().
--- include/linux/mm_inline.h | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
include/linux/mm_inline.h +8 -1
@@ -32,7 +32,7 @@ static inline int page_is_file_lru(struct page *page)return folio_is_file_lru (page_folio (page)); } -static __always_inline void update_lru_size(struct lruvec *lruvec, +static __always_inline void __update_lru_size(struct lruvec *lruvec,enum lru_list lru,enum zone_type zid,long nr_pages) {
@@ -41,6 +41,13 @@ static __always_inline void update_lru_size(struct lruvec *lruvec,__mod_lruvec_state (lruvec,NR_LRU_BASE + lru, nr_pages);__mod_zone_page_state (&pgdat->node_zones[zid],NR_ZONE_LRU_BASE + lru, nr_pages); +} + +static __always_inline void update_lru_size(struct lruvec *lruvec, +enum lru_list lru,enum zone_type zid, +int nr_pages) +{ +__update_lru_size (lruvec, lru, zid, nr_pages);#ifdef CONFIG_MEMCG mem_cgroup_update_lru_size (lruvec, lru, zid, nr_pages);#endif
Signature · 1 line
2.36.0.550.gb090851708-goog
PATCH v11 5/14 mm: multi-gen LRU: groundwork
In reply to Yu ZhaoEvictable pages are divided into multiple generations for each lruvec. The youngest generation number is stored in lrugen->max_seq for both anon and file types as they are aged on an equal footing. The oldest generation numbers are stored in lrugen->min_seq[] separately for anon and file types as clean file pages can be evicted regardless of swap constraints. These three variables are monotonically increasing.
Generation numbers are truncated into order_base_2(MAX_NR_GENS+1) bits in order to fit into the gen counter in folio->flags. Each truncated generation number is an index to lrugen->lists[]. The sliding window technique is used to track at least MIN_NR_GENS and at most MAX_NR_GENS generations. The gen counter stores a value within [1, MAX_NR_GENS] while a page is on one of lrugen->lists[]. Otherwise it stores 0.
There are two conceptually independent procedures: "the aging", which produces young generations, and "the eviction", which consumes old generations. They form a closed-loop system, i.e., "the page reclaim". Both procedures can be invoked from userspace for the purposes of working set estimation and proactive reclaim. These techniques are commonly used to optimize job scheduling (bin packing) in data centers [1][2].
To avoid confusion, the terms "hot" and "cold" will be applied to the multi-gen LRU, as a new convention; the terms "active" and "inactive" will be applied to the active/inactive LRU, as usual.
The protection of hot pages and the selection of cold pages are based on page access channels and patterns. There are two access channels:
one through page tables and the other through file descriptors. The protection of the former channel is by design stronger because:
1. The uncertainty in determining the access patterns of the former channel is higher due to the approximation of the accessed bit.
2. The cost of evicting the former channel is higher due to the TLB flushes required and the likelihood of encountering the dirty bit.
3. The penalty of underprotecting the former channel is higher because applications usually do not prepare themselves for major page faults like they do for blocked I/O. E.g., GUI applications commonly use dedicated I/O threads to avoid blocking the rendering threads.
There are also two access patterns: one with temporal locality and the other without. For the reasons listed above, the former channel is assumed to follow the former pattern unless VM_SEQ_READ or VM_RAND_READ is present; the latter channel is assumed to follow the latter pattern unless outlying refaults have been observed [3][4].
The next patch will address the "outlying refaults". Three macros, i.e., LRU_REFS_WIDTH, LRU_REFS_PGOFF and LRU_REFS_MASK, used later are added in this patch to make the entire patchset less diffy.
A page is added to the youngest generation on faulting. The aging needs to check the accessed bit at least twice before handing this page over to the eviction. The first check takes care of the accessed bit set on the initial fault; the second check makes sure this page has not been used since then. This protocol, AKA second chance, requires a minimum of two generations, hence MIN_NR_GENS.
[1] https://dl.acm.org/doi/10.1145/3297858.3304053
[2] https://dl.acm.org/doi/10.1145/3503222.3507731
[3] https://lwn.net/Articles/495543/
[4] https://lwn.net/Articles/815342/
--- fs/fuse/dev.c | 3 +- include/linux/mm.h | 2 + include/linux/mm_inline.h | 180 ++++++++++++++++++++++++++++++ include/linux/mmzone.h | 98 ++++++++++++++++ include/linux/page-flags-layout.h | 13 ++- include/linux/page-flags.h | 4 +- include/linux/sched.h | 4 + kernel/bounds.c | 5 + mm/Kconfig | 8 ++ mm/huge_memory.c | 3 +- mm/memcontrol.c | 2 + mm/memory.c | 25 +++++ mm/mm_init.c | 6 +- mm/mmzone.c | 2 + mm/swap.c | 9 +- mm/vmscan.c | 75 +++++++++++++ 16 files changed, 426 insertions(+), 13 deletions(-)
fs/fuse/dev.c +2 -1
@@ -777,7 +777,8 @@ static int fuse_check_page(struct page *page) 1 << PG_active | 1 << PG_workingset | 1 << PG_reclaim | - 1 << PG_waiters))) { + 1 << PG_waiters | + LRU_GEN_MASK | LRU_REFS_MASK))) {dump_page (page,"fuse: trying to steal weird page" );return 1 ; }
include/linux/mm.h +2 -0
@@ -1060,6 +1060,8 @@ vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf);#define ZONES_PGOFF (NODES_PGOFF - ZONES_WIDTH)#define LAST_CPUPID_PGOFF (ZONES_PGOFF - LAST_CPUPID_WIDTH)#define KASAN_TAG_PGOFF (LAST_CPUPID_PGOFF - KASAN_TAG_WIDTH) +#define LRU_GEN_PGOFF (KASAN_TAG_PGOFF - LRU_GEN_WIDTH) +#define LRU_REFS_PGOFF (LRU_GEN_PGOFF - LRU_REFS_WIDTH)/* * Define the bit shifts to access each section. For non-existent
include/linux/mm_inline.h +180 -0
209 lines not on this page · raw message
include/linux/mmzone.h +98 -0
110 lines not on this page · raw message
include/linux/page-flags-layout.h +8 -5
@@ -55,7 +55,8 @@#define SECTIONS_WIDTH 0#endif -#if ZONES_WIDTH +SECTIONS_WIDTH +NODES_SHIFT <=BITS_PER_LONG -NR_PAGEFLAGS +#if ZONES_WIDTH +LRU_GEN_WIDTH +SECTIONS_WIDTH +NODES_SHIFT \ + <=BITS_PER_LONG -NR_PAGEFLAGS #define NODES_WIDTH NODES_SHIFT#elif defined(CONFIG_SPARSEMEM_VMEMMAP )#error "Vmemmap: No space for nodes field in page flags"
@@ -89,8 +90,8 @@#define LAST_CPUPID_SHIFT 0#endif -#if ZONES_WIDTH +SECTIONS_WIDTH +NODES_WIDTH +KASAN_TAG_WIDTH +LAST_CPUPID_SHIFT \ - <=BITS_PER_LONG -NR_PAGEFLAGS +#if ZONES_WIDTH +LRU_GEN_WIDTH +SECTIONS_WIDTH +NODES_WIDTH + \ +KASAN_TAG_WIDTH +LAST_CPUPID_SHIFT <=BITS_PER_LONG -NR_PAGEFLAGS #define LAST_CPUPID_WIDTH LAST_CPUPID_SHIFT#else #define LAST_CPUPID_WIDTH 0
@@ -100,10 +101,12 @@#define LAST_CPUPID_NOT_IN_PAGE_FLAGS #endif -#if ZONES_WIDTH +SECTIONS_WIDTH +NODES_WIDTH +KASAN_TAG_WIDTH +LAST_CPUPID_WIDTH \ - >BITS_PER_LONG -NR_PAGEFLAGS +#if ZONES_WIDTH +LRU_GEN_WIDTH +SECTIONS_WIDTH +NODES_WIDTH + \ +KASAN_TAG_WIDTH +LAST_CPUPID_WIDTH >BITS_PER_LONG -NR_PAGEFLAGS #error "Not enough bits in page flags"#endif +#define LRU_REFS_WIDTH 0 +#endif #endif /* _LINUX_PAGE_FLAGS_LAYOUT */
include/linux/page-flags.h +2 -2
@@ -1017,7 +1017,7 @@ PAGEFLAG(Isolated, isolated, PF_ANY);1UL << PG_private |1UL << PG_private_2 | \1UL << PG_writeback |1UL << PG_reserved | \1UL << PG_slab |1UL << PG_active | \ - 1UL << PG_unevictable | __PG_MLOCKED) + 1UL << PG_unevictable | __PG_MLOCKED | LRU_GEN_MASK)/* * Flags checked when a page is prepped for return by the page allocator.
@@ -1028,7 +1028,7 @@ PAGEFLAG(Isolated, isolated, PF_ANY);* alloc-free cycle to prevent from reusing the page. */ #define PAGE_FLAGS_CHECK_AT_PREP \ - (PAGEFLAGS_MASK & ~__PG_HWPOISON) + ((PAGEFLAGS_MASK & ~__PG_HWPOISON) | LRU_GEN_MASK | LRU_REFS_MASK)#define PAGE_FLAGS_PRIVATE \ (1UL << PG_private | 1UL << PG_private_2)
include/linux/sched.h +4 -0
@@ -914,6 +914,10 @@ struct task_struct {#ifdef CONFIG_MEMCG unsigned in_user_fault:1;#endif +#ifdef CONFIG_LRU_GEN +/* whether the LRU algorithm may apply to this access */ + unsigned in_lru_fault:1; +#endif #ifdef CONFIG_COMPAT_BRK unsigned brk_randomized:1;#endif
kernel/bounds.c +5 -0
@@ -22,6 +22,11 @@ int main(void)DEFINE (NR_CPUS_BITS ,ilog2 (CONFIG_NR_CPUS ));#endif DEFINE (SPINLOCK_SIZE ,sizeof (spinlock_t)); +#ifdef CONFIG_LRU_GEN +DEFINE (LRU_GEN_WIDTH ,order_base_2 (MAX_NR_GENS +1 )); +#else +DEFINE (LRU_GEN_WIDTH ,0 ); +#endif /* End of constants */ return 0 ;
mm/Kconfig +8 -0
@@ -909,6 +909,14 @@ config ANON_VMA_NAME area from being merged with adjacent virtual memory areas due to the difference in their name. +config LRU_GEN + bool "Multi-Gen LRU" + depends on MMU + # make sure folio->flags has enough spare bits + depends on 64BIT || !SPARSEMEM || SPARSEMEM_VMEMMAP + help + A high performance LRU implementation to overcommit memory. + source "mm/damon/Kconfig" endmenu
mm/huge_memory.c +2 -1
@@ -2320,7 +2320,8 @@ static void __split_huge_page_tail(struct page *head, int tail,#ifdef CONFIG_64BIT (1L << PG_arch_2) | #endif - (1L << PG_dirty))); + (1L << PG_dirty) | + LRU_GEN_MASK | LRU_REFS_MASK));/* ->mapping in first tail page is compound_mapcount */ VM_BUG_ON_PAGE(tail > 2 && page_tail->mapping != TAIL_MAPPING,
mm/memcontrol.c +2 -0
@@ -5072,6 +5072,7 @@ static void __mem_cgroup_free(struct mem_cgroup *memcg)static void mem_cgroup_free (struct mem_cgroup *memcg) { +lru_gen_exit_memcg (memcg);memcg_wb_domain_exit (memcg);__mem_cgroup_free (memcg); }
@@ -5130,6 +5131,7 @@ static struct mem_cgroup *mem_cgroup_alloc(void) memcg->deferred_split_queue.split_queue_len =0 ;#endif idr_replace (&mem_cgroup_idr, memcg, memcg->id.id); +lru_gen_init_memcg (memcg);return memcg; fail:mem_cgroup_id_remove (memcg);
mm/memory.c +25 -0
@@ -4812,6 +4812,27 @@ static inline void mm_account_fault(struct pt_regs *regs,perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MIN ,1 , regs, address); } +#ifdef CONFIG_LRU_GEN +static void lru_gen_enter_fault (struct vm_area_struct *vma) +{ +/* the LRU algorithm doesn't apply to sequential or random reads */ + current->in_lru_fault = !(vma->vm_flags & (VM_SEQ_READ |VM_RAND_READ )); +} + +static void lru_gen_exit_fault (void ) +{ + current->in_lru_fault = false; +} +#else +static void lru_gen_enter_fault (struct vm_area_struct *vma) +{ +} + +static void lru_gen_exit_fault (void ) +{ +} +#endif /* CONFIG_LRU_GEN */ +/* * By the time we get here, we already hold the mm semaphore *
@@ -4843,11 +4864,15 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,if (flags &FAULT_FLAG_USER )mem_cgroup_enter_user_fault (); +lru_gen_enter_fault (vma); +if (unlikely (is_vm_hugetlb_page (vma))) ret =hugetlb_fault (vma->vm_mm, vma, address, flags);else ret =__handle_mm_fault (vma, address, flags); +lru_gen_exit_fault (); +if (flags &FAULT_FLAG_USER ) {mem_cgroup_exit_user_fault ();/*
mm/mm_init.c +4 -2
@@ -65,14 +65,16 @@ void __init mminit_verify_pageflags_layout(void) shift =8 *sizeof (unsigned long ); width = shift -SECTIONS_WIDTH -NODES_WIDTH -ZONES_WIDTH - -LAST_CPUPID_SHIFT -KASAN_TAG_WIDTH ; + -LAST_CPUPID_SHIFT -KASAN_TAG_WIDTH -LRU_GEN_WIDTH -LRU_REFS_WIDTH ;mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_widths" , -"Section %d Node %d Zone %d Lastcpupid %d Kasantag %d Flags %d\n" , +"Section %d Node %d Zone %d Lastcpupid %d Kasantag %d Gen %d Tier %d Flags %d\n" ,SECTIONS_WIDTH ,NODES_WIDTH ,ZONES_WIDTH ,LAST_CPUPID_WIDTH ,KASAN_TAG_WIDTH , +LRU_GEN_WIDTH , +LRU_REFS_WIDTH ,NR_PAGEFLAGS ); mminit_dprintk(MMINIT_TRACE, "pageflags_layout_shifts", "Section %d Node %d Zone %d Lastcpupid %d Kasantag %d\n",
mm/mmzone.c +2 -0
@@ -88,6 +88,8 @@ void lruvec_init(struct lruvec *lruvec)* Poison its list head, so that any operations on it would crash. */ list_del (&lruvec->lists[LRU_UNEVICTABLE ]); + +lru_gen_init_lruvec (lruvec); }#if defined(CONFIG_NUMA_BALANCING ) && !defined(LAST_CPUPID_NOT_IN_PAGE_FLAGS )
mm/swap.c +7 -2
@@ -460,6 +460,11 @@ void folio_add_lru(struct folio *folio)VM_BUG_ON_FOLIO (folio_test_active (folio) &&folio_test_unevictable (folio), folio);VM_BUG_ON_FOLIO (folio_test_lru (folio), folio); +/* see the comment in lru_gen_add_folio() */ +if (lru_gen_enabled () && !folio_test_unevictable (folio) && +lru_gen_in_fault () && !(current->flags &PF_MEMALLOC )) +folio_set_active (folio); +folio_get (folio);local_lock (&lru_pvecs.lock); pvec =this_cpu_ptr (&lru_pvecs.lru_add);
@@ -551,7 +556,7 @@ static void lru_deactivate_file_fn(struct page *page, struct lruvec *lruvec)static void lru_deactivate_fn (struct page *page,struct lruvec *lruvec) { -if (PageActive (page) && !PageUnevictable (page)) { +if (!PageUnevictable (page) && (PageActive (page) ||lru_gen_enabled ())) {int nr_pages =thp_nr_pages (page);del_page_from_lru_list (page, lruvec);
@@ -666,7 +671,7 @@ void deactivate_file_folio(struct folio *folio)*/ void deactivate_page (struct page *page) { -if (PageLRU (page) &&PageActive (page) && !PageUnevictable (page)) { +if (PageLRU (page) && !PageUnevictable (page) && (PageActive (page) ||lru_gen_enabled ())) {struct pagevec *pvec;local_lock (&lru_pvecs.lock);
mm/vmscan.c +75 -0
81 lines not on this page · raw message
Signature · 1 line
2.36.0.550.gb090851708-goog
Re: [PATCH v11 05/14] mm: multi-gen LRU: groundwork
Quoted 258 lines · Yu Zhao
Evictable pages are divided into multiple generations for each lruvec. The youngest generation number is stored in
lrugen->max_seqfor both anon and file types as they are aged on an equal footing. The oldest generation numbers are stored in lrugen->min_seq[] separately for anon and file types as clean file pages can be evicted regardless of swap constraints. These three variables are monotonically increasing.Generation numbers are truncated into
order_base_2(MAX_NR_GENS+1)bits in order to fit into the gen counter infolio->flags. Each truncated generation number is an index to lrugen->lists[]. The sliding window technique is used to track at leastMIN_NR_GENSand at mostMAX_NR_GENSgenerations. The gen counter stores a value within [1,MAX_NR_GENS] while a page is on one of lrugen->lists[]. Otherwise it stores 0.There are two conceptually independent procedures: "the aging", which produces young generations, and "the eviction", which consumes old generations. They form a closed-loop system, i.e., "the page reclaim". Both procedures can be invoked from userspace for the purposes of working set estimation and proactive reclaim. These techniques are commonly used to optimize job scheduling (bin packing) in data centers [1][2].
To avoid confusion, the terms "hot" and "cold" will be applied to the multi-gen LRU, as a new convention; the terms "active" and "inactive" will be applied to the active/inactive LRU, as usual.
The protection of hot pages and the selection of cold pages are based on page access channels and patterns. There are two access channels:
one through page tables and the other through file descriptors. The protection of the former channel is by design stronger because:
1. The uncertainty in determining the access patterns of the former channel is higher due to the approximation of the accessed bit.
2. The cost of evicting the former channel is higher due to the TLB flushes required and the likelihood of encountering the dirty bit.
3. The penalty of underprotecting the former channel is higher because applications usually do not prepare themselves for major page faults like they do for blocked I/O. E.g., GUI applications commonly use dedicated I/O threads to avoid blocking the rendering threads.
There are also two access patterns: one with temporal locality and the other without. For the reasons listed above, the former channel is assumed to follow the former pattern unlessVM_SEQ_READorVM_RAND_READis present; the latter channel is assumed to follow the latter pattern unless outlying refaults have been observed [3][4].The next patch will address the "outlying refaults". Three macros, i.e.,
LRU_REFS_WIDTH,LRU_REFS_PGOFFandLRU_REFS_MASK, used later are added in this patch to make the entire patchset less diffy.A page is added to the youngest generation on faulting. The aging needs to check the accessed bit at least twice before handing this page over to the eviction. The first check takes care of the accessed bit set on the initial fault; the second check makes sure this page has not been used since then. This protocol, AKA second chance, requires a minimum of two generations, hence
MIN_NR_GENS.[1] https://dl.acm.org/doi/10.1145/3297858.3304053
[2] https://dl.acm.org/doi/10.1145/3503222.3507731
[3] https://lwn.net/Articles/495543/
[4] https://lwn.net/Articles/815342/Signed-off-by Yu Zhao <[email protected]>Acked-by Brian Geffon <[email protected]>Acked-by Steven Barrett <[email protected]>Acked-by Suleiman Souhlal <[email protected]>Tested-by Daniel Byrne <[email protected]>Tested-by Donald Carr <[email protected]>Tested-by Holger Hoffstätte <[email protected]>Tested-by Konstantin Kharlamov <[email protected]>Tested-by Shuang Zhai <[email protected]>Tested-by Sofia Trinh <[email protected]>Tested-by Vaibhav Jain <[email protected]>--- fs/fuse/dev.c | 3 +- include/linux/mm.h | 2 + include/linux/mm_inline.h | 180 ++++++++++++++++++++++++++++++ include/linux/mmzone.h | 98 ++++++++++++++++ include/linux/page-flags-layout.h | 13 ++- include/linux/page-flags.h | 4 +- include/linux/sched.h | 4 + kernel/bounds.c | 5 + mm/Kconfig | 8 ++ mm/huge_memory.c | 3 +- mm/memcontrol.c | 2 + mm/memory.c | 25 +++++ mm/mm_init.c | 6 +- mm/mmzone.c | 2 + mm/swap.c | 9 +- mm/vmscan.c | 75 +++++++++++++ 16 files changed, 426 insertions(+), 13 deletions(-)fs/fuse/dev.c +2
-1@@ -777,7 +777,8 @@ static int fuse_check_page(struct page *page) 1 << PG_active | 1 << PG_workingset | 1 << PG_reclaim | - 1 << PG_waiters))) { + 1 << PG_waiters | + LRU_GEN_MASK | LRU_REFS_MASK))) {dump_page (page,"fuse: trying to steal weird page" );return 1 ; }include/linux/mm.h +2
-0@@ -1060,6 +1060,8 @@ vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf);#define ZONES_PGOFF (NODES_PGOFF - ZONES_WIDTH)#define LAST_CPUPID_PGOFF (ZONES_PGOFF - LAST_CPUPID_WIDTH)#define KASAN_TAG_PGOFF (LAST_CPUPID_PGOFF - KASAN_TAG_WIDTH) +#define LRU_GEN_PGOFF (KASAN_TAG_PGOFF - LRU_GEN_WIDTH) +#define LRU_REFS_PGOFF (LRU_GEN_PGOFF - LRU_REFS_WIDTH)/* * Define the bit shifts to access each section. For non-existent include/linux/mm_inline.h +122
-0@@ -38,6 +38,9 @@ static __always_inline void __update_lru_size(struct lruvec *lruvec, {struct pglist_data *pgdat =lruvec_pgdat (lruvec); +lockdep_assert_held (&lruvec->lru_lock); +WARN_ON_ONCE (nr_pages != (int )nr_pages); +__mod_lruvec_state (lruvec,NR_LRU_BASE + lru, nr_pages);__mod_zone_page_state (&pgdat->node_zones[zid],NR_ZONE_LRU_BASE + lru, nr_pages);@@ -99,11 +102,182 @@ static __always_inline enum lru_list folio_lru_list(struct folio *folio)return lru; } +#ifdef CONFIG_LRU_GEN + +static inline bool lru_gen_enabled (void ) +{ +return true; +} + +static inline bool lru_gen_in_fault (void ) +{ +return current->in_lru_fault; +} + +static inline int lru_gen_from_seq (unsigned long seq) +{ +return seq %MAX_NR_GENS ; +} + +static inline int folio_lru_gen (struct folio *folio) +{ +unsigned long flags =READ_ONCE (folio->flags); + +return ((flags &LRU_GEN_MASK ) >>LRU_GEN_PGOFF ) -1 ; +} + +static inline bool lru_gen_is_active (struct lruvec *lruvec,int gen) +{ +unsigned long max_seq = lruvec->lrugen.max_seq; + +VM_WARN_ON_ONCE (gen >=MAX_NR_GENS ); + +/* see the comment on MIN_NR_GENS */ +return gen ==lru_gen_from_seq (max_seq) || gen ==lru_gen_from_seq (max_seq -1 ); +} + +static inline void lru_gen_update_size (struct lruvec *lruvec,struct folio *folio, +int old_gen,int new_gen) +{ +int type =folio_is_file_lru (folio); +int zone =folio_zonenum (folio); +int delta =folio_nr_pages (folio); +enum lru_list lru = type *LRU_INACTIVE_FILE ; +struct lru_gen_struct *lrugen = &lruvec->lrugen; + +VM_WARN_ON_ONCE (old_gen !=-1 && old_gen >=MAX_NR_GENS ); +VM_WARN_ON_ONCE (new_gen !=-1 && new_gen >=MAX_NR_GENS ); +VM_WARN_ON_ONCE (old_gen ==-1 && new_gen ==-1 ); + +if (old_gen >=0 ) +WRITE_ONCE (lrugen->nr_pages[old_gen][type][zone], + lrugen->nr_pages[old_gen][type][zone] - delta); +if (new_gen >=0 ) +WRITE_ONCE (lrugen->nr_pages[new_gen][type][zone], + lrugen->nr_pages[new_gen][type][zone] + delta); + +/* addition */ +if (old_gen <0 ) { +if (lru_gen_is_active (lruvec, new_gen)) + lru +=LRU_ACTIVE ; +__update_lru_size (lruvec, lru, zone, delta); +return ; + } + +/* deletion */ +if (new_gen <0 ) { +if (lru_gen_is_active (lruvec, old_gen)) + lru +=LRU_ACTIVE ; +__update_lru_size (lruvec, lru, zone, -delta); +return ; + } +} + +static inline bool lru_gen_add_folio (struct lruvec *lruvec,struct folio *folio,bool reclaiming) +{ +unsigned long mask, flags; +int gen =folio_lru_gen (folio); +int type =folio_is_file_lru (folio); +int zone =folio_zonenum (folio); +struct lru_gen_struct *lrugen = &lruvec->lrugen; + +VM_WARN_ON_ONCE_FOLIO (gen !=-1 , folio); + +if (folio_test_unevictable (folio)) +return false; +/* +* There are three common cases for this page: +* 1. If it's hot, e.g., freshly faulted in or previously hot and +* migrated, add it to the youngest generation. +* 2. If it's cold but can't be evicted immediately, i.e., an anon page +* not in swapcache or a dirty page pending writeback, add it to the +* second oldest generation. +* 3. Everything else (clean, cold) is added to the oldest generation. +*/ +if (folio_test_active (folio)) + gen =lru_gen_from_seq (lrugen->max_seq); +else if ((type ==LRU_GEN_ANON && !folio_test_swapcache (folio)) || + (folio_test_reclaim (folio) && + (folio_test_dirty (folio) ||folio_test_writeback (folio)))) + gen =lru_gen_from_seq (lrugen->min_seq[type] +1 ); +else + gen =lru_gen_from_seq (lrugen->min_seq[type]); + +/* see the comment on MIN_NR_GENS */ + mask =LRU_GEN_MASK |BIT (PG_active); + flags = (gen +1UL ) <<LRU_GEN_PGOFF ; +set_mask_bits (&folio->flags, mask, flags); + +lru_gen_update_size (lruvec, folio,-1 , gen); +/* for folio_rotate_reclaimable() */ +if (reclaiming) +list_add_tail (&folio->lru, &lrugen->lists[gen][type][zone]); +else +list_add (&folio->lru, &lrugen->lists[gen][type][zone]); + +return true; +} + +static inline bool lru_gen_del_folio (struct lruvec *lruvec,struct folio *folio,bool reclaiming) +{ +unsigned long mask, flags;
include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 233
Quoted 10 lines
include/linux/mm_inline.h +10
-0+int gen =folio_lru_gen (folio); + +if (gen <0 ) +return false; + +VM_WARN_ON_ONCE_FOLIO (folio_test_active (folio), folio); +VM_WARN_ON_ONCE_FOLIO (folio_test_unevictable (folio), folio); + + mask =LRU_GEN_MASK ; + flags =0 ;
It always to keep the reference of page unless folios update the gen, which is the difference between v10 and v11.
I agree that.
include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 242
Quoted 10 lines
include/linux/mm_inline.h +9
-0+/* for shrink_page_list() or folio_migrate_flags() */ +if (reclaiming) + mask |=BIT (PG_referenced) |BIT (PG_reclaim); +else if (lru_gen_is_active (lruvec, gen)) + flags |=BIT (PG_active); + + flags =set_mask_bits (&folio->flags, mask, flags); + gen = ((flags &LRU_GEN_MASK ) >>LRU_GEN_PGOFF ) -1 ; +
the flags has remove the LRU_GEN_MASK by set_mask_bits, hence the gen will always be -1, it will fails
to decrease the specified lru size. Am I missing something?
Quoted 527 lines
include/linux/mm_inline.h +39
-0+lru_gen_update_size (lruvec, folio, gen,-1 ); +list_del (&folio->lru); + +return true; +} + +#else + +static inline bool lru_gen_enabled (void ) +{ +return false; +} + +static inline bool lru_gen_in_fault (void ) +{ +return false; +} + +static inline bool lru_gen_add_folio (struct lruvec *lruvec,struct folio *folio,bool reclaiming) +{ +return false; +} + +static inline bool lru_gen_del_folio (struct lruvec *lruvec,struct folio *folio,bool reclaiming) +{ +return false; +} + +#endif /* CONFIG_LRU_GEN */ +static __always_inline void lruvec_add_folio(struct lruvec *lruvec, struct folio *folio) {enum lru_list lru =folio_lru_list (folio); +if (lru_gen_add_folio (lruvec, folio, false)) +return ; +update_lru_size (lruvec, lru,folio_zonenum (folio),folio_nr_pages (folio));if (lru !=LRU_UNEVICTABLE )@@ -121,6 +295,9 @@ void lruvec_add_folio_tail(struct lruvec *lruvec, struct folio *folio) {enum lru_list lru =folio_lru_list (folio); +if (lru_gen_add_folio (lruvec, folio, true)) +return ; +update_lru_size (lruvec, lru,folio_zonenum (folio),folio_nr_pages (folio));/* This is not expected to be used on LRU_UNEVICTABLE */ @@ -138,6 +315,9 @@ void lruvec_del_folio(struct lruvec *lruvec, struct folio *folio) {enum lru_list lru =folio_lru_list (folio); +if (lru_gen_del_folio (lruvec, folio, false)) +return ; +if (lru !=LRU_UNEVICTABLE )list_del (&folio->lru); update_lru_size(lruvec, lru, folio_zonenum(folio),include/linux/mmzone.h +98
-0@@ -317,6 +317,100 @@ enum lruvec_flags {*/ }; +#endif /* !__GENERATING_BOUNDS_H */ + +/* +* Evictable pages are divided into multiple generations. The youngest and the +* oldest generation numbers, max_seq and min_seq, are monotonically increasing. +* They form a sliding window of a variable size [MIN_NR_GENS, MAX_NR_GENS]. An +* offset within MAX_NR_GENS, i.e., gen, indexes the LRU list of the +* corresponding generation. The gen counter in folio->flags stores gen+1 while +* a page is on one of lrugen->lists[]. Otherwise it stores 0. +* +* A page is added to the youngest generation on faulting. The aging needs to +* check the accessed bit at least twice before handing this page over to the +* eviction. The first check takes care of the accessed bit set on the initial +* fault; the second check makes sure this page hasn't been used since then. +* This process, AKA second chance, requires a minimum of two generations, +* hence MIN_NR_GENS. And to maintain ABI compatibility with the active/inactive +* LRU, e.g., /proc/vmstat, these two generations are considered active; the +* rest of generations, if they exist, are considered inactive. See +* lru_gen_is_active(). PG_active is always cleared while a page is on one of +* lrugen->lists[] so that the aging needs not to worry about it. And it's set +* again when a page considered active is isolated for non-reclaiming purposes, +* e.g., migration. See lru_gen_add_folio() and lru_gen_del_folio(). +* +* MAX_NR_GENS is set to 4 so that the multi-gen LRU can support twice the +* number of categories of the active/inactive LRU when keeping track of +* accesses through page tables. It requires order_base_2(MAX_NR_GENS+1) bits in +* folio->flags (LRU_GEN_MASK). +*/ +#define MIN_NR_GENS 2U +#define MAX_NR_GENS 4U + +#ifndef __GENERATING_BOUNDS_H + +struct lruvec ; + +#define LRU_GEN_MASK ((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF) +#define LRU_REFS_MASK ((BIT(LRU_REFS_WIDTH) - 1) << LRU_REFS_PGOFF) + +#ifdef CONFIG_LRU_GEN + +enum { +LRU_GEN_ANON , +LRU_GEN_FILE , +}; + +/* +* The youngest generation number is stored in max_seq for both anon and file +* types as they are aged on an equal footing. The oldest generation numbers are +* stored in min_seq[] separately for anon and file types as clean file pages +* can be evicted regardless of swap constraints. +* +* Normally anon and file min_seq are in sync. But if swapping is constrained, +* e.g., out of swap space, file min_seq is allowed to advance and leave anon +* min_seq behind. +* +* nr_pages[] are eventually consistent and therefore can be transiently +* negative. +*/ +struct lru_gen_struct { +/* the aging increments the youngest generation number */ +unsigned long max_seq; +/* the eviction increments the oldest generation numbers */ +unsigned long min_seq[ANON_AND_FILE ]; +/* the multi-gen LRU lists */ +struct list_head lists[MAX_NR_GENS ][ANON_AND_FILE ][MAX_NR_ZONES ]; +/* the sizes of the above lists */ +long nr_pages[MAX_NR_GENS ][ANON_AND_FILE ][MAX_NR_ZONES ]; +}; + +void lru_gen_init_lruvec (struct lruvec *lruvec); + +#ifdef CONFIG_MEMCG +void lru_gen_init_memcg (struct mem_cgroup *memcg); +void lru_gen_exit_memcg (struct mem_cgroup *memcg); +#endif + +#else /* !CONFIG_LRU_GEN */ + +static inline void lru_gen_init_lruvec (struct lruvec *lruvec) +{ +} + +#ifdef CONFIG_MEMCG +static inline void lru_gen_init_memcg (struct mem_cgroup *memcg) +{ +} + +static inline void lru_gen_exit_memcg (struct mem_cgroup *memcg) +{ +} +#endif + +#endif /* CONFIG_LRU_GEN */ + struct lruvec { struct list_head lists[NR_LRU_LISTS];/* per lruvec lru_lock for memcg */ @@ -334,6 +428,10 @@ struct lruvec {unsigned long refaults[ANON_AND_FILE ];/* Various lruvec state flags (enum lruvec_flags) */ unsigned long flags; +#ifdef CONFIG_LRU_GEN +/* evictable pages divided into generations */ +struct lru_gen_struct lrugen; +#endif #ifdef CONFIG_MEMCG struct pglist_data *pgdat;#endif include/linux/page-flags-layout.h +8
-5@@ -55,7 +55,8 @@#define SECTIONS_WIDTH 0#endif -#if ZONES_WIDTH +SECTIONS_WIDTH +NODES_SHIFT <=BITS_PER_LONG -NR_PAGEFLAGS +#if ZONES_WIDTH +LRU_GEN_WIDTH +SECTIONS_WIDTH +NODES_SHIFT \ + <=BITS_PER_LONG -NR_PAGEFLAGS #define NODES_WIDTH NODES_SHIFT#elif defined(CONFIG_SPARSEMEM_VMEMMAP )#error "Vmemmap: No space for nodes field in page flags"@@ -89,8 +90,8 @@#define LAST_CPUPID_SHIFT 0#endif -#if ZONES_WIDTH +SECTIONS_WIDTH +NODES_WIDTH +KASAN_TAG_WIDTH +LAST_CPUPID_SHIFT \ - <=BITS_PER_LONG -NR_PAGEFLAGS +#if ZONES_WIDTH +LRU_GEN_WIDTH +SECTIONS_WIDTH +NODES_WIDTH + \ +KASAN_TAG_WIDTH +LAST_CPUPID_SHIFT <=BITS_PER_LONG -NR_PAGEFLAGS #define LAST_CPUPID_WIDTH LAST_CPUPID_SHIFT#else #define LAST_CPUPID_WIDTH 0@@ -100,10 +101,12 @@#define LAST_CPUPID_NOT_IN_PAGE_FLAGS #endif -#if ZONES_WIDTH +SECTIONS_WIDTH +NODES_WIDTH +KASAN_TAG_WIDTH +LAST_CPUPID_WIDTH \ - >BITS_PER_LONG -NR_PAGEFLAGS +#if ZONES_WIDTH +LRU_GEN_WIDTH +SECTIONS_WIDTH +NODES_WIDTH + \ +KASAN_TAG_WIDTH +LAST_CPUPID_WIDTH >BITS_PER_LONG -NR_PAGEFLAGS #error "Not enough bits in page flags"#endif +#define LRU_REFS_WIDTH 0 +#endif #endif /* _LINUX_PAGE_FLAGS_LAYOUT */ include/linux/page-flags.h +2
-2@@ -1017,7 +1017,7 @@ PAGEFLAG(Isolated, isolated, PF_ANY);1UL << PG_private |1UL << PG_private_2 | \1UL << PG_writeback |1UL << PG_reserved | \1UL << PG_slab |1UL << PG_active | \ - 1UL << PG_unevictable | __PG_MLOCKED) + 1UL << PG_unevictable | __PG_MLOCKED | LRU_GEN_MASK)/* * Flags checked when a page is prepped for return by the page allocator. @@ -1028,7 +1028,7 @@ PAGEFLAG(Isolated, isolated, PF_ANY);* alloc-free cycle to prevent from reusing the page. */ #define PAGE_FLAGS_CHECK_AT_PREP \ - (PAGEFLAGS_MASK & ~__PG_HWPOISON) + ((PAGEFLAGS_MASK & ~__PG_HWPOISON) | LRU_GEN_MASK | LRU_REFS_MASK)#define PAGE_FLAGS_PRIVATE \ (1UL << PG_private | 1UL << PG_private_2)include/linux/sched.h +4
-0@@ -914,6 +914,10 @@ struct task_struct {#ifdef CONFIG_MEMCG unsigned in_user_fault:1;#endif +#ifdef CONFIG_LRU_GEN +/* whether the LRU algorithm may apply to this access */ + unsigned in_lru_fault:1; +#endif #ifdef CONFIG_COMPAT_BRK unsigned brk_randomized:1;#endif kernel/bounds.c +5
-0@@ -22,6 +22,11 @@ int main(void)DEFINE (NR_CPUS_BITS ,ilog2 (CONFIG_NR_CPUS ));#endif DEFINE (SPINLOCK_SIZE ,sizeof (spinlock_t)); +#ifdef CONFIG_LRU_GEN +DEFINE (LRU_GEN_WIDTH ,order_base_2 (MAX_NR_GENS +1 )); +#else +DEFINE (LRU_GEN_WIDTH ,0 ); +#endif /* End of constants */ return 0 ;mm/Kconfig +8
-0@@ -909,6 +909,14 @@ config ANON_VMA_NAME area from being merged with adjacent virtual memory areas due to the difference in their name. +config LRU_GEN + bool "Multi-Gen LRU" + depends on MMU + # make sure folio->flags has enough spare bits + depends on 64BIT || !SPARSEMEM || SPARSEMEM_VMEMMAP + help + A high performance LRU implementation to overcommit memory. + source "mm/damon/Kconfig" endmenumm/huge_memory.c +2
-1@@ -2320,7 +2320,8 @@ static void __split_huge_page_tail(struct page *head, int tail,#ifdef CONFIG_64BIT (1L << PG_arch_2) | #endif - (1L << PG_dirty))); + (1L << PG_dirty) | + LRU_GEN_MASK | LRU_REFS_MASK));/* ->mapping in first tail page is compound_mapcount */ VM_BUG_ON_PAGE(tail > 2 && page_tail->mapping != TAIL_MAPPING,mm/memcontrol.c +2
-0@@ -5072,6 +5072,7 @@ static void __mem_cgroup_free(struct mem_cgroup *memcg)static void mem_cgroup_free (struct mem_cgroup *memcg) { +lru_gen_exit_memcg (memcg);memcg_wb_domain_exit (memcg);__mem_cgroup_free (memcg); }@@ -5130,6 +5131,7 @@ static struct mem_cgroup *mem_cgroup_alloc(void) memcg->deferred_split_queue.split_queue_len =0 ;#endif idr_replace (&mem_cgroup_idr, memcg, memcg->id.id); +lru_gen_init_memcg (memcg);return memcg; fail:mem_cgroup_id_remove (memcg);mm/memory.c +25
-0@@ -4812,6 +4812,27 @@ static inline void mm_account_fault(struct pt_regs *regs,perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MIN ,1 , regs, address); } +#ifdef CONFIG_LRU_GEN +static void lru_gen_enter_fault (struct vm_area_struct *vma) +{ +/* the LRU algorithm doesn't apply to sequential or random reads */ + current->in_lru_fault = !(vma->vm_flags & (VM_SEQ_READ |VM_RAND_READ )); +} + +static void lru_gen_exit_fault (void ) +{ + current->in_lru_fault = false; +} +#else +static void lru_gen_enter_fault (struct vm_area_struct *vma) +{ +} + +static void lru_gen_exit_fault (void ) +{ +} +#endif /* CONFIG_LRU_GEN */ +/* * By the time we get here, we already hold the mm semaphore * @@ -4843,11 +4864,15 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,if (flags &FAULT_FLAG_USER )mem_cgroup_enter_user_fault (); +lru_gen_enter_fault (vma); +if (unlikely (is_vm_hugetlb_page (vma))) ret =hugetlb_fault (vma->vm_mm, vma, address, flags);else ret =__handle_mm_fault (vma, address, flags); +lru_gen_exit_fault (); +if (flags &FAULT_FLAG_USER ) {mem_cgroup_exit_user_fault ();/* mm/mm_init.c +4
-2@@ -65,14 +65,16 @@ void __init mminit_verify_pageflags_layout(void) shift =8 *sizeof (unsigned long ); width = shift -SECTIONS_WIDTH -NODES_WIDTH -ZONES_WIDTH - -LAST_CPUPID_SHIFT -KASAN_TAG_WIDTH ; + -LAST_CPUPID_SHIFT -KASAN_TAG_WIDTH -LRU_GEN_WIDTH -LRU_REFS_WIDTH ;mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_widths" , -"Section %d Node %d Zone %d Lastcpupid %d Kasantag %d Flags %d\n" , +"Section %d Node %d Zone %d Lastcpupid %d Kasantag %d Gen %d Tier %d Flags %d\n" ,SECTIONS_WIDTH ,NODES_WIDTH ,ZONES_WIDTH ,LAST_CPUPID_WIDTH ,KASAN_TAG_WIDTH , +LRU_GEN_WIDTH , +LRU_REFS_WIDTH ,NR_PAGEFLAGS ); mminit_dprintk(MMINIT_TRACE, "pageflags_layout_shifts", "Section %d Node %d Zone %d Lastcpupid %d Kasantag %d\n",mm/mmzone.c +2
-0@@ -88,6 +88,8 @@ void lruvec_init(struct lruvec *lruvec)* Poison its list head, so that any operations on it would crash. */ list_del (&lruvec->lists[LRU_UNEVICTABLE ]); + +lru_gen_init_lruvec (lruvec); }#if defined(CONFIG_NUMA_BALANCING ) && !defined(LAST_CPUPID_NOT_IN_PAGE_FLAGS )mm/swap.c +7
-2@@ -460,6 +460,11 @@ void folio_add_lru(struct folio *folio)VM_BUG_ON_FOLIO (folio_test_active (folio) &&folio_test_unevictable (folio), folio);VM_BUG_ON_FOLIO (folio_test_lru (folio), folio); +/* see the comment in lru_gen_add_folio() */ +if (lru_gen_enabled () && !folio_test_unevictable (folio) && +lru_gen_in_fault () && !(current->flags &PF_MEMALLOC )) +folio_set_active (folio); +folio_get (folio);local_lock (&lru_pvecs.lock); pvec =this_cpu_ptr (&lru_pvecs.lru_add);@@ -551,7 +556,7 @@ static void lru_deactivate_file_fn(struct page *page, struct lruvec *lruvec)static void lru_deactivate_fn (struct page *page,struct lruvec *lruvec) { -if (PageActive (page) && !PageUnevictable (page)) { +if (!PageUnevictable (page) && (PageActive (page) ||lru_gen_enabled ())) {int nr_pages =thp_nr_pages (page);del_page_from_lru_list (page, lruvec);@@ -666,7 +671,7 @@ void deactivate_file_folio(struct folio *folio)*/ void deactivate_page (struct page *page) { -if (PageLRU (page) &&PageActive (page) && !PageUnevictable (page)) { +if (PageLRU (page) && !PageUnevictable (page) && (PageActive (page) ||lru_gen_enabled ())) {struct pagevec *pvec;local_lock (&lru_pvecs.lock);mm/vmscan.c +75
-0@@ -2968,6 +2968,81 @@ static bool can_age_anon_pages(struct pglist_data *pgdat,return can_demote (pgdat->node_id, sc); } +#ifdef CONFIG_LRU_GEN + +/****************************************************************************** +* shorthand helpers +******************************************************************************/ + +#define for_each_gen_type_zone (gen, type, zone) \ + for ((gen) = 0; (gen) < MAX_NR_GENS; (gen)++) \ + for ((type) = 0; (type) < ANON_AND_FILE; (type)++) \ + for ((zone) = 0; (zone) < MAX_NR_ZONES; (zone)++) + +static struct lruvec __maybe_unused *get_lruvec (struct mem_cgroup *memcg,int nid) +{ +struct pglist_data *pgdat =NODE_DATA (nid); + +#ifdef CONFIG_MEMCG +if (memcg) { +struct lruvec *lruvec = &memcg->nodeinfo[nid]->lruvec; + +/* for hotadd_new_pgdat() */ +if (!lruvec->pgdat) + lruvec->pgdat = pgdat; + +return lruvec; + } +#endif +VM_WARN_ON_ONCE (!mem_cgroup_disabled ()); + +return pgdat ? &pgdat->__lruvec :NULL ; +} + +/****************************************************************************** +* initialization +******************************************************************************/ + +void lru_gen_init_lruvec (struct lruvec *lruvec) +{ +int gen, type, zone; +struct lru_gen_struct *lrugen = &lruvec->lrugen; + + lrugen->max_seq =MIN_NR_GENS +1 ; + +for_each_gen_type_zone (gen, type, zone) +INIT_LIST_HEAD (&lrugen->lists[gen][type][zone]); +} + +#ifdef CONFIG_MEMCG +void lru_gen_init_memcg (struct mem_cgroup *memcg) +{ +} + +void lru_gen_exit_memcg (struct mem_cgroup *memcg) +{ +int nid; + +for_each_node (nid) { +struct lruvec *lruvec =get_lruvec (memcg, nid); + +VM_WARN_ON_ONCE (memchr_inv (lruvec->lrugen.nr_pages,0 , +sizeof (lruvec->lrugen.nr_pages))); + } +} +#endif + +static int __init init_lru_gen (void ) +{ +BUILD_BUG_ON (MIN_NR_GENS +1 >=MAX_NR_GENS ); +BUILD_BUG_ON (BIT (LRU_GEN_WIDTH ) <=MAX_NR_GENS ); + +return 0 ; +}; +late_initcall (init_lru_gen); + +#endif /* CONFIG_LRU_GEN */ +static void shrink_lruvec (struct lruvec *lruvec,struct scan_control *sc) {unsigned long nr[NR_LRU_LISTS ];
PATCH v11 6/14 mm: multi-gen LRU: minimal implementation
In reply to Yu ZhaoTo avoid confusion, the terms "promotion" and "demotion" will be applied to the multi-gen LRU, as a new convention; the terms "activation" and "deactivation" will be applied to the active/inactive LRU, as usual.
The aging produces young generations. Given an lruvec, it increments max_seq when max_seq-min_seq+1 approaches MIN_NR_GENS. The aging promotes hot pages to the youngest generation when it finds them accessed through page tables; the demotion of cold pages happens consequently when it increments max_seq. The aging has the complexity O(nr_hot_pages), since it is only interested in hot pages. Promotion in the aging path does not involve any LRU list operations, only the updates of the gen counter and lrugen->nr_pages[]; demotion, unless as the result of the increment of max_seq, requires LRU list operations, e.g., lru_deactivate_fn().
The eviction consumes old generations. Given an lruvec, it increments min_seq when the lists indexed by min_seq%MAX_NR_GENS become empty. A feedback loop modeled after the PID controller monitors refaults over anon and file types and decides which type to evict when both types are available from the same generation.
Each generation is divided into multiple tiers. Tiers represent different ranges of numbers of accesses through file descriptors. A page accessed N times through file descriptors is in tier order_base_2(N). Tiers do not have dedicated lrugen->lists[], only bits in folio->flags. In contrast to moving across generations, which requires the LRU lock, moving across tiers only involves operations on folio->flags. The feedback loop also monitors refaults over all tiers and decides when to protect pages in which tiers (N>1), using the first tier (N=0,1) as a baseline. The first tier contains single-use unmapped clean pages, which are most likely the best choices. The eviction moves a page to the next generation, i.e., min_seq+1, if the feedback loop decides so. This approach has the following advantages:
1. It removes the cost of activation in the buffered access path by inferring whether pages accessed multiple times through file descriptors are statistically hot and thus worth protecting in the eviction path.
2. It takes pages accessed through page tables into account and avoids overprotecting pages accessed multiple times through file descriptors. (Pages accessed through page tables are in the first tier, since N=0.)
3. More tiers provide better protection for pages accessed more than twice through file descriptors, when under heavy buffered I/O workloads.
Server benchmark results:
Single workload:
fio (buffered I/O): +[40, 42]%
IOPS BW
5.18-rc1: 2463k 9621MiB/s
patch1-6: 3484k 13.3GiB/sSingle workload:
memcached (anon): +[44, 46]%
Ops/sec KB/sec
5.18-rc1: 771403.27 30004.17
patch1-6: 1120643.70 43588.06Configurations:
CPU: two Xeon 6154
Mem: total 256G
Node 1 was only used as a ram disk to reduce the variance in the
results.
patch drivers/block/brd.c <<EOF
99,100c99,100
< gfp_flags = GFP_NOIO | __GFP_ZERO | __GFP_HIGHMEM;
< page = alloc_page(gfp_flags);
---
> gfp_flags = GFP_NOIO | __GFP_ZERO | __GFP_HIGHMEM | __GFP_THISNODE;
> page = alloc_pages_node(1, gfp_flags, 0);
EOF
cat >>/etc/systemd/system.conf <<EOF
CPUAffinity=numa
NUMAPolicy=bind
NUMAMask=0
EOF
cat >>/etc/memcached.conf <<EOF
-m 184320
-s /var/run/memcached/memcached.sock
-a 0766
-t 36
-B binary
EOF
cat fio.sh
modprobe brd rd_nr=1 rd_size=113246208
swapoff -a
mkfs.ext4 /dev/ram0
mount -t ext4 /dev/ram0 /mnt
mkdir /sys/fs/cgroup/user.slice/test
echo 38654705664 >/sys/fs/cgroup/user.slice/test/memory.max
echo $$ >/sys/fs/cgroup/user.slice/test/cgroup.procs
fio -name=mglru --numjobs=72 --directory=/mnt --size=1408m \
--buffered=1 --ioengine=io_uring --iodepth=128 \
--iodepth_batch_submit=32 --iodepth_batch_complete=32 \
--rw=randread --random_distribution=random --norandommap \
--time_based --ramp_time=10m --runtime=5m --group_reporting
cat memcached.sh
modprobe brd rd_nr=1 rd_size=113246208
swapoff -a
mkswap /dev/ram0
swapon /dev/ram0
memtier_benchmark -S /var/run/memcached/memcached.sock \
-P memcache_binary -n allkeys --key-minimum=1 \
--key-maximum=65000000 --key-pattern=P:P -c 1 -t 36 \
--ratio 1:0 --pipeline 8 -d 2000
memtier_benchmark -S /var/run/memcached/memcached.sock \
-P memcache_binary -n allkeys --key-minimum=1 \
--key-maximum=65000000 --key-pattern=R:R -c 1 -t 36 \
--ratio 0:1 --pipeline 8 --randomize --distinct-client-seedClient benchmark results:
kswapd profiles:
5.18-rc1
40.53% page_vma_mapped_walk
20.37% lzo1x_1_do_compress (real work)
6.99% do_raw_spin_lock
3.93% _raw_spin_unlock_irq
2.08% vma_interval_tree_subtree_search
2.06% vma_interval_tree_iter_next
1.95% folio_referenced_one
1.93% anon_vma_interval_tree_iter_first
1.51% ptep_clear_flush
1.35% __anon_vma_interval_tree_subtree_search
patch1-6
35.99% lzo1x_1_do_compress (real work)
19.40% page_vma_mapped_walk
6.31% _raw_spin_unlock_irq
3.95% do_raw_spin_lock
2.39% anon_vma_interval_tree_iter_first
2.25% ptep_clear_flush
1.92% __anon_vma_interval_tree_subtree_search
1.70% folio_referenced_one
1.68% __zram_bvec_write
1.43% anon_vma_interval_tree_iter_next Configurations:
CPU: single Snapdragon 7c
Mem: total 4G
Chrome OS MemoryPressure [1]
[1] https://chromium.googlesource.com/chromiumos/platform/tast-tests/
--- include/linux/mm_inline.h | 36 ++ include/linux/mmzone.h | 42 ++ include/linux/page-flags-layout.h | 5 +- kernel/bounds.c | 2 + mm/Kconfig | 11 + mm/swap.c | 39 ++ mm/vmscan.c | 799 +++++++++++++++++++++++++++++- mm/workingset.c | 110 +++- 8 files changed, 1034 insertions(+), 10 deletions(-)
include/linux/mm_inline.h +36 -0
@@ -119,6 +119,33 @@ static inline int lru_gen_from_seq(unsigned long seq)return seq %MAX_NR_GENS ; } +static inline int lru_hist_from_seq (unsigned long seq) +{ +return seq %NR_HIST_GENS ; +} + +static inline int lru_tier_from_refs (int refs) +{ +VM_WARN_ON_ONCE (refs >BIT (LRU_REFS_WIDTH )); + +/* see the comment in folio_lru_refs() */ +return order_base_2 (refs +1 ); +} + +static inline int folio_lru_refs (struct folio *folio) +{ +unsigned long flags =READ_ONCE (folio->flags); +bool workingset = flags &BIT (PG_workingset); + +/* +* Return the number of accesses beyond PG_referenced, i.e., N-1 if the +* total number of accesses is N>1, since N=0,1 both map to the first +* tier. lru_tier_from_refs() will account for this off-by-one. Also see +* the comment on MAX_NR_TIERS. +*/ +return ((flags &LRU_REFS_MASK ) >>LRU_REFS_PGOFF ) + workingset; +} +static inline int folio_lru_gen (struct folio *folio) {unsigned long flags =READ_ONCE (folio->flags);
@@ -171,6 +198,15 @@ static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *foli__update_lru_size (lruvec, lru, zone, -delta);return ; } + +/* promotion */ +if (!lru_gen_is_active (lruvec, old_gen) &&lru_gen_is_active (lruvec, new_gen)) { +__update_lru_size (lruvec, lru, zone, -delta); +__update_lru_size (lruvec, lru +LRU_ACTIVE , zone, delta); + } + +/* demotion requires isolation, e.g., lru_deactivate_fn() */ +VM_WARN_ON_ONCE (lru_gen_is_active (lruvec, old_gen) && !lru_gen_is_active (lruvec, new_gen)); }static inline bool lru_gen_add_folio (struct lruvec *lruvec,struct folio *folio,bool reclaiming)
include/linux/mmzone.h +42 -0
60 lines not on this page · raw message
include/linux/page-flags-layout.h +4 -1
@@ -106,7 +106,10 @@#error "Not enough bits in page flags"#endif -#define LRU_REFS_WIDTH 0 +/* see the comment on MAX_NR_TIERS */ +#define LRU_REFS_WIDTH min(__LRU_REFS_WIDTH, BITS_PER_LONG - NR_PAGEFLAGS - \ + ZONES_WIDTH - LRU_GEN_WIDTH - SECTIONS_WIDTH - \ + NODES_WIDTH - KASAN_TAG_WIDTH - LAST_CPUPID_WIDTH)#endif #endif /* _LINUX_PAGE_FLAGS_LAYOUT */
kernel/bounds.c +2 -0
@@ -24,8 +24,10 @@ int main(void)DEFINE (SPINLOCK_SIZE ,sizeof (spinlock_t));#ifdef CONFIG_LRU_GEN DEFINE (LRU_GEN_WIDTH ,order_base_2 (MAX_NR_GENS +1 )); +DEFINE (__LRU_REFS_WIDTH,MAX_NR_TIERS -2 );#else DEFINE (LRU_GEN_WIDTH ,0 ); +DEFINE (__LRU_REFS_WIDTH,0 );#endif /* End of constants */
mm/Kconfig +11 -0
@@ -909,6 +909,7 @@ config ANON_VMA_NAME area from being merged with adjacent virtual memory areas due to the difference in their name. +# multi-gen LRU { config LRU_GEN bool "Multi-Gen LRU" depends on MMU
@@ -917,6 +918,16 @@ config LRU_GEN help A high performance LRU implementation to overcommit memory. +config LRU_GEN_STATS + bool "Full stats for debugging" + depends on LRU_GEN + help + Do not enable this option unless you plan to look at historical stats + from evicted generations for debugging purpose. + + This option has a per-memcg and per-node memory overhead. +# } + source "mm/damon/Kconfig" endmenu
mm/swap.c +39 -0
@@ -405,6 +405,40 @@ static void __lru_cache_activate_folio(struct folio *folio)local_unlock (&lru_pvecs.lock); } +#ifdef CONFIG_LRU_GEN +static void folio_inc_refs (struct folio *folio) +{ +unsigned long new_flags, old_flags =READ_ONCE (folio->flags); + +if (folio_test_unevictable (folio)) +return ; + +if (!folio_test_referenced (folio)) { +folio_set_referenced (folio); +return ; + } + +if (!folio_test_workingset (folio)) { +folio_set_workingset (folio); +return ; + } + +/* see the comment on MAX_NR_TIERS */ +do { + new_flags = old_flags &LRU_REFS_MASK ; +if (new_flags ==LRU_REFS_MASK ) +break ; + + new_flags +=BIT (LRU_REFS_PGOFF ); + new_flags |= old_flags & ~LRU_REFS_MASK ; + }while (!try_cmpxchg (&folio->flags, &old_flags, new_flags)); +} +#else +static void folio_inc_refs (struct folio *folio) +{ +} +#endif /* CONFIG_LRU_GEN */ +/* * Mark a page as having seen activity. *
@@ -417,6 +451,11 @@ static void __lru_cache_activate_folio(struct folio *folio)*/ void folio_mark_accessed (struct folio *folio) { +if (lru_gen_enabled ()) { +folio_inc_refs (folio); +return ; + } +if (!folio_test_referenced (folio)) {folio_set_referenced (folio); }else if (folio_test_unevictable (folio)) {
mm/vmscan.c +792 -7
861 lines not on this page · raw message
mm/workingset.c +108 -2
142 lines not on this page · raw message
Signature · 1 line
2.36.0.550.gb090851708-goog
Re: [PATCH v11 06/14] mm: multi-gen LRU: minimal implementation
Quoted 1,328 lines · Yu Zhao
To avoid confusion, the terms "promotion" and "demotion" will be applied to the multi-gen LRU, as a new convention; the terms "activation" and "deactivation" will be applied to the active/inactive LRU, as usual.
The aging produces young generations. Given an lruvec, it increments
max_seqwhen max_seq-min_seq+1 approachesMIN_NR_GENS. The aging promotes hot pages to the youngest generation when it finds them accessed through page tables; the demotion of cold pages happens consequently when it incrementsmax_seq. The aging has the complexityO(nr_hot_pages), since it is only interested in hot pages. Promotion in the aging path does not involve any LRU list operations, only the updates of the gen counter and lrugen->nr_pages[]; demotion, unless as the result of the increment ofmax_seq, requires LRU list operations, e.g.,lru_deactivate_fn().The eviction consumes old generations. Given an lruvec, it increments
min_seqwhen the lists indexed by min_seq%MAX_NR_GENS become empty. A feedback loop modeled after the PID controller monitors refaults over anon and file types and decides which type to evict when both types are available from the same generation.Each generation is divided into multiple tiers. Tiers represent different ranges of numbers of accesses through file descriptors. A page accessed N times through file descriptors is in tier
order_base_2(N). Tiers do not have dedicated lrugen->lists[], only bits infolio->flags. In contrast to moving across generations, which requires the LRU lock, moving across tiers only involves operations onfolio->flags. The feedback loop also monitors refaults over all tiers and decides when to protect pages in which tiers (N>1), using the first tier (N=0,1) as a baseline. The first tier contains single-use unmapped clean pages, which are most likely the best choices. The eviction moves a page to the next generation, i.e., min_seq+1, if the feedback loop decides so. This approach has the following advantages:
1. It removes the cost of activation in the buffered access path by inferring whether pages accessed multiple times through file descriptors are statistically hot and thus worth protecting in the eviction path.
2. It takes pages accessed through page tables into account and avoids overprotecting pages accessed multiple times through file descriptors. (Pages accessed through page tables are in the first tier, since N=0.)
3. More tiers provide better protection for pages accessed more than twice through file descriptors, when under heavy buffered I/O workloads.Server benchmark results:
Single workload: fio (buffered I/O): +[40, 42]% IOPS BW 5.18-rc1: 2463k 9621MiB/s patch1-6: 3484k 13.3GiB/s Single workload: memcached (anon): +[44, 46]% Ops/sec KB/sec 5.18-rc1: 771403.27 30004.17 patch1-6: 1120643.70 43588.06 Configurations: CPU: two Xeon 6154 Mem: total 256G Node 1 was only used as a ram disk to reduce the variance in the results. patch drivers/block/brd.c <<EOF 99,100c99,100 < gfp_flags = GFP_NOIO | __GFP_ZERO | __GFP_HIGHMEM; < page = alloc_page(gfp_flags); --- > gfp_flags = GFP_NOIO | __GFP_ZERO | __GFP_HIGHMEM | __GFP_THISNODE; > page = alloc_pages_node(1, gfp_flags, 0); EOF cat >>/etc/systemd/system.conf <<EOF CPUAffinity=numa NUMAPolicy=bind NUMAMask=0 EOF cat >>/etc/memcached.conf <<EOF -m 184320 -s /var/run/memcached/memcached.sock -a 0766 -t 36 -B binary EOF cat fio.sh modprobe brd rd_nr=1 rd_size=113246208 swapoff -a mkfs.ext4 /dev/ram0 mount -t ext4 /dev/ram0 /mnt mkdir /sys/fs/cgroup/user.slice/test echo 38654705664 >/sys/fs/cgroup/user.slice/test/memory.max echo $$ >/sys/fs/cgroup/user.slice/test/cgroup.procs fio -name=mglru --numjobs=72 --directory=/mnt --size=1408m \ --buffered=1 --ioengine=io_uring --iodepth=128 \ --iodepth_batch_submit=32 --iodepth_batch_complete=32 \ --rw=randread --random_distribution=random --norandommap \ --time_based --ramp_time=10m --runtime=5m --group_reporting cat memcached.sh modprobe brd rd_nr=1 rd_size=113246208 swapoff -a mkswap /dev/ram0 swapon /dev/ram0 memtier_benchmark -S /var/run/memcached/memcached.sock \ -P memcache_binary -n allkeys --key-minimum=1 \ --key-maximum=65000000 --key-pattern=P:P -c 1 -t 36 \ --ratio 1:0 --pipeline 8 -d 2000 memtier_benchmark -S /var/run/memcached/memcached.sock \ -P memcache_binary -n allkeys --key-minimum=1 \ --key-maximum=65000000 --key-pattern=R:R -c 1 -t 36 \ --ratio 0:1 --pipeline 8 --randomize --distinct-client-seedClient benchmark results:
kswapd profiles: 5.18-rc1 40.53% page_vma_mapped_walk 20.37% lzo1x_1_do_compress (real work) 6.99% do_raw_spin_lock 3.93% _raw_spin_unlock_irq 2.08% vma_interval_tree_subtree_search 2.06% vma_interval_tree_iter_next 1.95% folio_referenced_one 1.93% anon_vma_interval_tree_iter_first 1.51% ptep_clear_flush 1.35% __anon_vma_interval_tree_subtree_search patch1-6 35.99% lzo1x_1_do_compress (real work) 19.40% page_vma_mapped_walk 6.31% _raw_spin_unlock_irq 3.95% do_raw_spin_lock 2.39% anon_vma_interval_tree_iter_first 2.25% ptep_clear_flush 1.92% __anon_vma_interval_tree_subtree_search 1.70% folio_referenced_one 1.68% __zram_bvec_write 1.43% anon_vma_interval_tree_iter_next Configurations: CPU: single Snapdragon 7c Mem: total 4G Chrome OS MemoryPressure [1][1] https://chromium.googlesource.com/chromiumos/platform/tast-tests/
Signed-off-by Yu Zhao <[email protected]>Acked-by Brian Geffon <[email protected]>Acked-by Steven Barrett <[email protected]>Acked-by Suleiman Souhlal <[email protected]>Tested-by Daniel Byrne <[email protected]>Tested-by Donald Carr <[email protected]>Tested-by Holger Hoffstätte <[email protected]>Tested-by Konstantin Kharlamov <[email protected]>Tested-by Shuang Zhai <[email protected]>Tested-by Sofia Trinh <[email protected]>Tested-by Vaibhav Jain <[email protected]>--- include/linux/mm_inline.h | 36 ++ include/linux/mmzone.h | 42 ++ include/linux/page-flags-layout.h | 5 +- kernel/bounds.c | 2 + mm/Kconfig | 11 + mm/swap.c | 39 ++ mm/vmscan.c | 799 +++++++++++++++++++++++++++++- mm/workingset.c | 110 +++- 8 files changed, 1034 insertions(+), 10 deletions(-)include/linux/mm_inline.h +36
-0@@ -119,6 +119,33 @@ static inline int lru_gen_from_seq(unsigned long seq)return seq %MAX_NR_GENS ; } +static inline int lru_hist_from_seq (unsigned long seq) +{ +return seq %NR_HIST_GENS ; +} + +static inline int lru_tier_from_refs (int refs) +{ +VM_WARN_ON_ONCE (refs >BIT (LRU_REFS_WIDTH )); + +/* see the comment in folio_lru_refs() */ +return order_base_2 (refs +1 ); +} + +static inline int folio_lru_refs (struct folio *folio) +{ +unsigned long flags =READ_ONCE (folio->flags); +bool workingset = flags &BIT (PG_workingset); + +/* +* Return the number of accesses beyond PG_referenced, i.e., N-1 if the +* total number of accesses is N>1, since N=0,1 both map to the first +* tier. lru_tier_from_refs() will account for this off-by-one. Also see +* the comment on MAX_NR_TIERS. +*/ +return ((flags &LRU_REFS_MASK ) >>LRU_REFS_PGOFF ) + workingset; +} +static inline int folio_lru_gen (struct folio *folio) {unsigned long flags =READ_ONCE (folio->flags);@@ -171,6 +198,15 @@ static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *foli__update_lru_size (lruvec, lru, zone, -delta);return ; } + +/* promotion */ +if (!lru_gen_is_active (lruvec, old_gen) &&lru_gen_is_active (lruvec, new_gen)) { +__update_lru_size (lruvec, lru, zone, -delta); +__update_lru_size (lruvec, lru +LRU_ACTIVE , zone, delta); + } + +/* demotion requires isolation, e.g., lru_deactivate_fn() */ +VM_WARN_ON_ONCE (lru_gen_is_active (lruvec, old_gen) && !lru_gen_is_active (lruvec, new_gen)); }static inline bool lru_gen_add_folio (struct lruvec *lruvec,struct folio *folio,bool reclaiming)include/linux/mmzone.h +42
-0@@ -348,6 +348,29 @@ enum lruvec_flags {#define MIN_NR_GENS 2U#define MAX_NR_GENS 4U +/* +* Each generation is divided into multiple tiers. Tiers represent different +* ranges of numbers of accesses through file descriptors. A page accessed N +* times through file descriptors is in tier order_base_2(N). A page in the +* first tier (N=0,1) is marked by PG_referenced unless it was faulted in +* though page tables or read ahead. A page in any other tier (N>1) is marked +* by PG_referenced and PG_workingset. This implies a minimum of two tiers is +* supported without using additional bits in folio->flags. +* +* In contrast to moving across generations which requires the LRU lock, moving +* across tiers only involves atomic operations on folio->flags and therefore +* has a negligible cost in the buffered access path. In the eviction path, +* comparisons of refaulted/(evicted+protected) from the first tier and the +* rest infer whether pages accessed multiple times through file descriptors +* are statistically hot and thus worth protecting. +* +* MAX_NR_TIERS is set to 4 so that the multi-gen LRU can support twice the +* number of categories of the active/inactive LRU when keeping track of +* accesses through file descriptors. It uses MAX_NR_TIERS-2 spare bits in +* folio->flags (LRU_REFS_MASK). +*/ +#define MAX_NR_TIERS 4U +#ifndef __GENERATING_BOUNDS_Hstruct lruvec ;@@ -362,6 +385,16 @@ enum { LRU_GEN_FILE, }; +#define MIN_LRU_BATCH BITS_PER_LONG +#define MAX_LRU_BATCH (MIN_LRU_BATCH * 128) + +/* whether to keep historical stats from evicted generations */ +#ifdef CONFIG_LRU_GEN_STATS +#define NR_HIST_GENS MAX_NR_GENS +#else +#define NR_HIST_GENS 1U +#endif +/* * The youngest generation number is stored in max_seq for both anon and file * types as they are aged on an equal footing. The oldest generation numbers are @@ -384,6 +417,15 @@ struct lru_gen_struct {struct list_head lists[MAX_NR_GENS ][ANON_AND_FILE ][MAX_NR_ZONES ];/* the sizes of the above lists */ long nr_pages[MAX_NR_GENS ][ANON_AND_FILE ][MAX_NR_ZONES ]; +/* the exponential moving average of refaulted */ +unsigned long avg_refaulted[ANON_AND_FILE ][MAX_NR_TIERS ]; +/* the exponential moving average of evicted+protected */ +unsigned long avg_total[ANON_AND_FILE ][MAX_NR_TIERS ]; +/* the first tier doesn't need protection, hence the minus one */ +unsigned long protected[NR_HIST_GENS ][ANON_AND_FILE ][MAX_NR_TIERS -1 ]; +/* can be modified without holding the LRU lock */ +atomic_long_t evicted[NR_HIST_GENS ][ANON_AND_FILE ][MAX_NR_TIERS ]; +atomic_long_t refaulted[NR_HIST_GENS ][ANON_AND_FILE ][MAX_NR_TIERS ]; };void lru_gen_init_lruvec (struct lruvec *lruvec);include/linux/page-flags-layout.h +4
-1@@ -106,7 +106,10 @@#error "Not enough bits in page flags"#endif -#define LRU_REFS_WIDTH 0 +/* see the comment on MAX_NR_TIERS */ +#define LRU_REFS_WIDTH min(__LRU_REFS_WIDTH, BITS_PER_LONG - NR_PAGEFLAGS - \ + ZONES_WIDTH - LRU_GEN_WIDTH - SECTIONS_WIDTH - \ + NODES_WIDTH - KASAN_TAG_WIDTH - LAST_CPUPID_WIDTH)#endif #endif /* _LINUX_PAGE_FLAGS_LAYOUT */ kernel/bounds.c +2
-0@@ -24,8 +24,10 @@ int main(void)DEFINE (SPINLOCK_SIZE ,sizeof (spinlock_t));#ifdef CONFIG_LRU_GEN DEFINE (LRU_GEN_WIDTH ,order_base_2 (MAX_NR_GENS +1 )); +DEFINE (__LRU_REFS_WIDTH,MAX_NR_TIERS -2 );#else DEFINE (LRU_GEN_WIDTH ,0 ); +DEFINE (__LRU_REFS_WIDTH,0 );#endif /* End of constants */ mm/Kconfig +11
-0@@ -909,6 +909,7 @@ config ANON_VMA_NAME area from being merged with adjacent virtual memory areas due to the difference in their name. +# multi-gen LRU { config LRU_GEN bool "Multi-Gen LRU" depends on MMU@@ -917,6 +918,16 @@ config LRU_GEN help A high performance LRU implementation to overcommit memory. +config LRU_GEN_STATS + bool "Full stats for debugging" + depends on LRU_GEN + help + Do not enable this option unless you plan to look at historical stats + from evicted generations for debugging purpose. + + This option has a per-memcg and per-node memory overhead. +# } + source "mm/damon/Kconfig" endmenumm/swap.c +39
-0@@ -405,6 +405,40 @@ static void __lru_cache_activate_folio(struct folio *folio)local_unlock (&lru_pvecs.lock); } +#ifdef CONFIG_LRU_GEN +static void folio_inc_refs (struct folio *folio) +{ +unsigned long new_flags, old_flags =READ_ONCE (folio->flags); + +if (folio_test_unevictable (folio)) +return ; + +if (!folio_test_referenced (folio)) { +folio_set_referenced (folio); +return ; + } + +if (!folio_test_workingset (folio)) { +folio_set_workingset (folio); +return ; + } + +/* see the comment on MAX_NR_TIERS */ +do { + new_flags = old_flags &LRU_REFS_MASK ; +if (new_flags ==LRU_REFS_MASK ) +break ; + + new_flags +=BIT (LRU_REFS_PGOFF ); + new_flags |= old_flags & ~LRU_REFS_MASK ; + }while (!try_cmpxchg (&folio->flags, &old_flags, new_flags)); +} +#else +static void folio_inc_refs (struct folio *folio) +{ +} +#endif /* CONFIG_LRU_GEN */ +/* * Mark a page as having seen activity. * @@ -417,6 +451,11 @@ static void __lru_cache_activate_folio(struct folio *folio)*/ void folio_mark_accessed (struct folio *folio) { +if (lru_gen_enabled ()) { +folio_inc_refs (folio); +return ; + } +if (!folio_test_referenced (folio)) {folio_set_referenced (folio); }else if (folio_test_unevictable (folio)) {mm/vmscan.c +792
-7@@ -1275,9 +1275,11 @@ static int __remove_mapping(struct address_space *mapping, struct folio *folio,if (folio_test_swapcache (folio)) {swp_entry_t swap =folio_swap_entry (folio); -mem_cgroup_swapout (folio, swap); + +/* get a shadow entry before mem_cgroup_swapout() clears folio_memcg() */ if (reclaimed && !mapping_exiting (mapping)) shadow =workingset_eviction (folio, target_memcg); +mem_cgroup_swapout (folio, swap);__delete_from_swap_cache (&folio->page, swap, shadow);xa_unlock_irq (&mapping->i_pages);put_swap_page (&folio->page, swap);@@ -2649,6 +2651,9 @@ static void prepare_scan_count(pg_data_t *pgdat, struct scan_control *sc)unsigned long file;struct lruvec *target_lruvec; +if (lru_gen_enabled ()) +return ; + target_lruvec =mem_cgroup_lruvec (sc->target_mem_cgroup, pgdat);/* @@ -2974,6 +2979,17 @@ static bool can_age_anon_pages(struct pglist_data *pgdat,* shorthand helpers ******************************************************************************/ +#define LRU_REFS_FLAGS (BIT(PG_referenced) | BIT(PG_workingset)) + +#define DEFINE_MAX_SEQ (lruvec) \ + unsigned long max_seq = READ_ONCE((lruvec)->lrugen.max_seq) + +#define DEFINE_MIN_SEQ (lruvec) \ + unsigned long min_seq[ANON_AND_FILE] = { \ + READ_ONCE((lruvec)->lrugen.min_seq[LRU_GEN_ANON]), \ + READ_ONCE((lruvec)->lrugen.min_seq[LRU_GEN_FILE]), \ + } +#define for_each_gen_type_zone (gen, type, zone) \ for ((gen) = 0; (gen) < MAX_NR_GENS; (gen)++) \ for ((type) = 0; (type) < ANON_AND_FILE; (type)++) \@@ -2999,6 +3015,753 @@ static struct lruvec __maybe_unused *get_lruvec(struct mem_cgroup *memcg, int nireturn pgdat ? &pgdat->__lruvec :NULL ; } +static int get_swappiness (struct lruvec *lruvec,struct scan_control *sc) +{ +struct mem_cgroup *memcg =lruvec_memcg (lruvec); +struct pglist_data *pgdat =lruvec_pgdat (lruvec); + +if (!can_demote (pgdat->node_id, sc) && +mem_cgroup_get_nr_swap_pages (memcg) <MIN_LRU_BATCH ) +return 0 ; + +return mem_cgroup_swappiness (memcg); +} + +static int get_nr_gens (struct lruvec *lruvec,int type) +{ +return lruvec->lrugen.max_seq - lruvec->lrugen.min_seq[type] +1 ; +} + +static bool __maybe_unused seq_is_valid (struct lruvec *lruvec) +{ +/* see the comment on lru_gen_struct */ +return get_nr_gens (lruvec,LRU_GEN_FILE ) >=MIN_NR_GENS && +get_nr_gens (lruvec,LRU_GEN_FILE ) <=get_nr_gens (lruvec,LRU_GEN_ANON ) && +get_nr_gens (lruvec,LRU_GEN_ANON ) <=MAX_NR_GENS ; +} + +/****************************************************************************** +* refault feedback loop +******************************************************************************/ + +/* +* A feedback loop based on Proportional-Integral-Derivative (PID) controller. +* +* The P term is refaulted/(evicted+protected) from a tier in the generation +* currently being evicted; the I term is the exponential moving average of the +* P term over the generations previously evicted, using the smoothing factor +* 1/2; the D term isn't supported. +* +* The setpoint (SP) is always the first tier of one type; the process variable +* (PV) is either any tier of the other type or any other tier of the same +* type. +* +* The error is the difference between the SP and the PV; the correction is +* turn off protection when SP>PV or turn on protection when SP<PV. +* +* For future optimizations: +* 1. The D term may discount the other two terms over time so that long-lived +* generations can resist stale information. +*/ +struct ctrl_pos { +unsigned long refaulted; +unsigned long total; +int gain; +}; + +static void read_ctrl_pos (struct lruvec *lruvec,int type,int tier,int gain, +struct ctrl_pos *pos) +{ +struct lru_gen_struct *lrugen = &lruvec->lrugen; +int hist =lru_hist_from_seq (lrugen->min_seq[type]); + + pos->refaulted = lrugen->avg_refaulted[type][tier] + +atomic_long_read (&lrugen->refaulted[hist][type][tier]); + pos->total = lrugen->avg_total[type][tier] + +atomic_long_read (&lrugen->evicted[hist][type][tier]); +if (tier) + pos->total += lrugen->protected[hist][type][tier -1 ]; + pos->gain = gain; +} + +static void reset_ctrl_pos (struct lruvec *lruvec,int type,bool carryover) +{ +int hist, tier; +struct lru_gen_struct *lrugen = &lruvec->lrugen; +bool clear = carryover ?NR_HIST_GENS ==1 :NR_HIST_GENS >1 ; +unsigned long seq = carryover ? lrugen->min_seq[type] : lrugen->max_seq +1 ; + +lockdep_assert_held (&lruvec->lru_lock); + +if (!carryover && !clear) +return ; + + hist =lru_hist_from_seq (seq); + +for (tier =0 ; tier <MAX_NR_TIERS ; tier++) { +if (carryover) { +unsigned long sum; + + sum = lrugen->avg_refaulted[type][tier] + +atomic_long_read (&lrugen->refaulted[hist][type][tier]); +WRITE_ONCE (lrugen->avg_refaulted[type][tier], sum /2 ); + + sum = lrugen->avg_total[type][tier] + +atomic_long_read (&lrugen->evicted[hist][type][tier]); +if (tier) + sum += lrugen->protected[hist][type][tier -1 ]; +WRITE_ONCE (lrugen->avg_total[type][tier], sum /2 ); + } + +if (clear) { +atomic_long_set (&lrugen->refaulted[hist][type][tier],0 ); +atomic_long_set (&lrugen->evicted[hist][type][tier],0 ); +if (tier) +WRITE_ONCE (lrugen->protected[hist][type][tier -1 ],0 ); + } + } +} + +static bool positive_ctrl_err (struct ctrl_pos *sp,struct ctrl_pos *pv) +{ +/* +* Return true if the PV has a limited number of refaults or a lower +* refaulted/total than the SP. +*/ +return pv->refaulted <MIN_LRU_BATCH || + pv->refaulted * (sp->total +MIN_LRU_BATCH ) * sp->gain <= + (sp->refaulted +1 ) * pv->total * pv->gain; +} + +/****************************************************************************** +* the aging +******************************************************************************/ + +static int folio_inc_gen (struct lruvec *lruvec,struct folio *folio,bool reclaiming) +{ +int type =folio_is_file_lru (folio); +struct lru_gen_struct *lrugen = &lruvec->lrugen; +int new_gen, old_gen =lru_gen_from_seq (lrugen->min_seq[type]); +unsigned long new_flags, old_flags =READ_ONCE (folio->flags); + +VM_WARN_ON_ONCE_FOLIO (!(old_flags &LRU_GEN_MASK ), folio); + +do { + new_gen = (old_gen +1 ) %MAX_NR_GENS ; + + new_flags = old_flags & ~(LRU_GEN_MASK |LRU_REFS_MASK |LRU_REFS_FLAGS ); + new_flags |= (new_gen +1UL ) <<LRU_GEN_PGOFF ; +/* for folio_end_writeback() */ +if (reclaiming) + new_flags |=BIT (PG_reclaim); + }while (!try_cmpxchg (&folio->flags, &old_flags, new_flags)); + +lru_gen_update_size (lruvec, folio, old_gen, new_gen); + +return new_gen; +} + +static void inc_min_seq (struct lruvec *lruvec,int type) +{ +struct lru_gen_struct *lrugen = &lruvec->lrugen; + +reset_ctrl_pos (lruvec, type, true); +WRITE_ONCE (lrugen->min_seq[type], lrugen->min_seq[type] +1 ); +} + +static bool try_to_inc_min_seq (struct lruvec *lruvec,bool can_swap) +{ +int gen, type, zone; +bool success = false; +struct lru_gen_struct *lrugen = &lruvec->lrugen; +DEFINE_MIN_SEQ (lruvec); + +VM_WARN_ON_ONCE (!seq_is_valid (lruvec)); + +/* find the oldest populated generation */ +for (type = !can_swap; type <ANON_AND_FILE ; type++) { +while (min_seq[type] +MIN_NR_GENS <= lrugen->max_seq) { + gen =lru_gen_from_seq (min_seq[type]); + +for (zone =0 ; zone <MAX_NR_ZONES ; zone++) { +if (!list_empty (&lrugen->lists[gen][type][zone])) + goto next; + } + + min_seq[type]++; + } +next: + ; + } + +/* see the comment on lru_gen_struct */ +if (can_swap) { + min_seq[LRU_GEN_ANON ] =min (min_seq[LRU_GEN_ANON ], min_seq[LRU_GEN_FILE ]); + min_seq[LRU_GEN_FILE ] =max (min_seq[LRU_GEN_ANON ], lrugen->min_seq[LRU_GEN_FILE ]); + } + +for (type = !can_swap; type <ANON_AND_FILE ; type++) { +if (min_seq[type] == lrugen->min_seq[type]) +continue ; + +reset_ctrl_pos (lruvec, type, true); +WRITE_ONCE (lrugen->min_seq[type], min_seq[type]); + success = true; + } + +return success; +} + +static void inc_max_seq (struct lruvec *lruvec,unsigned long max_seq,bool can_swap) +{ +int prev, next; +int type, zone; +struct lru_gen_struct *lrugen = &lruvec->lrugen; + +spin_lock_irq (&lruvec->lru_lock); + +VM_WARN_ON_ONCE (!seq_is_valid (lruvec)); + +if (max_seq != lrugen->max_seq) + goto unlock; + +for (type =0 ; type <ANON_AND_FILE ; type++) { +if (get_nr_gens (lruvec, type) !=MAX_NR_GENS ) +continue ; + +VM_WARN_ON_ONCE (type ==LRU_GEN_FILE || can_swap); + +inc_min_seq (lruvec, type); + } + +/* +* Update the active/inactive LRU sizes for compatibility. Both sides of +* the current max_seq need to be covered, since max_seq+1 can overlap +* with min_seq[LRU_GEN_ANON] if swapping is constrained. And if they do +* overlap, cold/hot inversion happens. +*/ + prev =lru_gen_from_seq (lrugen->max_seq -1 ); + next =lru_gen_from_seq (lrugen->max_seq +1 ); + +for (type =0 ; type <ANON_AND_FILE ; type++) { +for (zone =0 ; zone <MAX_NR_ZONES ; zone++) { +enum lru_list lru = type *LRU_INACTIVE_FILE ; +long delta = lrugen->nr_pages[prev][type][zone] - + lrugen->nr_pages[next][type][zone]; + +if (!delta) +continue ; + +__update_lru_size (lruvec, lru, zone, delta); +__update_lru_size (lruvec, lru +LRU_ACTIVE , zone, -delta); + } + } + +for (type =0 ; type <ANON_AND_FILE ; type++) +reset_ctrl_pos (lruvec, type, false); + +/* make sure preceding modifications appear */ +smp_store_release (&lrugen->max_seq, lrugen->max_seq +1 ); +unlock: +spin_unlock_irq (&lruvec->lru_lock); +} + +static long get_nr_evictable (struct lruvec *lruvec,unsigned long max_seq, +unsigned long *min_seq,bool can_swap,bool *need_aging) +{ +int gen, type, zone; +long old =0 ; +long young =0 ; +long total =0 ; +struct lru_gen_struct *lrugen = &lruvec->lrugen; + +for (type = !can_swap; type <ANON_AND_FILE ; type++) { +unsigned long seq; + +for (seq = min_seq[type]; seq <= max_seq; seq++) { +long size =0 ; + + gen =lru_gen_from_seq (seq); + +for (zone =0 ; zone <MAX_NR_ZONES ; zone++) + size +=READ_ONCE (lrugen->nr_pages[gen][type][zone]); + + total += size; +if (seq == max_seq) + young += size; +if (seq +MIN_NR_GENS == max_seq) + old += size; + } + } + +/* +* The aging tries to be lazy to reduce the overhead. On the other hand, +* the eviction stalls when the number of generations reaches +* MIN_NR_GENS. So ideally, there should be MIN_NR_GENS+1 generations, +* hence the first two if's. +* +* Also it's ideal to spread pages out evenly, meaning 1/(MIN_NR_GENS+1) +* of the total number of pages for each generation. A reasonable range +* for this average portion is [1/MIN_NR_GENS, 1/(MIN_NR_GENS+2)]. The +* eviction cares about the lower bound of cold pages, whereas the aging +* cares about the upper bound of hot pages. +*/ +if (min_seq[!can_swap] +MIN_NR_GENS > max_seq) + *need_aging = true; +else if (min_seq[!can_swap] +MIN_NR_GENS < max_seq) + *need_aging = false; +else if (young *MIN_NR_GENS > total) + *need_aging = true; +else if (old * (MIN_NR_GENS +2 ) < total) + *need_aging = true; +else + *need_aging = false; + +return total >0 ? total :0 ; +} + +static void age_lruvec (struct lruvec *lruvec,struct scan_control *sc) +{ +bool need_aging; +long nr_to_scan; +int swappiness =get_swappiness (lruvec, sc); +struct mem_cgroup *memcg =lruvec_memcg (lruvec); +DEFINE_MAX_SEQ (lruvec); +DEFINE_MIN_SEQ (lruvec); + +VM_WARN_ON_ONCE (sc->memcg_low_reclaim); + +mem_cgroup_calculate_protection (NULL , memcg); + +if (mem_cgroup_below_min (memcg)) +return ; + + nr_to_scan =get_nr_evictable (lruvec, max_seq, min_seq, swappiness, &need_aging); +if (!nr_to_scan) +return ; + + nr_to_scan >>= sc->priority; + +if (!mem_cgroup_online (memcg)) + nr_to_scan++; + +if (nr_to_scan && need_aging) +inc_max_seq (lruvec, max_seq, swappiness); +} + +static void lru_gen_age_node (struct pglist_data *pgdat,struct scan_control *sc) +{ +struct mem_cgroup *memcg; + +VM_WARN_ON_ONCE (!current_is_kswapd ()); + + memcg =mem_cgroup_iter (NULL ,NULL ,NULL ); +do { +struct lruvec *lruvec =mem_cgroup_lruvec (memcg, pgdat); + +age_lruvec (lruvec, sc); + +cond_resched (); + }while ((memcg =mem_cgroup_iter (NULL , memcg,NULL ))); +} + +/****************************************************************************** +* the eviction +******************************************************************************/ + +static bool sort_folio (struct lruvec *lruvec,struct folio *folio,int tier_idx) +{ +bool success; +int gen =folio_lru_gen (folio); +int type =folio_is_file_lru (folio); +int zone =folio_zonenum (folio); +int delta =folio_nr_pages (folio); +int refs =folio_lru_refs (folio); +int tier =lru_tier_from_refs (refs); +struct lru_gen_struct *lrugen = &lruvec->lrugen; + +VM_WARN_ON_ONCE_FOLIO (gen >=MAX_NR_GENS , folio); + +/* unevictable */ +if (!folio_evictable (folio)) { + success =lru_gen_del_folio (lruvec, folio, true); +VM_WARN_ON_ONCE_FOLIO (!success, folio); +folio_set_unevictable (folio); +lruvec_add_folio (lruvec, folio); +__count_vm_events (UNEVICTABLE_PGCULLED , delta); +return true; + } + +/* dirtied lazyfree */ +if (type ==LRU_GEN_FILE &&folio_test_anon (folio) &&folio_test_dirty (folio)) { + success =lru_gen_del_folio (lruvec, folio, true); +VM_WARN_ON_ONCE_FOLIO (!success, folio); +folio_set_swapbacked (folio); +lruvec_add_folio_tail (lruvec, folio); +return true; + } + +/* protected */ +if (tier > tier_idx) { +int hist =lru_hist_from_seq (lrugen->min_seq[type]); + + gen =folio_inc_gen (lruvec, folio, false); +list_move_tail (&folio->lru, &lrugen->lists[gen][type][zone]); + +WRITE_ONCE (lrugen->protected[hist][type][tier -1 ], + lrugen->protected[hist][type][tier -1 ] + delta); +__mod_lruvec_state (lruvec,WORKINGSET_ACTIVATE_BASE + type, delta); +return true; + } + +/* waiting for writeback */ +if (folio_test_locked (folio) ||folio_test_writeback (folio) || + (type ==LRU_GEN_FILE &&folio_test_dirty (folio))) { + gen =folio_inc_gen (lruvec, folio, true); +list_move (&folio->lru, &lrugen->lists[gen][type][zone]); +return true; + } + +return false; +} + +static bool isolate_folio (struct lruvec *lruvec,struct folio *folio,struct scan_control *sc) +{ +bool success; + +if (!sc->may_unmap &&folio_mapped (folio)) +return false; + +if (!(sc->may_writepage && (sc->gfp_mask & __GFP_IO)) && + (folio_test_dirty (folio) || + (folio_test_anon (folio) && !folio_test_swapcache (folio)))) +return false; + +if (!folio_try_get (folio)) +return false; + +if (!folio_test_clear_lru (folio)) { +folio_put (folio); +return false; + } + + success =lru_gen_del_folio (lruvec, folio, true); +VM_WARN_ON_ONCE_FOLIO (!success, folio); + +return true; +} + +static int scan_folios (struct lruvec *lruvec,struct scan_control *sc, +int type,int tier,struct list_head *list) +{ +int gen, zone; +enum vm_event_item item; +int sorted =0 ; +int scanned =0 ; +int isolated =0 ; +int remaining =MAX_LRU_BATCH ; +struct lru_gen_struct *lrugen = &lruvec->lrugen; +struct mem_cgroup *memcg =lruvec_memcg (lruvec); + +VM_WARN_ON_ONCE (!list_empty (list)); + +if (get_nr_gens (lruvec, type) ==MIN_NR_GENS ) +return 0 ; + + gen =lru_gen_from_seq (lrugen->min_seq[type]); + +for (zone = sc->reclaim_idx; zone >=0 ; zone--) { +LIST_HEAD (moved); +int skipped =0 ; +struct list_head *head = &lrugen->lists[gen][type][zone]; + +while (!list_empty (head)) { +struct folio *folio =lru_to_folio (head); +int delta =folio_nr_pages (folio); + +VM_WARN_ON_ONCE_FOLIO (folio_test_unevictable (folio), folio); +VM_WARN_ON_ONCE_FOLIO (folio_test_active (folio), folio); +VM_WARN_ON_ONCE_FOLIO (folio_is_file_lru (folio) != type, folio); +VM_WARN_ON_ONCE_FOLIO (folio_zonenum (folio) != zone, folio); + + scanned += delta; + +if (sort_folio (lruvec, folio, tier)) + sorted += delta; +else if (isolate_folio (lruvec, folio, sc)) { +list_add (&folio->lru, list); + isolated += delta; + }else { +list_move (&folio->lru, &moved); + skipped += delta; + } + +if (!--remaining ||max (isolated, skipped) >=MIN_LRU_BATCH ) +break ; + } + +if (skipped) { +list_splice (&moved, head); +__count_zid_vm_events (PGSCAN_SKIP , zone, skipped); + } + +if (!remaining || isolated >=MIN_LRU_BATCH ) +break ; + } + + item =current_is_kswapd () ?PGSCAN_KSWAPD :PGSCAN_DIRECT ; +if (!cgroup_reclaim (sc)) { +__count_vm_events (item, isolated); +__count_vm_events (PGREFILL , sorted); + } +__count_memcg_events (memcg, item, isolated); +__count_memcg_events (memcg,PGREFILL , sorted); +__count_vm_events (PGSCAN_ANON + type, isolated); + +/* +* There might not be eligible pages due to reclaim_idx, may_unmap and +* may_writepage. Check the remaining to prevent livelock if there is no +* progress. +*/ +return isolated || !remaining ? scanned :0 ; +} + +static int get_tier_idx (struct lruvec *lruvec,int type) +{ +int tier; +struct ctrl_pos sp, pv; + +/* +* To leave a margin for fluctuations, use a larger gain factor (1:2). +* This value is chosen because any other tier would have at least twice +* as many refaults as the first tier. +*/ +read_ctrl_pos (lruvec, type,0 ,1 , &sp); +for (tier =1 ; tier <MAX_NR_TIERS ; tier++) { +read_ctrl_pos (lruvec, type, tier,2 , &pv); +if (!positive_ctrl_err (&sp, &pv)) +break ; + } + +return tier -1 ; +} + +static int get_type_to_scan (struct lruvec *lruvec,int swappiness,int *tier_idx) +{ +int type, tier; +struct ctrl_pos sp, pv; +int gain[ANON_AND_FILE ] = { swappiness,200 - swappiness }; + +/* +* Compare the first tier of anon with that of file to determine which +* type to scan. Also need to compare other tiers of the selected type +* with the first tier of the other type to determine the last tier (of +* the selected type) to evict. +*/ +read_ctrl_pos (lruvec,LRU_GEN_ANON ,0 , gain[LRU_GEN_ANON ], &sp); +read_ctrl_pos (lruvec,LRU_GEN_FILE ,0 , gain[LRU_GEN_FILE ], &pv); + type =positive_ctrl_err (&sp, &pv); + +read_ctrl_pos (lruvec, !type,0 , gain[!type], &sp); +for (tier =1 ; tier <MAX_NR_TIERS ; tier++) { +read_ctrl_pos (lruvec, type, tier, gain[type], &pv); +if (!positive_ctrl_err (&sp, &pv)) +break ; + } + + *tier_idx = tier -1 ; + +return type; +} + +static int isolate_folios (struct lruvec *lruvec,struct scan_control *sc,int swappiness, +int *type_scanned,struct list_head *list) +{ +int i; +int type; +int scanned; +int tier =-1 ; +DEFINE_MIN_SEQ (lruvec); + +/* +* Try to make the obvious choice first. When anon and file are both +* available from the same generation, interpret swappiness 1 as file +* first and 200 as anon first. +*/ +if (!swappiness) + type =LRU_GEN_FILE ; +else if (min_seq[LRU_GEN_ANON ] < min_seq[LRU_GEN_FILE ]) + type =LRU_GEN_ANON ; +else if (swappiness ==1 ) + type =LRU_GEN_FILE ; +else if (swappiness ==200 ) + type =LRU_GEN_ANON ; +else + type =get_type_to_scan (lruvec, swappiness, &tier); + +for (i = !swappiness; i <ANON_AND_FILE ; i++) { +if (tier <0 ) + tier =get_tier_idx (lruvec, type); + + scanned =scan_folios (lruvec, sc, type, tier, list); +if (scanned) +break ; + + type = !type; + tier =-1 ; + } + + *type_scanned = type; + +return scanned; +} + +static int evict_folios (struct lruvec *lruvec,struct scan_control *sc,int swappiness) +{ +int type; +int scanned; +int reclaimed; +LIST_HEAD (list); +struct folio *folio; +enum vm_event_item item; +struct reclaim_stat stat; +struct mem_cgroup *memcg =lruvec_memcg (lruvec); +struct pglist_data *pgdat =lruvec_pgdat (lruvec); + +spin_lock_irq (&lruvec->lru_lock); + + scanned =isolate_folios (lruvec, sc, swappiness, &type, &list); + +if (try_to_inc_min_seq (lruvec, swappiness)) + scanned++; + +if (get_nr_gens (lruvec, !swappiness) ==MIN_NR_GENS ) + scanned =0 ; + +spin_unlock_irq (&lruvec->lru_lock); + +if (list_empty (&list)) +return scanned; + + reclaimed =shrink_page_list (&list, pgdat, sc, &stat, false); + +/* +* To avoid livelock, don't add rejected pages back to the same lists +* they were isolated from. See lru_gen_add_folio(). +*/ +list_for_each_entry (folio, &list, lru) { +folio_clear_referenced (folio); +folio_clear_workingset (folio); + +if (folio_test_reclaim (folio) && + (folio_test_dirty (folio) ||folio_test_writeback (folio))) +folio_clear_active (folio); +else +folio_set_active (folio); + } + +spin_lock_irq (&lruvec->lru_lock); + +move_pages_to_lru (lruvec, &list); + + item =current_is_kswapd () ?PGSTEAL_KSWAPD :PGSTEAL_DIRECT ; +if (!cgroup_reclaim (sc)) +__count_vm_events (item, reclaimed); +__count_memcg_events (memcg, item, reclaimed); +__count_vm_events (PGSTEAL_ANON + type, reclaimed); + +spin_unlock_irq (&lruvec->lru_lock); + +mem_cgroup_uncharge_list (&list); +free_unref_page_list (&list); + + sc->nr_reclaimed += reclaimed; + +return scanned; +} + +static long get_nr_to_scan (struct lruvec *lruvec,struct scan_control *sc,bool can_swap) +{ +bool need_aging; +long nr_to_scan; +struct mem_cgroup *memcg =lruvec_memcg (lruvec); +DEFINE_MAX_SEQ (lruvec); +DEFINE_MIN_SEQ (lruvec); + +if (mem_cgroup_below_min (memcg) || + (mem_cgroup_below_low (memcg) && !sc->memcg_low_reclaim)) +return 0 ; + + nr_to_scan =get_nr_evictable (lruvec, max_seq, min_seq, can_swap, &need_aging); +if (!nr_to_scan) +return 0 ; + +/* reset the priority if the target has been met */ + nr_to_scan >>= sc->nr_reclaimed < sc->nr_to_reclaim ? sc->priority :DEF_PRIORITY ; + +if (!mem_cgroup_online (memcg)) + nr_to_scan++; + +if (!nr_to_scan) +return 0 ; + +if (!need_aging) +return nr_to_scan; + +/* leave the work to lru_gen_age_node() */ +if (current_is_kswapd ()) +return 0 ; + +/* try other memcgs before going to the aging path */ +if (!cgroup_reclaim (sc) && !sc->force_deactivate) { + sc->skipped_deactivate = true; +return 0 ; + } + +inc_max_seq (lruvec, max_seq, can_swap); + +return nr_to_scan; +} + +static void lru_gen_shrink_lruvec (struct lruvec *lruvec,struct scan_control *sc) +{ +struct blk_plug plug; +long scanned =0 ; + +lru_add_drain (); + +blk_start_plug (&plug); + +while (true) { +int delta; +int swappiness; +long nr_to_scan; + +if (sc->may_swap) + swappiness =get_swappiness (lruvec, sc); +else if (!cgroup_reclaim (sc) &&get_swappiness (lruvec, sc)) + swappiness =1 ; +else + swappiness =0 ; + + nr_to_scan =get_nr_to_scan (lruvec, sc, swappiness); +if (!nr_to_scan) +break ; + + delta =evict_folios (lruvec, sc, swappiness); +if (!delta) +break ; + + scanned += delta; +if (scanned >= nr_to_scan) +break ; + +cond_resched (); + } + +blk_finish_plug (&plug); +} +/****************************************************************************** * initialization ******************************************************************************/ @@ -3041,6 +3804,16 @@ static int __init init_lru_gen(void) };late_initcall (init_lru_gen); +#else + +static void lru_gen_age_node (struct pglist_data *pgdat,struct scan_control *sc) +{ +} + +static void lru_gen_shrink_lruvec (struct lruvec *lruvec,struct scan_control *sc) +{ +} +#endif /* CONFIG_LRU_GEN */ static void shrink_lruvec (struct lruvec *lruvec,struct scan_control *sc)@@ -3054,6 +3827,11 @@ static void shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)struct blk_plug plug;bool scan_adjusted; +if (lru_gen_enabled ()) { +lru_gen_shrink_lruvec (lruvec, sc); +return ; + } +get_scan_count (lruvec, sc, nr);/* Record the original scan target for proportional adjustments later */ @@ -3558,6 +4336,9 @@ static void snapshot_refaults(struct mem_cgroup *target_memcg, pg_data_t *pgdat)struct lruvec *target_lruvec;unsigned long refaults; +if (lru_gen_enabled ()) +return ; + target_lruvec =mem_cgroup_lruvec (target_memcg, pgdat); refaults =lruvec_page_state (target_lruvec,WORKINGSET_ACTIVATE_ANON ); target_lruvec->refaults[0 ] = refaults;@@ -3922,12 +4703,17 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, }#endif -static void age_active_anon (struct pglist_data *pgdat, +static void kswapd_age_node (struct pglist_data *pgdat,struct scan_control *sc) {struct mem_cgroup *memcg;struct lruvec *lruvec; +if (lru_gen_enabled ()) { +lru_gen_age_node (pgdat, sc); +return ; + } +if (!can_age_anon_pages (pgdat, sc))return ;@@ -4247,12 +5033,11 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int highest_zoneidx) sc.may_swap = !nr_boost_reclaim;/* -* Do some background aging of the anon list, to give -* pages a chance to be referenced before reclaiming. All -* pages are rotated regardless of classzone as this is -* about consistent aging. +* Do some background aging, to give pages a chance to be +* referenced before reclaiming. All pages are rotated +* regardless of classzone as this is about consistent aging. */ -age_active_anon (pgdat, &sc); +kswapd_age_node (pgdat, &sc);/* * If we're getting trouble reclaiming, start doing writepage mm/workingset.c +18
-2@@ -187,7 +187,6 @@ static unsigned int bucket_order __read_mostly;static void *pack_shadow (int memcgid,pg_data_t *pgdat,unsigned long eviction,bool workingset) { - eviction >>= bucket_order; eviction &=EVICTION_MASK ; eviction = (eviction <<MEM_CGROUP_ID_SHIFT ) | memcgid; eviction = (eviction <<NODES_SHIFT ) | pgdat->node_id;@@ -212,10 +211,107 @@ static void unpack_shadow(void *shadow, int *memcgidp, pg_data_t **pgdat, *memcgidp = memcgid; *pgdat =NODE_DATA (nid); - *evictionp = entry << bucket_order; + *evictionp = entry; *workingsetp = workingset; } +#ifdef CONFIG_LRU_GEN + +static void *lru_gen_eviction (struct folio *folio) +{ +int hist; +unsigned long token; +unsigned long min_seq; +struct lruvec *lruvec; +struct lru_gen_struct *lrugen; +int type =folio_is_file_lru (folio); +int delta =folio_nr_pages (folio); +int refs =folio_lru_refs (folio); +int tier =lru_tier_from_refs (refs); +struct mem_cgroup *memcg =folio_memcg (folio); +struct pglist_data *pgdat =folio_pgdat (folio); + +BUILD_BUG_ON (LRU_GEN_WIDTH +LRU_REFS_WIDTH >BITS_PER_LONG -EVICTION_SHIFT );
mm/workingset.c static void unpack_shadow(void *shadow, int *memcgidp, pg_data_t **pgdat, · line 244
Quoted 10 lines
mm/workingset.c +10
-0+ + lruvec =mem_cgroup_lruvec (memcg, pgdat); + lrugen = &lruvec->lrugen; + min_seq =READ_ONCE (lrugen->min_seq[type]); + token = (min_seq <<LRU_REFS_WIDTH ) |max (refs -1 ,0 ); + + hist =lru_hist_from_seq (min_seq); +atomic_long_add (delta, &lrugen->evicted[hist][type][tier]); + +return pack_shadow (mem_cgroup_id (memcg), pgdat, token, refs);
pack_shadow pass refs rather than PageWorkingset(page), is it done on purpose?
In my opinion, PageWorkingset can be removed due to failed to reclaim
Quoted 102 lines
mm/workingset.c +80
-0+} + +static void lru_gen_refault (struct folio *folio,void *shadow) +{ +int hist, tier, refs; +int memcg_id; +bool workingset; +unsigned long token; +unsigned long min_seq; +struct lruvec *lruvec; +struct lru_gen_struct *lrugen; +struct mem_cgroup *memcg; +struct pglist_data *pgdat; +int type =folio_is_file_lru (folio); +int delta =folio_nr_pages (folio); + +unpack_shadow (shadow, &memcg_id, &pgdat, &token, &workingset); + +if (folio_pgdat (folio) != pgdat) +return ; + +/* see the comment in folio_lru_refs() */ + refs = (token & (BIT (LRU_REFS_WIDTH ) -1 )) + workingset; + tier =lru_tier_from_refs (refs); + +rcu_read_lock (); + memcg =folio_memcg_rcu (folio); +if (mem_cgroup_id (memcg) != memcg_id) + goto unlock; + + lruvec =mem_cgroup_lruvec (memcg, pgdat); + lrugen = &lruvec->lrugen; + min_seq =READ_ONCE (lrugen->min_seq[type]); + + token >>=LRU_REFS_WIDTH ; +if (token != (min_seq & (EVICTION_MASK >>LRU_REFS_WIDTH ))) + goto unlock; + + hist =lru_hist_from_seq (min_seq); +atomic_long_add (delta, &lrugen->refaulted[hist][type][tier]); +mod_lruvec_state (lruvec,WORKINGSET_REFAULT_BASE + type, delta); + +/* +* Count the following two cases as stalls: +* 1. For pages accessed through page tables, hotter pages pushed out +* hot pages which refaulted immediately. +* 2. For pages accessed through file descriptors, numbers of accesses +* might have been beyond the limit. +*/ +if (lru_gen_in_fault () || refs ==BIT (LRU_REFS_WIDTH )) { +folio_set_workingset (folio); +mod_lruvec_state (lruvec,WORKINGSET_RESTORE_BASE + type, delta); + } +unlock: +rcu_read_unlock (); +} + +#else + +static void *lru_gen_eviction (struct folio *folio) +{ +return NULL ; +} + +static void lru_gen_refault (struct folio *folio,void *shadow) +{ +} + +#endif /* CONFIG_LRU_GEN */ +/** * workingset_age_nonresident - age non-resident entries as LRU ages * @lruvec: the lruvec that was aged @@ -264,10 +360,14 @@ void *workingset_eviction(struct folio *folio, struct mem_cgroup *target_memcg)VM_BUG_ON_FOLIO (folio_ref_count (folio), folio);VM_BUG_ON_FOLIO (!folio_test_locked (folio), folio); +if (lru_gen_enabled ()) +return lru_gen_eviction (folio); + lruvec =mem_cgroup_lruvec (target_memcg, pgdat);/* XXX: target_memcg can be NULL, go through lruvec */ memcgid =mem_cgroup_id (lruvec_memcg (lruvec)); eviction =atomic_long_read (&lruvec->nonresident_age); + eviction >>= bucket_order;workingset_age_nonresident (lruvec,folio_nr_pages (folio));return pack_shadow (memcgid, pgdat, eviction,folio_test_workingset (folio));@@ -298,7 +398,13 @@ void workingset_refault(struct folio *folio, void *shadow)int memcgid;long nr; +if (lru_gen_enabled ()) { +lru_gen_refault (folio, shadow); +return ; + } +unpack_shadow (shadow, &memcgid, &pgdat, &eviction, &workingset); + eviction <<= bucket_order;rcu_read_lock ();/*
Re: [PATCH v11 06/14] mm: multi-gen LRU: minimal implementation
Quoted 1,411 lines · zhong jiang
Quoted 1,396 lines · Yu Zhao
To avoid confusion, the terms "promotion" and "demotion" will be applied to the multi-gen LRU, as a new convention; the terms "activation" and "deactivation" will be applied to the active/inactive LRU, as usual.
The aging produces young generations. Given an lruvec, it increments
max_seqwhen max_seq-min_seq+1 approachesMIN_NR_GENS. The aging promotes hot pages to the youngest generation when it finds them accessed through page tables; the demotion of cold pages happens consequently when it incrementsmax_seq. The aging has the complexityO(nr_hot_pages), since it is only interested in hot pages. Promotion in the aging path does not involve any LRU list operations, only the updates of the gen counter and lrugen->nr_pages[]; demotion, unless as the result of the increment ofmax_seq, requires LRU list operations, e.g.,lru_deactivate_fn().The eviction consumes old generations. Given an lruvec, it increments
min_seqwhen the lists indexed by min_seq%MAX_NR_GENS become empty. A feedback loop modeled after the PID controller monitors refaults over anon and file types and decides which type to evict when both types are available from the same generation.Each generation is divided into multiple tiers. Tiers represent different ranges of numbers of accesses through file descriptors. A page accessed N times through file descriptors is in tier
order_base_2(N). Tiers do not have dedicated lrugen->lists[], only bits infolio->flags. In contrast to moving across generations, which requires the LRU lock, moving across tiers only involves operations onfolio->flags. The feedback loop also monitors refaults over all tiers and decides when to protect pages in which tiers (N>1), using the first tier (N=0,1) as a baseline. The first tier contains single-use unmapped clean pages, which are most likely the best choices. The eviction moves a page to the next generation, i.e., min_seq+1, if the feedback loop decides so. This approach has the following advantages:
1. It removes the cost of activation in the buffered access path by inferring whether pages accessed multiple times through file descriptors are statistically hot and thus worth protecting in the eviction path.
2. It takes pages accessed through page tables into account and avoids overprotecting pages accessed multiple times through file descriptors. (Pages accessed through page tables are in the first tier, since N=0.)
3. More tiers provide better protection for pages accessed more than twice through file descriptors, when under heavy buffered I/O workloads.Server benchmark results:
Single workload:
fio (buffered I/O): +[40, 42]%
IOPS BW
5.18-rc1: 2463k 9621MiB/s
patch1-6: 3484k 13.3GiB/sSingle workload:
memcached (anon): +[44, 46]%
Ops/sec KB/sec
5.18-rc1: 771403.27 30004.17
patch1-6: 1120643.70 43588.06Configurations:
CPU: two Xeon 6154
Mem: total 256GNode 1 was only used as a ram disk to reduce the variance in the results.
patch
drivers/block/brd.c<<EOF
99,100c99,100
<gfp_flags=GFP_NOIO|__GFP_ZERO|__GFP_HIGHMEM;
< page =alloc_page(gfp_flags);
---
>gfp_flags=GFP_NOIO|__GFP_ZERO|__GFP_HIGHMEM|__GFP_THISNODE; > page =alloc_pages_node(1, gfp_flags, 0);
EOFcat >>/etc/systemd/system.conf <<EOF
CPUAffinity=numa
NUMAPolicy=bind
NUMAMask=0
EOFcat >>/etc/memcached.conf <<EOF
-m 184320
-s/var/run/memcached/memcached.sock
-a 0766
-t 36
-B binary
EOFcat
fio.sh
modprobe brd rd_nr=1 rd_size=113246208
swapoff -a
mkfs.ext4/dev/ram0
mount -t ext4/dev/ram0/mntmkdir
/sys/fs/cgroup/user.slice/test
echo 38654705664 >/sys/fs/cgroup/user.slice/test/memory.max echo $$ >/sys/fs/cgroup/user.slice/test/cgroup.procs fio -name=mglru --numjobs=72 --directory=/mnt --size=1408m \
--buffered=1 --ioengine=io_uring --iodepth=128 \
--iodepth_batch_submit=32 --iodepth_batch_complete=32 \
--rw=randread --random_distribution=random --norandommap \
--time_based --ramp_time=10m --runtime=5m --group_reportingcat
memcached.sh
modprobe brd rd_nr=1 rd_size=113246208
swapoff -a
mkswap/dev/ram0
swapon/dev/ram0
memtier_benchmark-S/var/run/memcached/memcached.sock\
-Pmemcache_binary-n allkeys --key-minimum=1 \
--key-maximum=65000000 --key-pattern=P:P -c 1 -t 36 \
--ratio 1:0 --pipeline 8 -d 2000
memtier_benchmark-S/var/run/memcached/memcached.sock\
-Pmemcache_binary-n allkeys --key-minimum=1 \
--key-maximum=65000000 --key-pattern=R:R -c 1 -t 36 \
--ratio 0:1 --pipeline 8 --randomize --distinct-client-seedClient benchmark results:
kswapd profiles:
5.18-rc1
40.53%page_vma_mapped_walk
20.37%lzo1x_1_do_compress(real work) 6.99%do_raw_spin_lock
3.93%_raw_spin_unlock_irq
2.08%vma_interval_tree_subtree_search2.06%vma_interval_tree_iter_next1.95%folio_referenced_one
1.93%anon_vma_interval_tree_iter_first1.51%ptep_clear_flush
1.35%__anon_vma_interval_tree_subtree_searchpatch1-6
35.99%lzo1x_1_do_compress(real work) 19.40%page_vma_mapped_walk
6.31%_raw_spin_unlock_irq
3.95%do_raw_spin_lock
2.39%anon_vma_interval_tree_iter_first2.25%ptep_clear_flush
1.92%__anon_vma_interval_tree_subtree_search1.70%folio_referenced_one
1.68%__zram_bvec_write
1.43%anon_vma_interval_tree_iter_nextConfigurations:
CPU: single Snapdragon 7c
Mem: total 4GChrome OS MemoryPressure [1]
[1] https://chromium.googlesource.com/chromiumos/platform/tast-tests/
Signed-off-by Yu Zhao <[email protected]>Acked-by Brian Geffon <[email protected]>Acked-by Steven Barrett <[email protected]>Acked-by Suleiman Souhlal <[email protected]>Tested-by Daniel Byrne <[email protected]>Tested-by Donald Carr <[email protected]>Tested-by Holger Hoffstätte <[email protected]>Tested-by Konstantin Kharlamov <[email protected]>Tested-by Shuang Zhai <[email protected]>Tested-by Sofia Trinh <[email protected]>Tested-by Vaibhav Jain <[email protected]>---
include/linux/mm_inline.h | 36 ++ include/linux/mmzone.h | 42 ++include/linux/page-flags-layout.h| 5 +-
kernel/bounds.c | 2 +
mm/Kconfig | 11 +
mm/swap.c | 39 ++
mm/vmscan.c | 799 +++++++++++++++++++++++++++++- mm/workingset.c | 110 +++-8 files changed, 1034 insertions(+), 10 deletions(-)include/linux/mm_inline.h +19
-0@@ -119,6 +119,33 @@ static inline int lru_gen_from_seq(unsigned long seq) return seq % MAX_NR_GENS; } +static inline int lru_hist_from_seq(unsigned long seq) +{ + return seq % NR_HIST_GENS; +} + +static inline int lru_tier_from_refs (int refs) +{ + VM_WARN_ON_ONCE(refs > BIT(LRU_REFS_WIDTH)); + + /* see the comment in folio_lru_refs() */ + return order_base_2(refs + 1); +} + +static inline int folio_lru_refs (struct folio *folio) +{ + unsigned long flags = READ_ONCE(folio->flags); + bool workingset = flags & BIT(PG_workingset); + + /* +* Return the number of accesses beyond PG_referenced, i.e., N-1 if the
+ * total number of accesses is N>1, since N=0,1 both map to the first
+ * tier.lru_tier_from_refs()will account for this off-by-one. Also see
+ * the comment onMAX_NR_TIERS.
+ */
+ return ((flags &LRU_REFS_MASK) >>LRU_REFS_PGOFF) + workingset;include/linux/mm_inline.h +2
-0+} +static inline int
folio_lru_gen(struct folio *folio){
unsigned long flags =READ_ONCE(folio->flags);include/linux/mm_inline.h +0
-0@@ -171,6 +198,15 @@ static inline void lru_gen_update_size(structlruvec *lruvec, struct folio *foli
__update_lru_size(lruvec, lru, zone, -delta);
return;
}
+
+ /* promotion */
+ if (!lru_gen_is_active(lruvec,old_gen) &&lru_gen_is_active(lruvec, new_gen)) {
+__update_lru_size(lruvec, lru, zone, -delta);
+__update_lru_size(lruvec, lru + LRU_ACTIVE, zone, delta);
+ }
+
+ /* demotion requires isolation, e.g.,lru_deactivate_fn()*/
+ VM_WARN_ON_ONCE(lru_gen_is_active(lruvec,old_gen) &&
!lru_gen_is_active(lruvec,new_gen));
}
static inline boollru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming)include/linux/mmzone.h +0
-0@@ -348,6 +348,29 @@ enum lruvec_flags { #define MIN_NR_GENS 2U #define MAX_NR_GENS 4U+/*
+ * Each generation is divided into multiple tiers. Tiers represent different
+ * ranges of numbers of accesses through file descriptors. A page accessed N
+ * times through file descriptors is in tierorder_base_2(N). A page in the
+ * first tier (N=0,1) is marked byPG_referencedunless it was faulted in
+ * though page tables or read ahead. A page in any other tier (N>1) is marked
+ * byPG_referencedandPG_workingset. This implies a minimum of two tiers is
+ * supported without using additional bits infolio->flags.
+ *
+ * In contrast to moving across generations which requires the LRU lock, moving
+ * across tiers only involves atomic operations onfolio->flagsand therefore
+ * has a negligible cost in the buffered access path. In the eviction path,
+ * comparisons of refaulted/(evicted+protected) from the first tier and the
+ * rest infer whether pages accessed multiple times through file descriptors
+ * are statistically hot and thus worth protecting.
+ *
+ *MAX_NR_TIERSis set to 4 so that the multi-gen LRU can support twice the
+ * number of categories of the active/inactive LRU when keeping track of
+ * accesses through file descriptors. It uses MAX_NR_TIERS-2 spare bits in
+ *folio->flags(LRU_REFS_MASK).
+ */include/linux/mmzone.h +2
-0+#define MAX_NR_TIERS 4U +#ifndef
__GENERATING_BOUNDS_H
struct lruvec;include/linux/mmzone.h +0
-0@@ -362,6 +385,16 @@ enum { LRU_GEN_FILE, };+#define MIN_LRU_BATCH
BITS_PER_LONGinclude/linux/mmzone.h +9
-0+#define MAX_LRU_BATCH (MIN_LRU_BATCH * 128) + +/* whether to keep historical stats from evicted generations */ +#ifdef CONFIG_LRU_GEN_STATS +#define NR_HIST_GENS MAX_NR_GENS +#else +#define NR_HIST_GENS 1U +#endif +/*
* The youngest generation number is stored inmax_seqfor both anon and file
* types as they are aged on an equal footing. The oldest generation numbers areinclude/linux/mmzone.h +9
-0@@ -384,6 +417,15 @@ struct lru_gen_struct { struct list_head lists[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES]; /* the sizes of the above lists */ long nr_pages[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES]; + /* the exponential moving average of refaulted */ + unsigned long avg_refaulted[ANON_AND_FILE][MAX_NR_TIERS]; + /* the exponential moving average of evicted+protected */ + unsigned long avg_total[ANON_AND_FILE][MAX_NR_TIERS]; + /* the first tier doesn't need protection, hence the minus one */ + unsigned long protected[NR_HIST_GENS ][ANON_AND_FILE ][MAX_NR_TIERS -1 ]; + /* can be modified without holding the LRU lock */ + atomic_long_t evicted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS]; + atomic_long_t refaulted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS]; }; void lru_gen_init_lruvec(struct lruvec *lruvec);include/linux/page-flags-layout.h +0
-0@@ -106,7 +106,10 @@ #error "Not enough bits in page flags" #endif-#define LRU_REFS_WIDTH 0
include/linux/page-flags-layout.h +2
-0+/* see the comment on MAX_NR_TIERS */ +#define LRU_REFS_WIDTH min(__LRU_REFS_WIDTH, BITS_PER_LONG -
NR_PAGEFLAGS- \
+ZONES_WIDTH-LRU_GEN_WIDTH-SECTIONS_WIDTH- \
+NODES_WIDTH-KASAN_TAG_WIDTH-LAST_CPUPID_WIDTH) #endif
#endif /*_LINUX_PAGE_FLAGS_LAYOUT*/kernel/bounds.c +2
-0@@ -24,8 +24,10 @@ int main(void) DEFINE(SPINLOCK_SIZE, sizeof(spinlock_t)); #ifdef CONFIG_LRU_GEN DEFINE(LRU_GEN_WIDTH, order_base_2(MAX_NR_GENS + 1)); + DEFINE(__LRU_REFS_WIDTH, MAX_NR_TIERS - 2); #else DEFINE(LRU_GEN_WIDTH, 0); + DEFINE(__LRU_REFS_WIDTH, 0); #endif /* End of constants */diff --git a/mm/Kconfig b/mm/Kconfig
indexe62bd501082b..0aeacbd3361c100644mm/Kconfig +0
-0@@ -909,6 +909,7 @@ config ANON_VMA_NAMEarea from being merged with adjacent virtual memory areas due to the difference in their name.
+# multi-gen LRU {
configLRU_GEN
bool "Multi-Gen LRU"
depends on MMUmm/Kconfig +0
-0@@ -917,6 +918,16 @@ config LRU_GENhelp
A high performance LRU implementation to overcommit memory. +configLRU_GEN_STATS
+ bool "Full stats for debugging"
+ depends onLRU_GEN
+ help
+ Do not enable this option unless you plan to look at historical stats
+ from evicted generations for debugging purpose.mm/Kconfig +4
-0+ + This option has a per-memcg and per-node memory overhead. +# } +source "
mm/damon/Kconfig"
endmenumm/swap.c +0
-0@@ -405,6 +405,40 @@ static void __lru_cache_activate_folio(struct folio *folio) local_unlock(&lru_pvecs.lock); }+#ifdef
CONFIG_LRU_GENmm/swap.c +33
-0+static void folio_inc_refs (struct folio *folio) +{ + unsigned long new_flags, old_flags = READ_ONCE(folio->flags); + + if (folio_test_unevictable(folio)) + return; + + if (!folio_test_referenced(folio)) { + folio_set_referenced(folio); + return; + } + + if (!folio_test_workingset(folio)) { + folio_set_workingset(folio); + return; + } + + /* see the comment on MAX_NR_TIERS */ + do { + new_flags = old_flags & LRU_REFS_MASK; + if (new_flags == LRU_REFS_MASK) + break; + + new_flags += BIT(LRU_REFS_PGOFF); + new_flags |= old_flags & ~LRU_REFS_MASK; + } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); +} +#else +static void folio_inc_refs (struct folio *folio) +{ +} +#endif /* CONFIG_LRU_GEN */ +/*
* Mark a page as having seen activity.
*mm/swap.c +5
-0@@ -417,6 +451,11 @@ static void __lru_cache_activate_folio(structfolio *folio) */ void folio_mark_accessed(struct folio *folio) { + if (lru_gen_enabled()) { + folio_inc_refs(folio); + return; + } + if (!folio_test_referenced(folio)) { folio_set_referenced(folio);} else if (
folio_test_unevictable(folio)) {mm/vmscan.c +2
-1@@ -1275,9 +1275,11 @@ static int __remove_mapping(struct address_space *mapping, struct folio *folio, if (folio_test_swapcache(folio)) { swp_entry_t swap = folio_swap_entry(folio); - mem_cgroup_swapout(folio, swap); + + /* get a shadow entry before mem_cgroup_swapout() clears
folio_memcg()*/
if (reclaimed && !mapping_exiting(mapping)) shadow =workingset_eviction(folio, target_memcg); +mem_cgroup_swapout(folio, swap);__delete_from_swap_cache(&folio->page, swap, shadow);xa_unlock_irq(&mapping->i_pages);put_swap_page(&folio->page, swap);mm/vmscan.c +12
-0@@ -2649,6 +2651,9 @@ static void prepare_scan_count(pg_data_t *pgdat, struct scan_control *sc) unsigned long file; struct lruvec *target_lruvec; + if (lru_gen_enabled()) + return; + target_lruvec = mem_cgroup_lruvec(sc->target_mem_cgroup, pgdat); /*@@ -2974,6 +2979,17 @@ static bool can_age_anon_pages(structpglist_data *pgdat, * shorthand helpers ******************************************************************************/ +#define LRU_REFS_FLAGS (BIT(PG_referenced) | BIT(PG_workingset)) + +#define DEFINE_MAX_SEQ (lruvec) \ + unsigned long max_seq = READ_ONCE((lruvec)->lrugen.max_seq) + +#define DEFINE_MIN_SEQ (lruvec) \ + unsigned long min_seq[ANON_AND_FILE] = { \ + READ_ONCE((lruvec)->lrugen.min_seq[LRU_GEN_ANON]), \ + READ_ONCE((lruvec)->lrugen.min_seq[LRU_GEN_FILE]), \ + } + #define for_each_gen_type_zone(gen, type, zone) \ for ((gen) = 0; (gen) < MAX_NR_GENS; (gen)++) \for ((type) = 0; (type) <
ANON_AND_FILE; (type)++) \mm/vmscan.c +21
-0@@ -2999,6 +3015,753 @@ static struct lruvec __maybe_unused *get_lruvec(struct mem_cgroup *memcg, int ni return pgdat ? &pgdat->__lruvec : NULL; } +static int get_swappiness(struct lruvec *lruvec, struct scan_control *sc) +{ + struct mem_cgroup *memcg = lruvec_memcg(lruvec); + struct pglist_data *pgdat = lruvec_pgdat(lruvec); + + if (!can_demote(pgdat->node_id, sc) && + mem_cgroup_get_nr_swap_pages(memcg) < MIN_LRU_BATCH) + return 0; + + return mem_cgroup_swappiness(memcg); +} + +static int get_nr_gens (struct lruvec *lruvec,int type) +{ + return lruvec->lrugen.max_seq - lruvec->lrugen.min_seq[type] + 1; +} + +static bool __maybe_unused seq_is_valid (struct lruvec *lruvec) +{ + /* see the comment on lru_gen_struct */ + return get_nr_gens(lruvec, LRU_GEN_FILE) >= MIN_NR_GENS && + get_nr_gens(lruvec, LRU_GEN_FILE) <= get_nr_gens(lruvec,
LRU_GEN_ANON) &&
+get_nr_gens(lruvec, LRU_GEN_ANON)<=MAX_NR_GENS;mm/vmscan.c +5
-0+} + +/****************************************************************************** +* refault feedback loop +******************************************************************************/
mm/vmscan.c +3
-0+ +/* +* A feedback loop based on Proportional-Integral-Derivative (PID) controller.
+ *
+ * The P term is refaulted/(evicted+protected) from a tier in the generation
+ * currently being evicted; the I term is the exponential moving average of the
+ * P term over the generations previously evicted, using the smoothing factor
+ * 1/2; the D term isn't supported.
+ *
+ * The setpoint (SP) is always the first tier of one type; the process variable
+ * (PV) is either any tier of the other type or any other tier of the same
+ * type.
+ *
+ * The error is the difference between the SP and the PV; the correction is
+ * turn off protection when SP>PV or turn on protection when SP<PV.
+ *
+ * For future optimizations:
+ * 1. The D term may discount the other two terms over time so that long-lived
+ * generations can resist stale information.
+ */mm/vmscan.c +7
-0+struct ctrl_pos { + unsigned long refaulted; + unsigned long total; + int gain; +}; + +static void read_ctrl_pos(struct lruvec *lruvec, int type, int tier,int gain,
+ structctrl_pos*pos)mm/vmscan.c +14
-0+{ + struct lru_gen_struct *lrugen = &lruvec->lrugen; + int hist = lru_hist_from_seq(lrugen->min_seq[type]); + + pos->refaulted = lrugen->avg_refaulted[type][tier] + +atomic_long_read (&lrugen->refaulted[hist][type][tier]); + pos->total = lrugen->avg_total[type][tier] + +atomic_long_read (&lrugen->evicted[hist][type][tier]); + if (tier) + pos->total += lrugen->protected[hist][type][tier - 1]; + pos->gain = gain; +} + +static void reset_ctrl_pos(struct lruvec *lruvec, int type, boolcarryover)
+{
+ int hist, tier;
+ structlru_gen_struct*lrugen =&lruvec->lrugen;
+ bool clear = carryover ?NR_HIST_GENS== 1 :NR_HIST_GENS> 1; + unsigned long seq = carryover ? lrugen->min_seq[type] :lrugen->max_seq+ 1;mm/vmscan.c +36
-0+ + lockdep_assert_held(&lruvec->lru_lock); + + if (!carryover && !clear) + return; + + hist = lru_hist_from_seq(seq); + + for (tier = 0; tier < MAX_NR_TIERS; tier++) { + if (carryover) { + unsigned long sum; + + sum = lrugen->avg_refaulted[type][tier] + +atomic_long_read (&lrugen->refaulted[hist][type][tier]); + WRITE_ONCE(lrugen->avg_refaulted[type][tier], sum / 2); + + sum = lrugen->avg_total[type][tier] + +atomic_long_read (&lrugen->evicted[hist][type][tier]); + if (tier) + sum += lrugen->protected[hist][type][tier - 1]; + WRITE_ONCE(lrugen->avg_total[type][tier], sum / 2); + } + + if (clear) { +atomic_long_set (&lrugen->refaulted[hist][type][tier],0 ); +atomic_long_set (&lrugen->evicted[hist][type][tier],0 ); + if (tier) +WRITE_ONCE (lrugen->protected[hist][type][tier -1 ],0 ); + } + } +} + +static bool positive_ctrl_err (struct ctrl_pos *sp,struct ctrl_pos *pv) +{ + /* +* Return true if the PV has a limited number of refaults or a lower
+ * refaulted/total than the SP.
+ */
+ returnpv->refaulted<MIN_LRU_BATCH||
+pv->refaulted* (sp->total+MIN_LRU_BATCH) *sp->gain<= + (sp->refaulted+ 1) *pv->total*pv->gain;mm/vmscan.c +5
-0+} + +/****************************************************************************** +* the aging +******************************************************************************/
mm/vmscan.c +2
-0+ +static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio,bool reclaiming)
mm/vmscan.c +12
-0+{ + int type = folio_is_file_lru(folio); + struct lru_gen_struct *lrugen = &lruvec->lrugen; + int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]); + unsigned long new_flags, old_flags = READ_ONCE(folio->flags); + + VM_WARN_ON_ONCE_FOLIO(!(old_flags & LRU_GEN_MASK), folio); + + do { + new_gen = (old_gen + 1) % MAX_NR_GENS; + + new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_MASK |
LRU_REFS_FLAGS);
+new_flags|= (new_gen+ 1UL) <<LRU_GEN_PGOFF; + /* forfolio_end_writeback()*/
+ if (reclaiming)
+new_flags|=BIT(PG_reclaim);
+ } while (!try_cmpxchg(&folio->flags,&old_flags,new_flags));mm/vmscan.c +42
-0+ + lru_gen_update_size(lruvec, folio, old_gen, new_gen); + + return new_gen; +} + +static void inc_min_seq (struct lruvec *lruvec,int type) +{ + struct lru_gen_struct *lrugen = &lruvec->lrugen; + + reset_ctrl_pos(lruvec, type, true); + WRITE_ONCE(lrugen->min_seq[type], lrugen->min_seq[type] + 1); +} + +static bool try_to_inc_min_seq (struct lruvec *lruvec,bool can_swap) +{ + int gen, type, zone; + bool success = false; + struct lru_gen_struct *lrugen = &lruvec->lrugen; + DEFINE_MIN_SEQ(lruvec); + + VM_WARN_ON_ONCE(!seq_is_valid(lruvec)); + + /* find the oldest populated generation */ + for (type = !can_swap; type < ANON_AND_FILE; type++) { + while (min_seq[type] + MIN_NR_GENS <= lrugen->max_seq) { + gen = lru_gen_from_seq(min_seq[type]); + + for (zone = 0; zone < MAX_NR_ZONES; zone++) { + if (!list_empty(&lrugen->lists[gen][type][zone])) + goto next; + } + + min_seq[type]++; + } +next: + ; + } + + /* see the comment on lru_gen_struct */ + if (can_swap) { + min_seq[LRU_GEN_ANON] = min(min_seq[LRU_GEN_ANON],min_seq[LRU_GEN_FILE]);
+ min_seq[LRU_GEN_FILE] =max(min_seq[LRU_GEN_ANON], lrugen->min_seq[LRU_GEN_FILE]);
+ }mm/vmscan.c +14
-0+ + for (type = !can_swap; type < ANON_AND_FILE; type++) { + if (min_seq[type] == lrugen->min_seq[type]) + continue; + + reset_ctrl_pos(lruvec, type, true); + WRITE_ONCE(lrugen->min_seq[type], min_seq[type]); + success = true; + } + + return success; +} + +static void inc_max_seq(struct lruvec *lruvec, unsigned long
max_seq, boolcan_swap)mm/vmscan.c +23
-0+{ + int prev, next; + int type, zone; + struct lru_gen_struct *lrugen = &lruvec->lrugen; + + spin_lock_irq(&lruvec->lru_lock); + + VM_WARN_ON_ONCE(!seq_is_valid(lruvec)); + + if (max_seq != lrugen->max_seq) + goto unlock; + + for (type = 0; type < ANON_AND_FILE; type++) { + if (get_nr_gens(lruvec, type) != MAX_NR_GENS) + continue; + + VM_WARN_ON_ONCE(type == LRU_GEN_FILE || can_swap); + + inc_min_seq(lruvec, type); + } + + /* +* Update the active/inactive LRU sizes for compatibility. Both sides of
+ * the currentmax_seqneed to be covered, since max_seq+1 can overlap
+ * with min_seq[LRU_GEN_ANON] if swapping is constrained. And if they do
+ * overlap, cold/hot inversion happens.
+ */
+ prev =lru_gen_from_seq(lrugen->max_seq - 1);
+ next =lru_gen_from_seq(lrugen->max_seq + 1);mm/vmscan.c +25
-0+ + for (type = 0; type < ANON_AND_FILE; type++) { + for (zone = 0; zone < MAX_NR_ZONES; zone++) { + enum lru_list lru = type * LRU_INACTIVE_FILE; + long delta = lrugen->nr_pages[prev][type][zone] - + lrugen->nr_pages[next][type][zone]; + + if (!delta) + continue; + + __update_lru_size(lruvec, lru, zone, delta); + __update_lru_size(lruvec, lru + LRU_ACTIVE, zone, -delta); + } + } + + for (type = 0; type < ANON_AND_FILE; type++) + reset_ctrl_pos(lruvec, type, false); + + /* make sure preceding modifications appear */ + smp_store_release(&lrugen->max_seq, lrugen->max_seq + 1); +unlock: + spin_unlock_irq(&lruvec->lru_lock); +} + +static long get_nr_evictable(struct lruvec *lruvec, unsigned long
max_seq,
+ unsigned long*min_seq, boolcan_swap, bool*need_aging)mm/vmscan.c +28
-0+{ + int gen, type, zone; + long old = 0; + long young = 0; + long total = 0; + struct lru_gen_struct *lrugen = &lruvec->lrugen; + + for (type = !can_swap; type < ANON_AND_FILE; type++) { + unsigned long seq; + + for (seq = min_seq[type]; seq <= max_seq; seq++) { + long size = 0; + + gen = lru_gen_from_seq(seq); + + for (zone = 0; zone < MAX_NR_ZONES; zone++) + size += READ_ONCE(lrugen->nr_pages[gen][type][zone]); + + total += size; + if (seq == max_seq) + young += size; + if (seq + MIN_NR_GENS == max_seq) + old += size; + } + } + + /* +* The aging tries to be lazy to reduce the overhead. On the other hand,
+ * the eviction stalls when the number of generations reaches + *MIN_NR_GENS. So ideally, there should be MIN_NR_GENS+1 generations,
+ * hence the first two if's.
+ *
+ * Also it's ideal to spread pages out evenly, meaning 1/(MIN_NR_GENS+1)
+ * of the total number of pages for each generation. A reasonable range
+ * for this average portion is [1/MIN_NR_GENS, 1/(MIN_NR_GENS+2)]. The
+ * eviction cares about the lower bound of cold pages, whereas the aging
+ * cares about the upper bound of hot pages.
+ */
+ if (min_seq[!can_swap] +MIN_NR_GENS>max_seq)
+*need_aging= true;
+ else if (min_seq[!can_swap] +MIN_NR_GENS<max_seq) +*need_aging= false;
+ else if (young *MIN_NR_GENS> total)
+*need_aging= true;
+ else if (old * (MIN_NR_GENS+ 2) < total)
+*need_aging= true;
+ else
+*need_aging= false;mm/vmscan.c +21
-0+ + return total > 0 ? total : 0; +} + +static void age_lruvec (struct lruvec *lruvec,struct scan_control *sc) +{ + bool need_aging; + long nr_to_scan; + int swappiness = get_swappiness(lruvec, sc); + struct mem_cgroup *memcg = lruvec_memcg(lruvec); + DEFINE_MAX_SEQ(lruvec); + DEFINE_MIN_SEQ(lruvec); + + VM_WARN_ON_ONCE(sc->memcg_low_reclaim); + + mem_cgroup_calculate_protection(NULL, memcg); + + if (mem_cgroup_below_min(memcg)) + return; + + nr_to_scan = get_nr_evictable(lruvec, max_seq, min_seq,swappiness,
&need_aging);
+ if (!nr_to_scan)
+ return;mm/vmscan.c +11
-0+ + nr_to_scan >>= sc->priority; + + if (!mem_cgroup_online(memcg)) + nr_to_scan++; + + if (nr_to_scan && need_aging) + inc_max_seq(lruvec, max_seq, swappiness); +} + +static void lru_gen_age_node(struct pglist_data *pgdat, struct
scan_control*sc)mm/vmscan.c +18
-0+{ + struct mem_cgroup *memcg; + + VM_WARN_ON_ONCE(!current_is_kswapd()); + + memcg = mem_cgroup_iter(NULL, NULL, NULL); + do { + struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); + + age_lruvec(lruvec, sc); + + cond_resched(); + } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL))); +} + +/****************************************************************************** +* the eviction +******************************************************************************/
mm/vmscan.c +2
-0+ +static bool sort_folio(struct lruvec *lruvec, struct folio *folio,int
tier_idx)mm/vmscan.c +24
-0+{ + bool success; + int gen = folio_lru_gen(folio); + int type = folio_is_file_lru(folio); + int zone = folio_zonenum(folio); + int delta = folio_nr_pages(folio); + int refs = folio_lru_refs(folio); + int tier = lru_tier_from_refs(refs); + struct lru_gen_struct *lrugen = &lruvec->lrugen; + + VM_WARN_ON_ONCE_FOLIO(gen >= MAX_NR_GENS, folio); + + /* unevictable */ + if (!folio_evictable(folio)) { + success = lru_gen_del_folio(lruvec, folio, true); + VM_WARN_ON_ONCE_FOLIO(!success, folio); + folio_set_unevictable(folio); + lruvec_add_folio(lruvec, folio); + __count_vm_events(UNEVICTABLE_PGCULLED, delta); + return true; + } + + /* dirtied lazyfree */ + if (type == LRU_GEN_FILE && folio_test_anon(folio) &&
folio_test_dirty(folio)) {
+ success =lru_gen_del_folio(lruvec, folio, true); +VM_WARN_ON_ONCE_FOLIO(!success, folio); +folio_set_swapbacked(folio);
+lruvec_add_folio_tail(lruvec, folio);
+ return true;
+ }mm/vmscan.c +11
-0+ + /* protected */ + if (tier > tier_idx) { + int hist = lru_hist_from_seq(lrugen->min_seq[type]); + + gen = folio_inc_gen(lruvec, folio, false); + list_move_tail(&folio->lru, &lrugen->lists[gen][type][zone]); + + WRITE_ONCE(lrugen->protected[hist][type][tier - 1], + lrugen->protected[hist][type][tier - 1] + delta); + __mod_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE + type,delta);
+ return true;
+ }mm/vmscan.c +13
-0+ + /* waiting for writeback */ + if (folio_test_locked(folio) || folio_test_writeback(folio) || + (type == LRU_GEN_FILE && folio_test_dirty(folio))) { + gen = folio_inc_gen(lruvec, folio, true); + list_move(&folio->lru, &lrugen->lists[gen][type][zone]); + return true; + } + + return false; +} + +static bool isolate_folio(struct lruvec *lruvec, struct folio*folio, struct
scan_control*sc)mm/vmscan.c +56
-0+{ + bool success; + + if (!sc->may_unmap && folio_mapped(folio)) + return false; + + if (!(sc->may_writepage && (sc->gfp_mask & __GFP_IO)) && + (folio_test_dirty(folio) || + (folio_test_anon(folio) && !folio_test_swapcache(folio)))) + return false; + + if (!folio_try_get(folio)) + return false; + + if (!folio_test_clear_lru(folio)) { + folio_put(folio); + return false; + } + + success = lru_gen_del_folio(lruvec, folio, true); + VM_WARN_ON_ONCE_FOLIO(!success, folio); + + return true; +} + +static int scan_folios (struct lruvec *lruvec,struct scan_control *sc, + int type, int tier, struct list_head *list) +{ + int gen, zone; + enum vm_event_item item; + int sorted = 0; + int scanned = 0; + int isolated = 0; + int remaining = MAX_LRU_BATCH; + struct lru_gen_struct *lrugen = &lruvec->lrugen; + struct mem_cgroup *memcg = lruvec_memcg(lruvec); + + VM_WARN_ON_ONCE(!list_empty(list)); + + if (get_nr_gens(lruvec, type) == MIN_NR_GENS) + return 0; + + gen = lru_gen_from_seq(lrugen->min_seq[type]); + + for (zone = sc->reclaim_idx; zone >= 0; zone--) { + LIST_HEAD(moved); + int skipped = 0; + struct list_head *head = &lrugen->lists[gen][type][zone]; + + while (!list_empty(head)) { + struct folio *folio = lru_to_folio(head); + int delta = folio_nr_pages(folio); + +VM_WARN_ON_ONCE_FOLIO (folio_test_unevictable (folio), folio); + VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio); + VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type,folio);
+VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);mm/vmscan.c +14
-0+ + scanned += delta; + + if (sort_folio(lruvec, folio, tier)) + sorted += delta; + else if (isolate_folio(lruvec, folio, sc)) { + list_add(&folio->lru, list); + isolated += delta; + } else { + list_move(&folio->lru, &moved); + skipped += delta; + } + + if (!--remaining || max(isolated, skipped) >=
MIN_LRU_BATCH)
+ break;
+ }mm/vmscan.c +21
-0+ + if (skipped) { + list_splice(&moved, head); + __count_zid_vm_events(PGSCAN_SKIP, zone, skipped); + } + + if (!remaining || isolated >= MIN_LRU_BATCH) + break; + } + + item = current_is_kswapd() ? PGSCAN_KSWAPD : PGSCAN_DIRECT; + if (!cgroup_reclaim(sc)) { + __count_vm_events(item, isolated); + __count_vm_events(PGREFILL, sorted); + } + __count_memcg_events(memcg, item, isolated); + __count_memcg_events(memcg, PGREFILL, sorted); + __count_vm_events(PGSCAN_ANON + type, isolated); + + /* +* There might not be eligible pages due to reclaim_idx,
may_unmapand
+ *may_writepage. Check the remaining to prevent livelock if there is no
+ * progress.
+ */
+ return isolated || !remaining ? scanned : 0;mm/vmscan.c +9
-0+} + +static int get_tier_idx (struct lruvec *lruvec,int type) +{ + int tier; + struct ctrl_pos sp, pv; + + /* +* To leave a margin for fluctuations, use a larger gain factor (1:2).
+ * This value is chosen because any other tier would have at least twice
+ * as many refaults as the first tier.
+ */
+read_ctrl_pos(lruvec, type, 0, 1, &sp);
+ for (tier = 1; tier <MAX_NR_TIERS; tier++) {
+read_ctrl_pos(lruvec, type, tier, 2, &pv);
+ if (!positive_ctrl_err(&sp, &pv))
+ break;
+ }mm/vmscan.c +5
-0+ + return tier - 1; +} + +static int get_type_to_scan(struct lruvec *lruvec, int swappiness,int
*tier_idx)mm/vmscan.c +7
-0+{ + int type, tier; + struct ctrl_pos sp, pv; + int gain[ANON_AND_FILE] = { swappiness, 200 - swappiness }; + + /* +* Compare the first tier of anon with that of file to determine which
+ * type to scan. Also need to compare other tiers of the selected type
+ * with the first tier of the other type to determine the last tier (of
+ * the selected type) to evict.
+ */
+read_ctrl_pos(lruvec, LRU_GEN_ANON, 0, gain[LRU_GEN_ANON], &sp); +read_ctrl_pos(lruvec, LRU_GEN_FILE, 0, gain[LRU_GEN_FILE], &pv); + type =positive_ctrl_err(&sp, &pv);mm/vmscan.c +14
-0+ + read_ctrl_pos(lruvec, !type, 0, gain[!type], &sp); + for (tier = 1; tier < MAX_NR_TIERS; tier++) { + read_ctrl_pos(lruvec, type, tier, gain[type], &pv); + if (!positive_ctrl_err(&sp, &pv)) + break; + } + + *tier_idx = tier - 1; + + return type; +} + +static int isolate_folios(struct lruvec *lruvec, struct scan_control*sc, int swappiness,
+ int*type_scanned, structlist_head*list)mm/vmscan.c +9
-0+{ + int i; + int type; + int scanned; + int tier = -1; + DEFINE_MIN_SEQ(lruvec); + + /* +* Try to make the obvious choice first. When anon and file are both
+ * available from the same generation, interpret swappiness 1 as file
+ * first and 200 as anon first.
+ */
+ if (!swappiness)
+ type =LRU_GEN_FILE;
+ else if (min_seq[LRU_GEN_ANON] < min_seq[LRU_GEN_FILE]) + type =LRU_GEN_ANON;
+ else if (swappiness == 1)
+ type =LRU_GEN_FILE;
+ else if (swappiness == 200)
+ type =LRU_GEN_ANON;
+ else
+ type =get_type_to_scan(lruvec, swappiness, &tier);mm/vmscan.c +19
-0+ + for (i = !swappiness; i < ANON_AND_FILE; i++) { + if (tier < 0) + tier = get_tier_idx(lruvec, type); + + scanned = scan_folios(lruvec, sc, type, tier, list); + if (scanned) + break; + + type = !type; + tier = -1; + } + + *type_scanned = type; + + return scanned; +} + +static int evict_folios(struct lruvec *lruvec, struct scan_control*sc, int swappiness)
mm/vmscan.c +30
-0+{ + int type; + int scanned; + int reclaimed; + LIST_HEAD(list); + struct folio *folio; + enum vm_event_item item; + struct reclaim_stat stat; + struct mem_cgroup *memcg = lruvec_memcg(lruvec); + struct pglist_data *pgdat = lruvec_pgdat(lruvec); + + spin_lock_irq(&lruvec->lru_lock); + + scanned = isolate_folios(lruvec, sc, swappiness, &type, &list); + + if (try_to_inc_min_seq(lruvec, swappiness)) + scanned++; + + if (get_nr_gens(lruvec, !swappiness) == MIN_NR_GENS) + scanned = 0; + + spin_unlock_irq(&lruvec->lru_lock); + + if (list_empty(&list)) + return scanned; + + reclaimed = shrink_page_list(&list, pgdat, sc, &stat, false); + + /* +* To avoid livelock, don't add rejected pages back to the same lists
+ * they were isolated from. Seelru_gen_add_folio(). + */
+list_for_each_entry(folio, &list, lru){
+folio_clear_referenced(folio);
+folio_clear_workingset(folio);mm/vmscan.c +29
-0+ + if (folio_test_reclaim(folio) && + (folio_test_dirty(folio) || folio_test_writeback(folio))) + folio_clear_active(folio); + else + folio_set_active(folio); + } + + spin_lock_irq(&lruvec->lru_lock); + + move_pages_to_lru(lruvec, &list); + + item = current_is_kswapd() ? PGSTEAL_KSWAPD : PGSTEAL_DIRECT; + if (!cgroup_reclaim(sc)) + __count_vm_events(item, reclaimed); + __count_memcg_events(memcg, item, reclaimed); + __count_vm_events(PGSTEAL_ANON + type, reclaimed); + + spin_unlock_irq(&lruvec->lru_lock); + + mem_cgroup_uncharge_list(&list); + free_unref_page_list(&list); + + sc->nr_reclaimed += reclaimed; + + return scanned; +} + +static long get_nr_to_scan(struct lruvec *lruvec, struct
scan_control*sc, boolcan_swap)mm/vmscan.c +12
-0+{ + bool need_aging; + long nr_to_scan; + struct mem_cgroup *memcg = lruvec_memcg(lruvec); + DEFINE_MAX_SEQ(lruvec); + DEFINE_MIN_SEQ(lruvec); + + if (mem_cgroup_below_min(memcg) || + (mem_cgroup_below_low(memcg) && !sc->memcg_low_reclaim)) + return 0; + + nr_to_scan = get_nr_evictable(lruvec, max_seq, min_seq,
can_swap,&need_aging);
+ if (!nr_to_scan)
+ return 0;
+
+ /* reset the priority if the target has been met */ +nr_to_scan>>=sc->nr_reclaimed<sc->nr_to_reclaim?sc->priority:DEF_PRIORITY;mm/vmscan.c +26
-0+ + if (!mem_cgroup_online(memcg)) + nr_to_scan++; + + if (!nr_to_scan) + return 0; + + if (!need_aging) + return nr_to_scan; + + /* leave the work to lru_gen_age_node() */ + if (current_is_kswapd()) + return 0; + + /* try other memcgs before going to the aging path */ + if (!cgroup_reclaim(sc) && !sc->force_deactivate) { + sc->skipped_deactivate = true; + return 0; + } + + inc_max_seq(lruvec, max_seq, can_swap); + + return nr_to_scan; +} + +static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct
scan_control*sc)mm/vmscan.c +38
-0+{ + struct blk_plug plug; + long scanned = 0; + + lru_add_drain(); + + blk_start_plug(&plug); + + while (true) { + int delta; + int swappiness; + long nr_to_scan; + + if (sc->may_swap) + swappiness = get_swappiness(lruvec, sc); + else if (!cgroup_reclaim(sc) && get_swappiness(lruvec, sc)) + swappiness = 1; + else + swappiness = 0; + + nr_to_scan = get_nr_to_scan(lruvec, sc, swappiness); + if (!nr_to_scan) + break; + + delta = evict_folios(lruvec, sc, swappiness); + if (!delta) + break; + + scanned += delta; + if (scanned >= nr_to_scan) + break; + + cond_resched(); + } + + blk_finish_plug(&plug); +} +/****************************************************************************** * initialization ******************************************************************************/
mm/vmscan.c +0
-0@@ -3041,6 +3804,16 @@ static int __init init_lru_gen(void) }; late_initcall(init_lru_gen);+#else
mm/vmscan.c +2
-0+ +static void lru_gen_age_node(struct pglist_data *pgdat, struct
scan_control*sc)mm/vmscan.c +4
-0+{ +} + +static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct
scan_control*sc)mm/vmscan.c +3
-0+{ +} +#endif /*
CONFIG_LRU_GEN*/
static voidshrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)mm/vmscan.c +6
-0@@ -3054,6 +3827,11 @@ static void shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) struct blk_plug plug; bool scan_adjusted; + if (lru_gen_enabled()) { + lru_gen_shrink_lruvec(lruvec, sc); + return; + } + get_scan_count(lruvec, sc, nr); /* Record the original scan target for proportional adjustments later */@@ -3558,6 +4336,9 @@ static void snapshot_refaults(struct mem_cgroup *target_memcg, pg_data_t *pgdat) struct lruvec *target_lruvec; unsigned long refaults; + if (lru_gen_enabled()) + return; + target_lruvec = mem_cgroup_lruvec(target_memcg, pgdat); refaults = lruvec_page_state(target_lruvec, WORKINGSET_ACTIVATE_ANON);target_lruvec->refaults[0] = refaults;
mm/vmscan.c +5
-0@@ -3922,12 +4703,17 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, } #endif -static void age_active_anon(struct pglist_data *pgdat, +static void kswapd_age_node(struct pglist_data *pgdat, struct scan_control *sc) { struct mem_cgroup *memcg; struct lruvec *lruvec; + if (lru_gen_enabled()) { + lru_gen_age_node(pgdat, sc); + return; + } + if (!can_age_anon_pages(pgdat, sc)) return; @@ -4247,12 +5033,11 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int highest_zoneidx)sc.may_swap = !nr_boost_reclaim;
/*
- * Do some background aging of the anon list, to give - * pages a chance to be referenced before reclaiming. All - * pages are rotated regardless of classzone as this is - * about consistent aging.
+ * Do some background aging, to give pages a chance to be + * referenced before reclaiming. All pages are rotated + * regardless of classzone as this is about consistent aging. */
-age_active_anon(pgdat, &sc);
+kswapd_age_node(pgdat, &sc);
/*
* If we're getting trouble reclaiming, start doing writepagemm/workingset.c +1
-2@@ -187,7 +187,6 @@ static unsigned int bucket_order __read_mostly; static void *pack_shadow(int memcgid, pg_data_t *pgdat, unsigned long eviction, bool workingset) { - eviction >>= bucket_order; eviction &= EVICTION_MASK; eviction = (eviction << MEM_CGROUP_ID_SHIFT) | memcgid; eviction = (eviction << NODES_SHIFT) | pgdat->node_id;@@ -212,10 +211,107 @@ static void unpack_shadow(void *shadow, int *memcgidp, pg_data_t **pgdat, *memcgidp = memcgid; *pgdat = NODE_DATA(nid); - *evictionp = entry << bucket_order; + *evictionp = entry; *workingsetp = workingset; }+#ifdef
CONFIG_LRU_GENmm/workingset.c +16
-0+ +static void *lru_gen_eviction(struct folio *folio) +{ + int hist; + unsigned long token; + unsigned long min_seq; + struct lruvec *lruvec; + struct lru_gen_struct *lrugen; + int type = folio_is_file_lru(folio); + int delta = folio_nr_pages(folio); + int refs = folio_lru_refs(folio); + int tier = lru_tier_from_refs(refs); + struct mem_cgroup *memcg = folio_memcg(folio); + struct pglist_data *pgdat = folio_pgdat(folio); + + BUILD_BUG_ON(LRU_GEN_WIDTH + LRU_REFS_WIDTH > BITS_PER_LONG -
EVICTION_SHIFT);mm/workingset.c static void unpack_shadow(void *shadow, int *memcgidp, pg_data_t **pgdat, · line 244
Quoted 10 lines
mm/workingset.c +10
-0+ + lruvec = mem_cgroup_lruvec(memcg, pgdat); + lrugen = &lruvec->lrugen; + min_seq = READ_ONCE(lrugen->min_seq[type]); + token = (min_seq << LRU_REFS_WIDTH) | max(refs - 1, 0); + + hist = lru_hist_from_seq(min_seq); + atomic_long_add(delta, &lrugen->evicted[hist][type][tier]); + + return pack_shadow(mem_cgroup_id(memcg), pgdat, token, refs);
pack_shadowpass refs rather than PageWorkingset(page), is it done on purpose?
Quoted 1 line
In my opinion, PageWorkingset can be removed due to failed to reclaim
look at this again, It's my fault , the workingset do not mean the orignal page status. Thanks,
Quoted 105 lines
Quoted 105 lines
mm/workingset.c +45
-0+} + +static void lru_gen_refault (struct folio *folio,void *shadow) +{ + int hist, tier, refs; + int memcg_id; + bool workingset; + unsigned long token; + unsigned long min_seq; + struct lruvec *lruvec; + struct lru_gen_struct *lrugen; + struct mem_cgroup *memcg; + struct pglist_data *pgdat; + int type = folio_is_file_lru(folio); + int delta = folio_nr_pages(folio); + + unpack_shadow(shadow, &memcg_id, &pgdat, &token, &workingset); + + if (folio_pgdat(folio) != pgdat) + return; + + /* see the comment in folio_lru_refs() */ + refs = (token & (BIT(LRU_REFS_WIDTH) - 1)) + workingset; + tier = lru_tier_from_refs(refs); + + rcu_read_lock(); + memcg = folio_memcg_rcu(folio); + if (mem_cgroup_id(memcg) != memcg_id) + goto unlock; + + lruvec = mem_cgroup_lruvec(memcg, pgdat); + lrugen = &lruvec->lrugen; + min_seq = READ_ONCE(lrugen->min_seq[type]); + + token >>= LRU_REFS_WIDTH; + if (token != (min_seq & (EVICTION_MASK >> LRU_REFS_WIDTH))) + goto unlock; + + hist = lru_hist_from_seq(min_seq); + atomic_long_add(delta, &lrugen->refaulted[hist][type][tier]); + mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + type, delta); + + /* +* Count the following two cases as stalls: +* 1. For pages accessed through page tables, hotter pages pushed out
+ * hot pages which refaulted immediately.
+ * 2. For pages accessed through file descriptors, numbers of accesses
+ * might have been beyond the limit.
+ */
+ if (lru_gen_in_fault()|| refs ==BIT(LRU_REFS_WIDTH)) { +folio_set_workingset(folio);
+mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + type, delta);
+ }mm/workingset.c +17
-0+unlock: + rcu_read_unlock(); +} + +#else + +static void *lru_gen_eviction (struct folio *folio) +{ + return NULL; +} + +static void lru_gen_refault (struct folio *folio,void *shadow) +{ +} + +#endif /* CONFIG_LRU_GEN */ +/**
*workingset_age_nonresident- age non-resident entries as LRU ages
* @lruvec: the lruvec that was agedmm/workingset.c +3
-0@@ -264,10 +360,14 @@ void *workingset_eviction(struct folio *folio, struct mem_cgroup *target_memcg) VM_BUG_ON_FOLIO(folio_ref_count(folio), folio); VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + if (lru_gen_enabled()) + return lru_gen_eviction(folio); + lruvec = mem_cgroup_lruvec(target_memcg, pgdat); /* XXX: target_memcg can be NULL, go through lruvec */ memcgid = mem_cgroup_id(lruvec_memcg(lruvec)); eviction = atomic_long_read(&lruvec->nonresident_age); + eviction >>= bucket_order; workingset_age_nonresident(lruvec, folio_nr_pages(folio)); return pack_shadow(memcgid, pgdat, eviction,
folio_test_workingset(folio));mm/workingset.c +5
-0@@ -298,7 +398,13 @@ void workingset_refault(struct folio *folio, void *shadow) int memcgid; long nr; + if (lru_gen_enabled()) { + lru_gen_refault(folio, shadow); + return; + } + unpack_shadow(shadow, &memcgid, &pgdat, &eviction, &workingset); + eviction <<= bucket_order; rcu_read_lock(); /*
PATCH v11 7/14 mm: multi-gen LRU: exploit locality in rmap
In reply to Yu ZhaoSearching the rmap for PTEs mapping each page on an LRU list (to test and clear the accessed bit) can be expensive because pages from different VMAs (PA space) are not cache friendly to the rmap (VA space). For workloads mostly using mapped pages, the rmap has a high CPU cost in the reclaim path.
This patch exploits spatial locality to reduce the trips into the rmap. When shrink_page_list() walks the rmap and finds a young PTE, a new function lru_gen_look_around() scans at most BITS_PER_LONG-1 adjacent PTEs. On finding another young PTE, it clears the accessed bit and updates the gen counter of the page mapped by this PTE to (max_seq%MAX_NR_GENS)+1.
Server benchmark results:
Single workload:
fio (buffered I/O): no change
Single workload:
memcached (anon): +[5.5, 7.5]%
Ops/sec KB/sec
patch1-6: 1120643.70 43588.06
patch1-7: 1193918.93 46438.15 Configurations:
no change
Client benchmark results:
kswapd profiles:
patch1-6
35.99% lzo1x_1_do_compress (real work)
19.40% page_vma_mapped_walk
6.31% _raw_spin_unlock_irq
3.95% do_raw_spin_lock
2.39% anon_vma_interval_tree_iter_first
2.25% ptep_clear_flush
1.92% __anon_vma_interval_tree_subtree_search
1.70% folio_referenced_one
1.68% __zram_bvec_write
1.43% anon_vma_interval_tree_iter_next
patch1-7
45.90% lzo1x_1_do_compress (real work)
9.14% page_vma_mapped_walk
6.81% _raw_spin_unlock_irq
2.80% ptep_clear_flush
2.34% __zram_bvec_write
2.29% do_raw_spin_lock
1.84% lru_gen_look_around
1.78% memmove
1.74% obj_malloc
1.50% free_unref_page_list Configurations:
no change
--- include/linux/memcontrol.h | 31 ++++++++ include/linux/mm.h | 5 ++ include/linux/mmzone.h | 6 ++ mm/internal.h | 1 + mm/memcontrol.c | 1 + mm/rmap.c | 7 ++ mm/swap.c | 4 +- mm/vmscan.c | 157 +++++++++++++++++++++++++++++++++++++ 8 files changed, 210 insertions(+), 2 deletions(-)
include/linux/memcontrol.h +31 -0
@@ -438,6 +438,7 @@ static inline struct obj_cgroup *__folio_objcg(struct folio *folio)* - LRU isolation * - lock_page_memcg() * - exclusive reference +* - mem_cgroup_trylock_pages() * * For a kmem folio a caller should hold an rcu read lock to protect memcg * associated with a kmem folio from being released.
@@ -499,6 +500,7 @@ static inline struct mem_cgroup *folio_memcg_rcu(struct folio *folio)* - LRU isolation * - lock_page_memcg() * - exclusive reference +* - mem_cgroup_trylock_pages() * * For a kmem page a caller should hold an rcu read lock to protect memcg * associated with a kmem page from being released.
@@ -948,6 +950,23 @@ void unlock_page_memcg(struct page *page);void __mod_memcg_state (struct mem_cgroup *memcg,int idx,int val); +/* try to stablize folio_memcg() for all the pages in a memcg */ +static inline bool mem_cgroup_trylock_pages (struct mem_cgroup *memcg) +{ +rcu_read_lock (); + +if (mem_cgroup_disabled () || !atomic_read (&memcg->moving_account)) +return true; + +rcu_read_unlock (); +return false; +} + +static inline void mem_cgroup_unlock_pages (void ) +{ +rcu_read_unlock (); +} +/* idx can be of type enum memcg_stat_item or node_stat_item */ static inline void mod_memcg_state (struct mem_cgroup *memcg,int idx,int val)
@@ -1386,6 +1405,18 @@ static inline void folio_memcg_unlock(struct folio *folio) { } +static inline bool mem_cgroup_trylock_pages (struct mem_cgroup *memcg) +{ +/* to match folio_memcg_rcu() */ +rcu_read_lock (); +return true; +} + +static inline void mem_cgroup_unlock_pages (void ) +{ +rcu_read_unlock (); +} +static inline void mem_cgroup_handle_over_high (void ) { }
include/linux/mm.h +5 -0
@@ -1523,6 +1523,11 @@ static inline unsigned long folio_pfn(struct folio *folio)return page_to_pfn (&folio->page); } +static inline struct folio *pfn_folio (unsigned long pfn) +{ +return page_folio (pfn_to_page (pfn)); +} +static inline atomic_t *folio_pincount_ptr (struct folio *folio) {return &folio_page (folio,1 )->compound_pincount;
include/linux/mmzone.h +6 -0
@@ -374,6 +374,7 @@ enum lruvec_flags {#ifndef __GENERATING_BOUNDS_Hstruct lruvec ; +struct page_vma_mapped_walk ;#define LRU_GEN_MASK ((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF)#define LRU_REFS_MASK ((BIT(LRU_REFS_WIDTH) - 1) << LRU_REFS_PGOFF)
@@ -429,6 +430,7 @@ struct lru_gen_struct { };void lru_gen_init_lruvec (struct lruvec *lruvec); +void lru_gen_look_around (struct page_vma_mapped_walk *pvmw);#ifdef CONFIG_MEMCG void lru_gen_init_memcg (struct mem_cgroup *memcg);
@@ -441,6 +443,10 @@ static inline void lru_gen_init_lruvec(struct lruvec *lruvec) { } +static inline void lru_gen_look_around (struct page_vma_mapped_walk *pvmw) +{ +} +#ifdef CONFIG_MEMCG static inline void lru_gen_init_memcg (struct mem_cgroup *memcg) {
mm/internal.h +1 -0
@@ -68,6 +68,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf);void folio_rotate_reclaimable (struct folio *folio);bool __folio_end_writeback (struct folio *folio);void deactivate_file_folio (struct folio *folio); +void folio_activate (struct folio *folio);void free_pgtables (struct mmu_gather *tlb,struct vm_area_struct *start_vma,unsigned long floor,unsigned long ceiling);
mm/memcontrol.c +1 -0
@@ -2769,6 +2769,7 @@ static void commit_charge(struct folio *folio, struct mem_cgroup *memcg)* - LRU isolation * - lock_page_memcg() * - exclusive reference +* - mem_cgroup_trylock_pages() */ folio->memcg_data = (unsigned long )memcg; }
mm/rmap.c +7 -0
@@ -73,6 +73,7 @@#include <linux/page_idle.h> #include <linux/memremap.h> #include <linux/userfaultfd_k.h> +#include <linux/mm_inline.h> #include <asm/tlbflush.h>
@@ -821,6 +822,12 @@ static bool folio_referenced_one(struct folio *folio, }if (pvmw.pte) { +if (lru_gen_enabled () &&pte_young (*pvmw.pte) && + !(vma->vm_flags & (VM_SEQ_READ |VM_RAND_READ ))) { +lru_gen_look_around (&pvmw); + referenced++; + } +if (ptep_clear_flush_young_notify (vma, address, pvmw.pte)) {/*
mm/swap.c +2 -2
@@ -342,7 +342,7 @@ static bool need_activate_page_drain(int cpu)return pagevec_count (&per_cpu (lru_pvecs.activate_page, cpu)) !=0 ; } -static void folio_activate (struct folio *folio) +void folio_activate (struct folio *folio) {if (folio_test_lru (folio) && !folio_test_active (folio) && !folio_test_unevictable (folio)) {
@@ -362,7 +362,7 @@ static inline void activate_page_drain(int cpu) { } -static void folio_activate (struct folio *folio) +void folio_activate (struct folio *folio) {struct lruvec *lruvec;
mm/vmscan.c +157 -0
187 lines not on this page · raw message
Signature · 1 line
2.36.0.550.gb090851708-goog
Re: [PATCH v11 07/14] mm: multi-gen LRU: exploit locality in rmap
Quoted 227 lines · Yu Zhao
Searching the rmap for PTEs mapping each page on an LRU list (to test and clear the accessed bit) can be expensive because pages from different VMAs (PA space) are not cache friendly to the rmap (VA space). For workloads mostly using mapped pages, the rmap has a high CPU cost in the reclaim path.
This patch exploits spatial locality to reduce the trips into the rmap. When
shrink_page_list()walks the rmap and finds a young PTE, a new functionlru_gen_look_around()scans at most BITS_PER_LONG-1 adjacent PTEs. On finding another young PTE, it clears the accessed bit and updates the gen counter of the page mapped by this PTE to (max_seq%MAX_NR_GENS)+1.Server benchmark results:
Single workload:
fio (buffered I/O): no changeSingle workload:
memcached (anon): +[5.5, 7.5]% Ops/sec KB/sec patch1-6: 1120643.70 43588.06 patch1-7: 1193918.93 46438.15Configurations:
no changeClient benchmark results:
kswapd profiles:patch1-6 35.99% lzo1x_1_do_compress (real work) 19.40% page_vma_mapped_walk 6.31% _raw_spin_unlock_irq 3.95% do_raw_spin_lock 2.39% anon_vma_interval_tree_iter_first 2.25% ptep_clear_flush 1.92% __anon_vma_interval_tree_subtree_search 1.70% folio_referenced_one 1.68% __zram_bvec_write 1.43% anon_vma_interval_tree_iter_next patch1-7 45.90% lzo1x_1_do_compress (real work) 9.14% page_vma_mapped_walk 6.81% _raw_spin_unlock_irq 2.80% ptep_clear_flush 2.34% __zram_bvec_write 2.29% do_raw_spin_lock 1.84% lru_gen_look_around 1.78% memmove 1.74% obj_malloc 1.50% free_unref_page_listConfigurations:
no changeSigned-off-by Yu Zhao <[email protected]>Acked-by Brian Geffon <[email protected]>Acked-by Steven Barrett <[email protected]>Acked-by Suleiman Souhlal <[email protected]>Tested-by Daniel Byrne <[email protected]>Tested-by Donald Carr <[email protected]>Tested-by Holger Hoffstätte <[email protected]>Tested-by Konstantin Kharlamov <[email protected]>Tested-by Shuang Zhai <[email protected]>Tested-by Sofia Trinh <[email protected]>Tested-by Vaibhav Jain <[email protected]>--- include/linux/memcontrol.h | 31 ++++++++ include/linux/mm.h | 5 ++ include/linux/mmzone.h | 6 ++ mm/internal.h | 1 + mm/memcontrol.c | 1 + mm/rmap.c | 7 ++ mm/swap.c | 4 +- mm/vmscan.c | 157 +++++++++++++++++++++++++++++++++++++ 8 files changed, 210 insertions(+), 2 deletions(-)include/linux/memcontrol.h +31
-0@@ -438,6 +438,7 @@ static inline struct obj_cgroup *__folio_objcg(struct folio *folio)* - LRU isolation * - lock_page_memcg() * - exclusive reference +* - mem_cgroup_trylock_pages() * * For a kmem folio a caller should hold an rcu read lock to protect memcg * associated with a kmem folio from being released. @@ -499,6 +500,7 @@ static inline struct mem_cgroup *folio_memcg_rcu(struct folio *folio)* - LRU isolation * - lock_page_memcg() * - exclusive reference +* - mem_cgroup_trylock_pages() * * For a kmem page a caller should hold an rcu read lock to protect memcg * associated with a kmem page from being released. @@ -948,6 +950,23 @@ void unlock_page_memcg(struct page *page);void __mod_memcg_state (struct mem_cgroup *memcg,int idx,int val); +/* try to stablize folio_memcg() for all the pages in a memcg */ +static inline bool mem_cgroup_trylock_pages (struct mem_cgroup *memcg) +{ +rcu_read_lock (); + +if (mem_cgroup_disabled () || !atomic_read (&memcg->moving_account)) +return true; + +rcu_read_unlock (); +return false; +} + +static inline void mem_cgroup_unlock_pages (void ) +{ +rcu_read_unlock (); +} +/* idx can be of type enum memcg_stat_item or node_stat_item */ static inline void mod_memcg_state (struct mem_cgroup *memcg,int idx,int val)@@ -1386,6 +1405,18 @@ static inline void folio_memcg_unlock(struct folio *folio) { } +static inline bool mem_cgroup_trylock_pages (struct mem_cgroup *memcg) +{ +/* to match folio_memcg_rcu() */ +rcu_read_lock (); +return true; +} + +static inline void mem_cgroup_unlock_pages (void ) +{ +rcu_read_unlock (); +} +static inline void mem_cgroup_handle_over_high (void ) { }include/linux/mm.h +5
-0@@ -1523,6 +1523,11 @@ static inline unsigned long folio_pfn(struct folio *folio)return page_to_pfn (&folio->page); } +static inline struct folio *pfn_folio (unsigned long pfn) +{ +return page_folio (pfn_to_page (pfn)); +} +static inline atomic_t *folio_pincount_ptr (struct folio *folio) {return &folio_page (folio,1 )->compound_pincount;include/linux/mmzone.h +6
-0@@ -374,6 +374,7 @@ enum lruvec_flags {#ifndef __GENERATING_BOUNDS_Hstruct lruvec ; +struct page_vma_mapped_walk ;#define LRU_GEN_MASK ((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF)#define LRU_REFS_MASK ((BIT(LRU_REFS_WIDTH) - 1) << LRU_REFS_PGOFF)@@ -429,6 +430,7 @@ struct lru_gen_struct { };void lru_gen_init_lruvec (struct lruvec *lruvec); +void lru_gen_look_around (struct page_vma_mapped_walk *pvmw);#ifdef CONFIG_MEMCG void lru_gen_init_memcg (struct mem_cgroup *memcg);@@ -441,6 +443,10 @@ static inline void lru_gen_init_lruvec(struct lruvec *lruvec) { } +static inline void lru_gen_look_around (struct page_vma_mapped_walk *pvmw) +{ +} +#ifdef CONFIG_MEMCG static inline void lru_gen_init_memcg (struct mem_cgroup *memcg) {mm/internal.h +1
-0@@ -68,6 +68,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf);void folio_rotate_reclaimable (struct folio *folio);bool __folio_end_writeback (struct folio *folio);void deactivate_file_folio (struct folio *folio); +void folio_activate (struct folio *folio);void free_pgtables (struct mmu_gather *tlb,struct vm_area_struct *start_vma,unsigned long floor,unsigned long ceiling);mm/memcontrol.c +1
-0@@ -2769,6 +2769,7 @@ static void commit_charge(struct folio *folio, struct mem_cgroup *memcg)* - LRU isolation * - lock_page_memcg() * - exclusive reference +* - mem_cgroup_trylock_pages() */ folio->memcg_data = (unsigned long )memcg; }mm/rmap.c +1
-0@@ -73,6 +73,7 @@#include <linux/page_idle.h> #include <linux/memremap.h> #include <linux/userfaultfd_k.h> +#include <linux/mm_inline.h> #include <asm/tlbflush.h>
mm/rmap.c static bool folio_referenced_one(struct folio *folio, · line 831
Quoted 10 lines
mm/rmap.c +6
-0@@ -821,6 +822,12 @@ static bool folio_referenced_one(struct folio *folio, }if (pvmw.pte) { +if (lru_gen_enabled () &&pte_young (*pvmw.pte) && + !(vma->vm_flags & (VM_SEQ_READ |VM_RAND_READ ))) { +lru_gen_look_around (&pvmw); + referenced++; + } + if (ptep_clear_flush_young_notify(vma, address,
Hello, Yu.look_around() is calling ptep_test_and_clear_young(pvmw->vma, addr, pte + i) only without flush and notify. for flush, there is a tlb operation for arm64:
static inline int ptep_clear_flush_young(struct vm_area_struct *vma,
unsigned long address, pte_t *ptep)
{
int young = ptep_test_and_clear_young(vma, address, ptep);
if (young) {
/*
* We can elide the trailing DSB here since the worst that can
* happen is that a CPU continues to use the young entry in its
* TLB and we mistakenly reclaim the associated page. The
* window for such an event is bounded by the next
* context-switch, which provides a DSB to complete the TLB
* invalidation.
*/
flush_tlb_page_nosync(vma, address);
}
return young;
}Does it mean the current kernel is over cautious? is it safe to call ptep_test_and_clear_young() only?
btw, lru_gen_look_around() has already included 'address', are we doing pte check for 'address' twice here?
Quoted 221 lines
mm/rmap.c +0
-0pvmw.pte)) {/* mm/swap.c +2
-2@@ -342,7 +342,7 @@ static bool need_activate_page_drain(int cpu)return pagevec_count (&per_cpu (lru_pvecs.activate_page, cpu)) !=0 ; } -static void folio_activate (struct folio *folio) +void folio_activate (struct folio *folio) {if (folio_test_lru (folio) && !folio_test_active (folio) && !folio_test_unevictable (folio)) {@@ -362,7 +362,7 @@ static inline void activate_page_drain(int cpu) { } -static void folio_activate (struct folio *folio) +void folio_activate (struct folio *folio) {struct lruvec *lruvec;mm/vmscan.c +157
-1@@ -1554,6 +1554,11 @@ static unsigned int shrink_page_list(struct list_head *page_list,if (!sc->may_unmap &&page_mapped (page)) goto keep_locked; +/* folio_update_gen() tried to promote this page? */ +if (lru_gen_enabled () && !ignore_references && +page_mapped (page) &&PageReferenced (page)) + goto keep_locked; + may_enter_fs = (sc->gfp_mask & __GFP_FS) || (PageSwapCache (page) && (sc->gfp_mask & __GFP_IO));@@ -3137,6 +3142,28 @@ static bool positive_ctrl_err(struct ctrl_pos *sp, struct ctrl_pos *pv)* the aging ******************************************************************************/ +static int folio_update_gen (struct folio *folio,int gen) +{ +unsigned long new_flags, old_flags =READ_ONCE (folio->flags); + +VM_WARN_ON_ONCE (gen >=MAX_NR_GENS ); +VM_WARN_ON_ONCE (!rcu_read_lock_held ()); + +do { +/* lru_gen_del_folio() has isolated this page? */ +if (!(old_flags &LRU_GEN_MASK )) { +/* for shrink_page_list() */ + new_flags = old_flags |BIT (PG_referenced); +continue ; + } + + new_flags = old_flags & ~(LRU_GEN_MASK |LRU_REFS_MASK |LRU_REFS_FLAGS ); + new_flags |= (gen +1UL ) <<LRU_GEN_PGOFF ; + }while (!try_cmpxchg (&folio->flags, &old_flags, new_flags)); + +return ((old_flags &LRU_GEN_MASK ) >>LRU_GEN_PGOFF ) -1 ; +} +static int folio_inc_gen (struct lruvec *lruvec,struct folio *folio,bool reclaiming) {int type =folio_is_file_lru (folio);@@ -3147,6 +3174,11 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool reclaiVM_WARN_ON_ONCE_FOLIO (!(old_flags &LRU_GEN_MASK ), folio); do { + new_gen = ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; + /* folio_update_gen() has promoted this page? */ + if (new_gen >= 0 && new_gen != old_gen) + return new_gen; + new_gen = (old_gen + 1) % MAX_NR_GENS; new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_MASK | LRU_REFS_FLAGS);@@ -3365,6 +3397,125 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc) }while ((memcg =mem_cgroup_iter (NULL , memcg,NULL ))); } +/* +* This function exploits spatial locality when shrink_page_list() walks the +* rmap. It scans the adjacent PTEs of a young PTE and promotes hot pages. +*/ +void lru_gen_look_around (struct page_vma_mapped_walk *pvmw) +{ +int i; +pte_t *pte; +unsigned long start; +unsigned long end; +unsigned long addr; +unsigned long bitmap[BITS_TO_LONGS (MIN_LRU_BATCH )] = {}; +struct folio *folio =pfn_folio (pvmw->pfn); +struct mem_cgroup *memcg =folio_memcg (folio); +struct pglist_data *pgdat =folio_pgdat (folio); +struct lruvec *lruvec =mem_cgroup_lruvec (memcg, pgdat); +DEFINE_MAX_SEQ (lruvec); +int old_gen, new_gen =lru_gen_from_seq (max_seq); + +lockdep_assert_held (pvmw->ptl); +VM_WARN_ON_ONCE_FOLIO (folio_test_lru (folio), folio); + +if (spin_is_contended (pvmw->ptl)) +return ; + + start =max (pvmw->address &PMD_MASK , pvmw->vma->vm_start); + end =pmd_addr_end (pvmw->address, pvmw->vma->vm_end); + +if (end - start >MIN_LRU_BATCH *PAGE_SIZE ) { +if (pvmw->address - start <MIN_LRU_BATCH *PAGE_SIZE /2 ) + end = start +MIN_LRU_BATCH *PAGE_SIZE ; +else if (end - pvmw->address <MIN_LRU_BATCH *PAGE_SIZE /2 ) + start = end -MIN_LRU_BATCH *PAGE_SIZE ; +else { + start = pvmw->address -MIN_LRU_BATCH *PAGE_SIZE /2 ; + end = pvmw->address +MIN_LRU_BATCH *PAGE_SIZE /2 ; + } + } + + pte = pvmw->pte - (pvmw->address - start) /PAGE_SIZE ; + +rcu_read_lock (); +arch_enter_lazy_mmu_mode (); + +for (i =0 , addr = start; addr != end; i++, addr +=PAGE_SIZE ) { +unsigned long pfn =pte_pfn (pte[i]); + +VM_WARN_ON_ONCE (addr < pvmw->vma->vm_start || addr >= pvmw->vma->vm_end); + +if (!pte_present (pte[i]) ||is_zero_pfn (pfn)) +continue ; + +if (WARN_ON_ONCE (pte_devmap (pte[i]) ||pte_special (pte[i]))) +continue ; + +if (!pte_young (pte[i])) +continue ; + +VM_WARN_ON_ONCE (!pfn_valid (pfn)); +if (pfn < pgdat->node_start_pfn || pfn >=pgdat_end_pfn (pgdat)) +continue ; + + folio =pfn_folio (pfn); +if (folio_nid (folio) != pgdat->node_id) +continue ; + +if (folio_memcg_rcu (folio) != memcg) +continue ; + +if (!ptep_test_and_clear_young (pvmw->vma, addr, pte + i)) +continue ; + +if (pte_dirty (pte[i]) && !folio_test_dirty (folio) && + !(folio_test_anon (folio) &&folio_test_swapbacked (folio) && + !folio_test_swapcache (folio))) +folio_mark_dirty (folio); + + old_gen =folio_lru_gen (folio); +if (old_gen <0 ) +folio_set_referenced (folio); +else if (old_gen != new_gen) +__set_bit (i, bitmap); + } + +arch_leave_lazy_mmu_mode (); +rcu_read_unlock (); + +if (bitmap_weight (bitmap,MIN_LRU_BATCH ) <PAGEVEC_SIZE ) { +for_each_set_bit (i, bitmap,MIN_LRU_BATCH ) { + folio =pfn_folio (pte_pfn (pte[i])); +folio_activate (folio); + } +return ; + } + +/* folio_update_gen() requires stable folio_memcg() */ +if (!mem_cgroup_trylock_pages (memcg)) +return ; + +spin_lock_irq (&lruvec->lru_lock); + new_gen =lru_gen_from_seq (lruvec->lrugen.max_seq); + +for_each_set_bit (i, bitmap,MIN_LRU_BATCH ) { + folio =pfn_folio (pte_pfn (pte[i])); +if (folio_memcg_rcu (folio) != memcg) +continue ; + + old_gen =folio_update_gen (folio, new_gen); +if (old_gen <0 || old_gen == new_gen) +continue ; + +lru_gen_update_size (lruvec, folio, old_gen, new_gen); + } + +spin_unlock_irq (&lruvec->lru_lock); + +mem_cgroup_unlock_pages (); +} +/****************************************************************************** * the eviction ******************************************************************************/ @@ -3401,6 +3552,12 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, int tier_idx)return true; } +/* promoted */ +if (gen !=lru_gen_from_seq (lrugen->min_seq[type])) { +list_move (&folio->lru, &lrugen->lists[gen][type][zone]); +return true; + } +/* protected */ if (tier > tier_idx) {int hist =lru_hist_from_seq (lrugen->min_seq[type]); --
Quoted 2 lines
2.36.0.550.gb090851708-goog
Thanks
Barry
Re: [PATCH v11 07/14] mm: multi-gen LRU: exploit locality in rmap
Quoted 263 lines · Barry Song
Quoted 227 lines · Yu Zhao
Searching the rmap for PTEs mapping each page on an LRU list (to test and clear the accessed bit) can be expensive because pages from different VMAs (PA space) are not cache friendly to the rmap (VA space). For workloads mostly using mapped pages, the rmap has a high CPU cost in the reclaim path.
This patch exploits spatial locality to reduce the trips into the rmap. When
shrink_page_list()walks the rmap and finds a young PTE, a new functionlru_gen_look_around()scans at most BITS_PER_LONG-1 adjacent PTEs. On finding another young PTE, it clears the accessed bit and updates the gen counter of the page mapped by this PTE to (max_seq%MAX_NR_GENS)+1.Server benchmark results:
Single workload:
fio (buffered I/O): no changeSingle workload:
memcached (anon): +[5.5, 7.5]% Ops/sec KB/sec patch1-6: 1120643.70 43588.06 patch1-7: 1193918.93 46438.15Configurations:
no changeClient benchmark results:
kswapd profiles:patch1-6 35.99% lzo1x_1_do_compress (real work) 19.40% page_vma_mapped_walk 6.31% _raw_spin_unlock_irq 3.95% do_raw_spin_lock 2.39% anon_vma_interval_tree_iter_first 2.25% ptep_clear_flush 1.92% __anon_vma_interval_tree_subtree_search 1.70% folio_referenced_one 1.68% __zram_bvec_write 1.43% anon_vma_interval_tree_iter_next patch1-7 45.90% lzo1x_1_do_compress (real work) 9.14% page_vma_mapped_walk 6.81% _raw_spin_unlock_irq 2.80% ptep_clear_flush 2.34% __zram_bvec_write 2.29% do_raw_spin_lock 1.84% lru_gen_look_around 1.78% memmove 1.74% obj_malloc 1.50% free_unref_page_listConfigurations:
no changeSigned-off-by Yu Zhao <[email protected]>Acked-by Brian Geffon <[email protected]>Acked-by Steven Barrett <[email protected]>Acked-by Suleiman Souhlal <[email protected]>Tested-by Daniel Byrne <[email protected]>Tested-by Donald Carr <[email protected]>Tested-by Holger Hoffstätte <[email protected]>Tested-by Konstantin Kharlamov <[email protected]>Tested-by Shuang Zhai <[email protected]>Tested-by Sofia Trinh <[email protected]>Tested-by Vaibhav Jain <[email protected]>--- include/linux/memcontrol.h | 31 ++++++++ include/linux/mm.h | 5 ++ include/linux/mmzone.h | 6 ++ mm/internal.h | 1 + mm/memcontrol.c | 1 + mm/rmap.c | 7 ++ mm/swap.c | 4 +- mm/vmscan.c | 157 +++++++++++++++++++++++++++++++++++++ 8 files changed, 210 insertions(+), 2 deletions(-)include/linux/memcontrol.h +31
-0@@ -438,6 +438,7 @@ static inline struct obj_cgroup *__folio_objcg(struct folio *folio)* - LRU isolation * - lock_page_memcg() * - exclusive reference +* - mem_cgroup_trylock_pages() * * For a kmem folio a caller should hold an rcu read lock to protect memcg * associated with a kmem folio from being released. @@ -499,6 +500,7 @@ static inline struct mem_cgroup *folio_memcg_rcu(struct folio *folio)* - LRU isolation * - lock_page_memcg() * - exclusive reference +* - mem_cgroup_trylock_pages() * * For a kmem page a caller should hold an rcu read lock to protect memcg * associated with a kmem page from being released. @@ -948,6 +950,23 @@ void unlock_page_memcg(struct page *page);void __mod_memcg_state (struct mem_cgroup *memcg,int idx,int val); +/* try to stablize folio_memcg() for all the pages in a memcg */ +static inline bool mem_cgroup_trylock_pages (struct mem_cgroup *memcg) +{ +rcu_read_lock (); + +if (mem_cgroup_disabled () || !atomic_read (&memcg->moving_account)) +return true; + +rcu_read_unlock (); +return false; +} + +static inline void mem_cgroup_unlock_pages (void ) +{ +rcu_read_unlock (); +} +/* idx can be of type enum memcg_stat_item or node_stat_item */ static inline void mod_memcg_state (struct mem_cgroup *memcg,int idx,int val)@@ -1386,6 +1405,18 @@ static inline void folio_memcg_unlock(struct folio *folio) { } +static inline bool mem_cgroup_trylock_pages (struct mem_cgroup *memcg) +{ +/* to match folio_memcg_rcu() */ +rcu_read_lock (); +return true; +} + +static inline void mem_cgroup_unlock_pages (void ) +{ +rcu_read_unlock (); +} +static inline void mem_cgroup_handle_over_high (void ) { }include/linux/mm.h +5
-0@@ -1523,6 +1523,11 @@ static inline unsigned long folio_pfn(struct folio *folio)return page_to_pfn (&folio->page); } +static inline struct folio *pfn_folio (unsigned long pfn) +{ +return page_folio (pfn_to_page (pfn)); +} +static inline atomic_t *folio_pincount_ptr (struct folio *folio) {return &folio_page (folio,1 )->compound_pincount;include/linux/mmzone.h +6
-0@@ -374,6 +374,7 @@ enum lruvec_flags {#ifndef __GENERATING_BOUNDS_Hstruct lruvec ; +struct page_vma_mapped_walk ;#define LRU_GEN_MASK ((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF)#define LRU_REFS_MASK ((BIT(LRU_REFS_WIDTH) - 1) << LRU_REFS_PGOFF)@@ -429,6 +430,7 @@ struct lru_gen_struct { };void lru_gen_init_lruvec (struct lruvec *lruvec); +void lru_gen_look_around (struct page_vma_mapped_walk *pvmw);#ifdef CONFIG_MEMCG void lru_gen_init_memcg (struct mem_cgroup *memcg);@@ -441,6 +443,10 @@ static inline void lru_gen_init_lruvec(struct lruvec *lruvec) { } +static inline void lru_gen_look_around (struct page_vma_mapped_walk *pvmw) +{ +} +#ifdef CONFIG_MEMCG static inline void lru_gen_init_memcg (struct mem_cgroup *memcg) {mm/internal.h +1
-0@@ -68,6 +68,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf);void folio_rotate_reclaimable (struct folio *folio);bool __folio_end_writeback (struct folio *folio);void deactivate_file_folio (struct folio *folio); +void folio_activate (struct folio *folio);void free_pgtables (struct mmu_gather *tlb,struct vm_area_struct *start_vma,unsigned long floor,unsigned long ceiling);mm/memcontrol.c +1
-0@@ -2769,6 +2769,7 @@ static void commit_charge(struct folio *folio, struct mem_cgroup *memcg)* - LRU isolation * - lock_page_memcg() * - exclusive reference +* - mem_cgroup_trylock_pages() */ folio->memcg_data = (unsigned long )memcg; }mm/rmap.c +1
-0@@ -73,6 +73,7 @@#include <linux/page_idle.h> #include <linux/memremap.h> #include <linux/userfaultfd_k.h> +#include <linux/mm_inline.h> #include <asm/tlbflush.h> mm/rmap.c static bool folio_referenced_one(struct folio *folio, · line 831
Quoted 10 lines
mm/rmap.c +6
-0@@ -821,6 +822,12 @@ static bool folio_referenced_one(struct folio *folio, }if (pvmw.pte) { +if (lru_gen_enabled () &&pte_young (*pvmw.pte) && + !(vma->vm_flags & (VM_SEQ_READ |VM_RAND_READ ))) { +lru_gen_look_around (&pvmw); + referenced++; + } + if (ptep_clear_flush_young_notify(vma, address,Hello, Yu.
look_around()is callingptep_test_and_clear_young(pvmw->vma, addr, pte + i)only without flush and notify. for flush, there is a tlb operation for arm64:
static inline int ptep_clear_flush_young(structvm_area_struct*vma,
unsigned long address,pte_t*ptep){ int young = ptep_test_and_clear_young(vma, address, ptep); if (young) { /* * We can elide the trailing DSB here since the worst that can * happen is that a CPU continues to use the young entry in its * TLB and we mistakenly reclaim the associated page. The * window for such an event is bounded by the next * context-switch, which provides a DSB to complete the TLB * invalidation. */ flush_tlb_page_nosync(vma, address); } return young; }
Quoted 2 lines
Does it mean the current kernel is over cautious? is it safe to call
ptep_test_and_clear_young()only?
I can't really explain why we are getting a random app/java vm crash in monkey test by using ptep_test_and_clear_young() only in lru_gen_look_around() on an armv8-a machine without hardware PTE young support.
Moving to ptep_clear_flush_young() in look_around can make the random hang disappear according to zhanyuan(Cc-ed).
On x86, ptep_clear_flush_young() is exactly ptep_test_and_clear_young() after
'commit b13b1d2d8692 ("x86/mm: In the PTE swapout page reclaim case clear
the accessed bit instead of flushing the TLB")'
But on arm64, they are different. according to Will's comments in this thread which
tried to make arm64 same with x86,
https://www.mail-archive.com/[email protected]/msg1793881.html
"
This is blindly copied from x86 and isn't true for us: we don't invalidate the TLB on context switch. That means our window for keeping the stale entries around is potentially much bigger and might not be a great idea.
If we roll a TLB invalidation routine without the trailing DSB, what sort of performance does that get you?
"
We shouldn't think ptep_clear_flush_young() is safe enough in LRU to clear PTE young? Any comments from Will?
Quoted 4 lines
btw,
lru_gen_look_around()has already included 'address', are we doing pte check for 'address' twice here?
Thanks
Barry
Will Deacon
maintainerRe: [PATCH v11 07/14] mm: multi-gen LRU: exploit locality in rmap
Quoted 74 lines · Barry Song
Quoted 48 lines · Barry Song
Quoted 13 lines · Yu Zhao
mm/rmap.c +1
-0@@ -73,6 +73,7 @@#include <linux/page_idle.h> #include <linux/memremap.h> #include <linux/userfaultfd_k.h> +#include <linux/mm_inline.h> #include <asm/tlbflush.h> mm/rmap.c static bool folio_referenced_one(struct folio *folio, · line 831
Quoted 10 lines
mm/rmap.c +6
-0@@ -821,6 +822,12 @@ static bool folio_referenced_one(struct folio *folio, }if (pvmw.pte) { +if (lru_gen_enabled () &&pte_young (*pvmw.pte) && + !(vma->vm_flags & (VM_SEQ_READ |VM_RAND_READ ))) { +lru_gen_look_around (&pvmw); + referenced++; + } + if (ptep_clear_flush_young_notify(vma, address,Hello, Yu.
look_around()is callingptep_test_and_clear_young(pvmw->vma, addr, pte + i)only without flush and notify. for flush, there is a tlb operation for arm64:
static inline int ptep_clear_flush_young(structvm_area_struct*vma,
unsigned long address,pte_t*ptep){ int young = ptep_test_and_clear_young(vma, address, ptep); if (young) { /* * We can elide the trailing DSB here since the worst that can * happen is that a CPU continues to use the young entry in its * TLB and we mistakenly reclaim the associated page. The * window for such an event is bounded by the next * context-switch, which provides a DSB to complete the TLB * invalidation. */ flush_tlb_page_nosync(vma, address); } return young; }Quoted 2 lines
Does it mean the current kernel is over cautious? is it safe to call
ptep_test_and_clear_young()only?I can't really explain why we are getting a random app/java vm crash in monkey test by using
ptep_test_and_clear_young()only inlru_gen_look_around()on an armv8-a machine without hardware PTE young support.Moving to
ptep_clear_flush_young()inlook_aroundcan make the random hang disappear according to zhanyuan(Cc-ed).On x86,
ptep_clear_flush_young()is exactlyptep_test_and_clear_young()after
'commitb13b1d2d8692("x86/mm: In the PTE swapout page reclaim case clear
the accessed bit instead of flushing the TLB")'But on arm64, they are different. according to Will's comments in this thread which
tried to make arm64 same with x86,
https://www.mail-archive.com/[email protected]/msg1793881.html"
This is blindly copied from x86 and isn't true for us: we don't invalidate the TLB on context switch. That means our window for keeping the stale entries around is potentially much bigger and might not be a great idea.
Quoted 5 lines
If we roll a TLB invalidation routine without the trailing DSB, what sort of performance does that get you?
"
We shouldn't thinkptep_clear_flush_young()is safe enough in LRU to clear PTE young? Any comments from Will?
Given that this issue is specific to the multi-gen LRU work, I think Yu is the best person to comment. However, looking quickly at your analysis above, I wonder if the code is relying on this sequence:
ptep_test_and_clear_young(vma, address, ptep); ptep_clear_flush_young(vma, address, ptep);
to invalidate the TLB. On arm64, that won't be the case, as the invalidation in ptep_clear_flush_young() is predicated on the pte being young (and this patches the generic implementation in mm/pgtable-generic.c. In fact, that second function call is always going to be a no-op unless the pte became young again in the middle.
Will
Re: [PATCH v11 07/14] mm: multi-gen LRU: exploit locality in rmap
Quoted 91 lines · Will Deacon
Quoted 74 lines · Barry Song
Quoted 48 lines · Barry Song
Quoted 13 lines · Yu Zhao
mm/rmap.c +1
-0@@ -73,6 +73,7 @@#include <linux/page_idle.h> #include <linux/memremap.h> #include <linux/userfaultfd_k.h> +#include <linux/mm_inline.h> #include <asm/tlbflush.h> mm/rmap.c static bool folio_referenced_one(struct folio *folio, · line 831
Quoted 10 lines
mm/rmap.c +6
-0@@ -821,6 +822,12 @@ static bool folio_referenced_one(struct folio *folio, }if (pvmw.pte) { +if (lru_gen_enabled () &&pte_young (*pvmw.pte) && + !(vma->vm_flags & (VM_SEQ_READ |VM_RAND_READ ))) { +lru_gen_look_around (&pvmw); + referenced++; + } + if (ptep_clear_flush_young_notify(vma, address,Hello, Yu.
look_around()is callingptep_test_and_clear_young(pvmw->vma, addr, pte + i)only without flush and notify. for flush, there is a tlb operation for arm64:
static inline int ptep_clear_flush_young(structvm_area_struct*vma,
unsigned long address,pte_t*ptep){ int young = ptep_test_and_clear_young(vma, address, ptep); if (young) { /* * We can elide the trailing DSB here since the worst that can * happen is that a CPU continues to use the young entry in its * TLB and we mistakenly reclaim the associated page. The * window for such an event is bounded by the next * context-switch, which provides a DSB to complete the TLB * invalidation. */ flush_tlb_page_nosync(vma, address); } return young; }Quoted 2 lines
Does it mean the current kernel is over cautious? is it safe to call
ptep_test_and_clear_young()only?I can't really explain why we are getting a random app/java vm crash in monkey test by using
ptep_test_and_clear_young()only inlru_gen_look_around()on an armv8-a machine without hardware PTE young support.Moving to
ptep_clear_flush_young()inlook_aroundcan make the random hang disappear according to zhanyuan(Cc-ed).On x86,
ptep_clear_flush_young()is exactlyptep_test_and_clear_young()after
'commitb13b1d2d8692("x86/mm: In the PTE swapout page reclaim case clear
the accessed bit instead of flushing the TLB")'But on arm64, they are different. according to Will's comments in this thread which
tried to make arm64 same with x86,
https://www.mail-archive.com/[email protected]/msg1793881.html"
This is blindly copied from x86 and isn't true for us: we don't invalidate the TLB on context switch. That means our window for keeping the stale entries around is potentially much bigger and might not be a great idea.Quoted 5 lines
If we roll a TLB invalidation routine without the trailing DSB, what sort of performance does that get you?
"
We shouldn't thinkptep_clear_flush_young()is safe enough in LRU to clear PTE young? Any comments from Will?Given that this issue is specific to the multi-gen LRU work, I think Yu is the best person to comment. However, looking quickly at your analysis above, I wonder if the code is relying on this sequence:
ptep_test_and_clear_young(vma, address, ptep); ptep_clear_flush_young(vma, address, ptep);
Quoted 5 lines
to invalidate the TLB. On arm64, that won't be the case, as the invalidation in
ptep_clear_flush_young()is predicated on the pte being young (and this patches the generic implementation inmm/pgtable-generic.c. In fact, that second function call is always going to be a no-op unless the pte became young again in the middle.
Hi Will,
thanks for your reply, sorry for failing to let you understand my question. my question is actually as below,
right now lru_gen_look_around() is using ptep_test_and_clear_young() only without flush to clear pte for a couple of pages including the specific address:
void lru_gen_look_around(struct page_vma_mapped_walk *pvmw)
{
...
for (i = 0, addr = start; addr != end; i++, addr += PAGE_SIZE) {
...
if (!ptep_test_and_clear_young(pvmw->vma, addr, pte + i))
continue;
...
}I wonder if it is safe to arm64. Do we need to move to ptep_clear_flush_young() in the loop?
Quoted 2 lines
Will
Thanks
Barry
Will Deacon
maintainerRe: [PATCH v11 07/14] mm: multi-gen LRU: exploit locality in rmap
Quoted 63 lines · Barry Song
Quoted 38 lines · Will Deacon
Quoted 22 lines · Barry Song
I can't really explain why we are getting a random app/java vm crash in monkey test by using
ptep_test_and_clear_young()only inlru_gen_look_around()on an armv8-a machine without hardware PTE young support.Moving to
ptep_clear_flush_young()inlook_aroundcan make the random hang disappear according to zhanyuan(Cc-ed).On x86,
ptep_clear_flush_young()is exactlyptep_test_and_clear_young()after
'commitb13b1d2d8692("x86/mm: In the PTE swapout page reclaim case clear
the accessed bit instead of flushing the TLB")'But on arm64, they are different. according to Will's comments in this thread which
tried to make arm64 same with x86,
https://www.mail-archive.com/[email protected]/msg1793881.html"
This is blindly copied from x86 and isn't true for us: we don't invalidate the TLB on context switch. That means our window for keeping the stale entries around is potentially much bigger and might not be a great idea.Quoted 5 lines
If we roll a TLB invalidation routine without the trailing DSB, what sort of performance does that get you?
"
We shouldn't thinkptep_clear_flush_young()is safe enough in LRU to clear PTE young? Any comments from Will?Given that this issue is specific to the multi-gen LRU work, I think Yu is the best person to comment. However, looking quickly at your analysis above, I wonder if the code is relying on this sequence:
ptep_test_and_clear_young(vma, address, ptep); ptep_clear_flush_young(vma, address, ptep);Quoted 5 lines
to invalidate the TLB. On arm64, that won't be the case, as the invalidation in
ptep_clear_flush_young()is predicated on the pte being young (and this patches the generic implementation inmm/pgtable-generic.c. In fact, that second function call is always going to be a no-op unless the pte became young again in the middle.thanks for your reply, sorry for failing to let you understand my question. my question is actually as below,
right nowlru_gen_look_around()is usingptep_test_and_clear_young()only without flush to clear pte for a couple of pages including the specific address:
voidlru_gen_look_around(struct page_vma_mapped_walk *pvmw){ ... for (i = 0, addr = start; addr != end; i++, addr += PAGE_SIZE) { ... if (!ptep_test_and_clear_young(pvmw->vma, addr, pte + i)) continue; ... }
Quoted 2 lines
I wonder if it is safe to arm64. Do we need to move to
ptep_clear_flush_young()in the loop?
I don't know what this code is doing, so Yu is the best person to answer that. There's nothing inherently dangerous about eliding the TLB maintenance; it really depends on the guarantees needed by the caller.
However, the snippet you posted from folio_referenced_one():
| if (pvmw.pte) {
| + if (lru_gen_enabled() && pte_young(*pvmw.pte) &&
| + !(vma->vm_flags & (VM_SEQ_READ | VM_RAND_READ))) {
| + lru_gen_look_around(&pvmw);
| + referenced++;
| + }
| +
| if (ptep_clear_flush_young_notify(vma, address,Does seem to call lru_gen_look_around() *and* ptep_clear_flush_young_notify(), which is what prompted my question as it looks pretty suspicious to me.
Will
Re: [PATCH v11 07/14] mm: multi-gen LRU: exploit locality in rmap
Quoted 65 lines · Will Deacon
Quoted 63 lines · Barry Song
Quoted 38 lines · Will Deacon
Quoted 22 lines · Barry Song
I can't really explain why we are getting a random app/java vm crash in monkey test by using
ptep_test_and_clear_young()only inlru_gen_look_around()on an armv8-a machine without hardware PTE young support.Moving to
ptep_clear_flush_young()inlook_aroundcan make the random hang disappear according to zhanyuan(Cc-ed).On x86,
ptep_clear_flush_young()is exactlyptep_test_and_clear_young()after
'commitb13b1d2d8692("x86/mm: In the PTE swapout page reclaim case clear
the accessed bit instead of flushing the TLB")'But on arm64, they are different. according to Will's comments in this thread which
tried to make arm64 same with x86,
https://www.mail-archive.com/[email protected]/msg1793881.html"
This is blindly copied from x86 and isn't true for us: we don't invalidate the TLB on context switch. That means our window for keeping the stale entries around is potentially much bigger and might not be a great idea.Quoted 5 lines
If we roll a TLB invalidation routine without the trailing DSB, what sort of performance does that get you?
"
We shouldn't thinkptep_clear_flush_young()is safe enough in LRU to clear PTE young? Any comments from Will?Given that this issue is specific to the multi-gen LRU work, I think Yu is the best person to comment. However, looking quickly at your analysis above, I wonder if the code is relying on this sequence:
ptep_test_and_clear_young(vma, address, ptep); ptep_clear_flush_young(vma, address, ptep);Quoted 5 lines
to invalidate the TLB. On arm64, that won't be the case, as the invalidation in
ptep_clear_flush_young()is predicated on the pte being young (and this patches the generic implementation inmm/pgtable-generic.c. In fact, that second function call is always going to be a no-op unless the pte became young again in the middle.thanks for your reply, sorry for failing to let you understand my question. my question is actually as below,
right nowlru_gen_look_around()is usingptep_test_and_clear_young()only without flush to clear pte for a couple of pages including the specific address:
voidlru_gen_look_around(struct page_vma_mapped_walk *pvmw){ ... for (i = 0, addr = start; addr != end; i++, addr += PAGE_SIZE) { ... if (!ptep_test_and_clear_young(pvmw->vma, addr, pte + i)) continue; ... }
Quoted 6 lines
Quoted 2 lines
I wonder if it is safe to arm64. Do we need to move to
ptep_clear_flush_young()in the loop?I don't know what this code is doing, so Yu is the best person to answer that. There's nothing inherently dangerous about eliding the TLB maintenance; it really depends on the guarantees needed by the caller.
Ack.
Quoted 12 lines
However, the snippet you posted from
folio_referenced_one():| if (pvmw.pte) { | + if (lru_gen_enabled() && pte_young(*pvmw.pte) && | + !(vma->vm_flags & (VM_SEQ_READ | VM_RAND_READ))) { | + lru_gen_look_around(&pvmw); | + referenced++; | + } | + | if (ptep_clear_flush_young_notify(vma, address,
Quoted 3 lines
Does seem to call
lru_gen_look_around()*and*ptep_clear_flush_young_notify(), which is what prompted my question as it looks pretty suspicious to me.
The _notify varint reaches into the MMU notifier -- lru_gen_look_around() doesn't do that because GPA space generally has no locality. I hope this explains why both.
As to why the code is organized this way -- it depends on the point of view. Mine is that lru_gen_look_around() is an add-on, since its logic is independent/separable from ptep_clear_flush_young_notify(). We can make lru_gen_look_around() include ptep_clear_flush_young_notify(), but that would make the code functionally interwinted, which is bad for my taste.
Re: [PATCH v11 07/14] mm: multi-gen LRU: exploit locality in rmap
Quoted 96 lines · Yu Zhao
Quoted 65 lines · Will Deacon
Quoted 63 lines · Barry Song
Quoted 38 lines · Will Deacon
Quoted 22 lines · Barry Song
I can't really explain why we are getting a random app/java vm crash in monkey test by using
ptep_test_and_clear_young()only inlru_gen_look_around()on an armv8-a machine without hardware PTE young support.Moving to
ptep_clear_flush_young()inlook_aroundcan make the random hang disappear according to zhanyuan(Cc-ed).On x86,
ptep_clear_flush_young()is exactlyptep_test_and_clear_young()after
'commitb13b1d2d8692("x86/mm: In the PTE swapout page reclaim case clear
the accessed bit instead of flushing the TLB")'But on arm64, they are different. according to Will's comments in this thread which
tried to make arm64 same with x86,
https://www.mail-archive.com/[email protected]/msg1793881.html"
This is blindly copied from x86 and isn't true for us: we don't invalidate the TLB on context switch. That means our window for keeping the stale entries around is potentially much bigger and might not be a great idea.Quoted 5 lines
If we roll a TLB invalidation routine without the trailing DSB, what sort of performance does that get you?
"
We shouldn't thinkptep_clear_flush_young()is safe enough in LRU to clear PTE young? Any comments from Will?Given that this issue is specific to the multi-gen LRU work, I think Yu is the best person to comment. However, looking quickly at your analysis above, I wonder if the code is relying on this sequence:
ptep_test_and_clear_young(vma, address, ptep); ptep_clear_flush_young(vma, address, ptep);Quoted 5 lines
to invalidate the TLB. On arm64, that won't be the case, as the invalidation in
ptep_clear_flush_young()is predicated on the pte being young (and this patches the generic implementation inmm/pgtable-generic.c. In fact, that second function call is always going to be a no-op unless the pte became young again in the middle.thanks for your reply, sorry for failing to let you understand my question. my question is actually as below,
right nowlru_gen_look_around()is usingptep_test_and_clear_young()only without flush to clear pte for a couple of pages including the specific address:
voidlru_gen_look_around(struct page_vma_mapped_walk *pvmw){ ... for (i = 0, addr = start; addr != end; i++, addr += PAGE_SIZE) { ... if (!ptep_test_and_clear_young(pvmw->vma, addr, pte + i)) continue; ... }Quoted 6 lines
Quoted 2 lines
I wonder if it is safe to arm64. Do we need to move to
ptep_clear_flush_young()in the loop?I don't know what this code is doing, so Yu is the best person to answer that. There's nothing inherently dangerous about eliding the TLB maintenance; it really depends on the guarantees needed by the caller.
Ack.
Quoted 12 lines
However, the snippet you posted from
folio_referenced_one():| if (pvmw.pte) { | + if (lru_gen_enabled() && pte_young(*pvmw.pte) && | + !(vma->vm_flags & (VM_SEQ_READ | VM_RAND_READ))) { | + lru_gen_look_around(&pvmw); | + referenced++; | + } | + | if (ptep_clear_flush_young_notify(vma, address,Quoted 3 lines
Does seem to call
lru_gen_look_around()*and*ptep_clear_flush_young_notify(), which is what prompted my question as it looks pretty suspicious to me.The
_notifyvarint reaches into the MMU notifier --lru_gen_look_around()doesn't do that because GPA space generally has no locality. I hope this explains why both.
Quoted 6 lines
As to why the code is organized this way -- it depends on the point of view. Mine is that
lru_gen_look_around()is an add-on, since its logic is independent/separable fromptep_clear_flush_young_notify(). We can makelru_gen_look_around()includeptep_clear_flush_young_notify(), but that would make the code functionally interwinted, which is bad for my taste.
Given we used to have a flush for clear pte young in LRU, right now we are moving to nop in almost all cases for the flush unless the address becomes young exactly after look_around and before ptep_clear_flush_young_notify. It means we are actually dropping flush. So the question is, were we overcautious? we actually don't need the flush at all even without mglru? for arm64, without the flush, stale data might be used for a relatively long time
as commented in [1], does it actually harm? [1]https://www.mail-archive.com/[email protected]/msg1793881.html
Thanks
Barry
Re: [PATCH v11 07/14] mm: multi-gen LRU: exploit locality in rmap
Quoted 6 lines · Barry Song
Given we used to have a flush for clear pte young in LRU, right now we are moving to nop in almost all cases for the flush unless the address becomes young exactly after
look_aroundand beforeptep_clear_flush_young_notify. It means we are actually dropping flush. So the question is, were we overcautious? we actually don't need the flush at all even without mglru?
We stopped flushing the TLB on A bit clears on x86 back in 2014.
See commit b13b1d2d8692 ("x86/mm: In the PTE swapout page reclaim case clear the accessed bit instead of flushing the TLB").
Linus
Re: [PATCH v11 07/14] mm: multi-gen LRU: exploit locality in rmap
Quoted 9 lines · Linus Torvalds
Quoted 6 lines · Barry Song
Given we used to have a flush for clear pte young in LRU, right now we are moving to nop in almost all cases for the flush unless the address becomes young exactly after
look_aroundand beforeptep_clear_flush_young_notify. It means we are actually dropping flush. So the question is, were we overcautious? we actually don't need the flush at all even without mglru?
Quoted 4 lines
We stopped flushing the TLB on A bit clears on x86 back in 2014.
See commit
b13b1d2d8692("x86/mm: In the PTE swapout page reclaim case clear the accessed bit instead of flushing the TLB").
This is true for x86, RISC-V, powerpc and S390. but it is not true for most platforms.
There was an attempt to do the same thing in arm64:
https://www.mail-archive.com/[email protected]/msg1793830.html
but arm64 still sent a nosync tlbi and depent on a deferred to dsb :
https://www.mail-archive.com/[email protected]/msg1794484.html
Plus, generic code will also send a tlb flush:
int ptep_clear_flush_young(struct vm_area_struct *vma,
unsigned long address, pte_t *ptep)
{
int young;
young = ptep_test_and_clear_young(vma, address, ptep);
if (young)
flush_tlb_page(vma, address);
return young;
}We used to use ptep_test_and_clear_young() only in rmap.c for page_referenced() in 2.6.0:
https://git.kernel.org/pub/scm/linux/kernel/git/mpe/linux-fullhistory.git/tree/mm/rmap.c?h=v2.6.0
int page_referenced(struct page * page)
{
...
if (ptep_test_and_clear_young(p))
...
}but in 2.6.12, it has been already ptep_clear_flush_young() in page_referenced_one()
https://git.kernel.org/pub/scm/linux/kernel/git/mpe/linux-fullhistory.git/tree/mm/rmap.c?h=v2.6.12
I failed to find the history to figure out the motivation for 2.6.12 to use ptep_clear_flush_young()
in LRU, but I am still curious how using flush or not will affect LRU on those platforms whoseptep_clear_flush_young() and ptep_test_and_clear_young() are different.
Quoted 2 lines
Linus
Thanks
Barry
Re: [PATCH v11 07/14] mm: multi-gen LRU: exploit locality in rmap
Quoted 17 lines · Barry Song
Quoted 9 lines · Linus Torvalds
Quoted 6 lines · Barry Song
Given we used to have a flush for clear pte young in LRU, right now we are moving to nop in almost all cases for the flush unless the address becomes young exactly after
look_aroundand beforeptep_clear_flush_young_notify. It means we are actually dropping flush. So the question is, were we overcautious? we actually don't need the flush at all even without mglru?Quoted 4 lines
We stopped flushing the TLB on A bit clears on x86 back in 2014.
See commit
b13b1d2d8692("x86/mm: In the PTE swapout page reclaim case clear the accessed bit instead of flushing the TLB").
Quoted 7 lines
This is true for x86, RISC-V, powerpc and S390. but it is not true for most platforms.
There was an attempt to do the same thing in arm64:
https://www.mail-archive.com/[email protected]/msg1793830.html
but arm64 still sent a nosync tlbi and depent on a deferred to dsb :
https://www.mail-archive.com/[email protected]/msg1794484.html
Barry, you've already answered your own question.
Without commit 07509e10dcc7 arm64: pgtable: Fix pte_accessible():
#define pte_accessible(mm, pte) \
- (mm_tlb_flush_pending(mm) ? pte_present(pte) : pte_valid_young(pte))
+ (mm_tlb_flush_pending(mm) ? pte_present(pte) : pte_valid(pte))
You missed all TLB flushes for PTEs that have gone through ptep_test_and_clear_young() on the reclaim path. But most of the time, you got away with it, only occasional app crashes:
https://lore.kernel.org/r/CAGsJ_4w6JjuG4rn2P=d974wBOUtXUUnaZKnx+-G6a8_mSROa+Q@mail.gmail.com/
Why?
Re: [PATCH v11 07/14] mm: multi-gen LRU: exploit locality in rmap
Quoted 34 lines · Yu Zhao
Quoted 17 lines · Barry Song
Quoted 9 lines · Linus Torvalds
Quoted 6 lines · Barry Song
Given we used to have a flush for clear pte young in LRU, right now we are moving to nop in almost all cases for the flush unless the address becomes young exactly after
look_aroundand beforeptep_clear_flush_young_notify. It means we are actually dropping flush. So the question is, were we overcautious? we actually don't need the flush at all even without mglru?Quoted 4 lines
We stopped flushing the TLB on A bit clears on x86 back in 2014.
See commit
b13b1d2d8692("x86/mm: In the PTE swapout page reclaim case clear the accessed bit instead of flushing the TLB").Quoted 7 lines
This is true for x86, RISC-V, powerpc and S390. but it is not true for most platforms.
There was an attempt to do the same thing in arm64:
https://www.mail-archive.com/[email protected]/msg1793830.html
but arm64 still sent a nosync tlbi and depent on a deferred to dsb :
https://www.mail-archive.com/[email protected]/msg1794484.htmlBarry, you've already answered your own question.
Without commit
07509e10dcc7arm64: pgtable: Fixpte_accessible():#define pte_accessible(mm, pte) \- (
mm_tlb_flush_pending(mm)?pte_present(pte):pte_valid_young(pte))
+ (mm_tlb_flush_pending(mm)?pte_present(pte):pte_valid(pte))
Quoted 6 lines
You missed all TLB flushes for PTEs that have gone through
ptep_test_and_clear_young()on the reclaim path. But most of the time, you got away with it, only occasional app crashes:
https://lore.kernel.org/r/CAGsJ_4w6JjuG4rn2P=d974wBOUtXUUnaZKnx+-G6a8_mSROa+Q@mail.gmail.com/Why?
Yes. On the arm64 platform, ptep_test_and_clear_young() without flush can cause random
App to crash.ptep_test_and_clear_young() + flush won't have this kind of crashes though. But after applying commit 07509e10dcc7 arm64: pgtable: Fix pte_accessible(), on arm64,ptep_test_and_clear_young() without flush won't cause App to crash.
ptep_test_and_clear_young(), with flush, without commit 07509e10dcc7: OKptep_test_and_clear_young(), without flush, with commit 07509e10dcc7: OKptep_test_and_clear_young(), without flush, without commit 07509e10dcc7: CRASH
So is it possible that other platforms have similar problems with arm64 while commit07509e10dcc7 isn't there? but anyway, we depend on those platforms which can really use mglru to expose this kind of potential bugs.
BTW, do you think it is safe to totally remove the flush code even for the original
LRU? I don't see fundamental difference between MGLRU and LRU on this "flush" thing. Since MGLRU doesn't need flush, why does LRU need it? flush is very expensive, if we do think this flush is unnecessary, will we remove it for the original LRU as well?
BTW, look_around is a great idea. Our experiments also show some great decrease on the cpu consumption of page_referenced().
Thanks
Barry
Re: [PATCH v11 07/14] mm: multi-gen LRU: exploit locality in rmap
Quoted 51 lines · Barry Song
Quoted 34 lines · Yu Zhao
Quoted 17 lines · Barry Song
Quoted 9 lines · Linus Torvalds
Quoted 6 lines · Barry Song
Given we used to have a flush for clear pte young in LRU, right now we are moving to nop in almost all cases for the flush unless the address becomes young exactly after
look_aroundand beforeptep_clear_flush_young_notify. It means we are actually dropping flush. So the question is, were we overcautious? we actually don't need the flush at all even without mglru?Quoted 4 lines
We stopped flushing the TLB on A bit clears on x86 back in 2014.
See commit
b13b1d2d8692("x86/mm: In the PTE swapout page reclaim case clear the accessed bit instead of flushing the TLB").Quoted 7 lines
This is true for x86, RISC-V, powerpc and S390. but it is not true for most platforms.
There was an attempt to do the same thing in arm64:
https://www.mail-archive.com/[email protected]/msg1793830.html
but arm64 still sent a nosync tlbi and depent on a deferred to dsb :
https://www.mail-archive.com/[email protected]/msg1794484.htmlBarry, you've already answered your own question.
Without commit
07509e10dcc7arm64: pgtable: Fixpte_accessible():#define pte_accessible(mm, pte) \- (
mm_tlb_flush_pending(mm)?pte_present(pte):pte_valid_young(pte))
+ (mm_tlb_flush_pending(mm)?pte_present(pte):pte_valid(pte))Quoted 6 lines
You missed all TLB flushes for PTEs that have gone through
ptep_test_and_clear_young()on the reclaim path. But most of the time, you got away with it, only occasional app crashes:
https://lore.kernel.org/r/CAGsJ_4w6JjuG4rn2P=d974wBOUtXUUnaZKnx+-G6a8_mSROa+Q@mail.gmail.com/Why?
Yes. On the arm64 platform,
ptep_test_and_clear_young()without flush can cause random
App to crash.ptep_test_and_clear_young()+ flush won't have this kind of crashes though. But after applying commit07509e10dcc7arm64: pgtable: Fixpte_accessible(), on arm64,ptep_test_and_clear_young()without flush won't cause App to crash.
Quoted 3 lines
ptep_test_and_clear_young(), with flush, without commit07509e10dcc7: OKptep_test_and_clear_young(), without flush, with commit07509e10dcc7: OKptep_test_and_clear_young(), without flush, without commit07509e10dcc7: CRASH
I agree -- my question was rhetorical :)
I was trying to imply this logic:
1. We cleared the A-bit in PTEs with ptep_test_and_clear_young()
2. We missed TLB flush for those PTEs on the reclaim path, i.e., case 3 (case 1 & 2 guarantee flushes)
3. We saw crashes, but only occasionally
Assuming TLB cached those PTEs, we would have seen the crashes more often, which contradicts our observation. So the conclusion is TLB didn't cache them most of the time, meaning flushing TLB just for the sake of the A-bit isn't necessary.
Quoted 3 lines
do you think it is safe to totally remove the flush code even for the original
LRU?
Affirmative, based on not only my words, but 3rd parties':
1. Your (indirect) observation
2. Alexander's benchmark:
https://lore.kernel.org/r/BYAPR12MB271295B398729E07F31082A7CFAA0@BYAPR12MB2712.namprd12.prod.outlook.com/
3. The fundamental hardware limitation in terms of the TLB scalability (Fig. 1): https://www.usenix.org/legacy/events/osdi02/tech/full_papers/navarro/navarro.pdf
Re: [PATCH v11 07/14] mm: multi-gen LRU: exploit locality in rmap
Quoted 74 lines · Yu Zhao
Quoted 51 lines · Barry Song
Quoted 34 lines · Yu Zhao
Quoted 17 lines · Barry Song
Quoted 9 lines · Linus Torvalds
Quoted 6 lines · Barry Song
Given we used to have a flush for clear pte young in LRU, right now we are moving to nop in almost all cases for the flush unless the address becomes young exactly after
look_aroundand beforeptep_clear_flush_young_notify. It means we are actually dropping flush. So the question is, were we overcautious? we actually don't need the flush at all even without mglru?Quoted 4 lines
We stopped flushing the TLB on A bit clears on x86 back in 2014.
See commit
b13b1d2d8692("x86/mm: In the PTE swapout page reclaim case clear the accessed bit instead of flushing the TLB").Quoted 7 lines
This is true for x86, RISC-V, powerpc and S390. but it is not true for most platforms.
There was an attempt to do the same thing in arm64:
https://www.mail-archive.com/[email protected]/msg1793830.html
but arm64 still sent a nosync tlbi and depent on a deferred to dsb :
https://www.mail-archive.com/[email protected]/msg1794484.htmlBarry, you've already answered your own question.
Without commit
07509e10dcc7arm64: pgtable: Fixpte_accessible():#define pte_accessible(mm, pte) \- (
mm_tlb_flush_pending(mm)?pte_present(pte):pte_valid_young(pte))
+ (mm_tlb_flush_pending(mm)?pte_present(pte):pte_valid(pte))Quoted 6 lines
You missed all TLB flushes for PTEs that have gone through
ptep_test_and_clear_young()on the reclaim path. But most of the time, you got away with it, only occasional app crashes:
https://lore.kernel.org/r/CAGsJ_4w6JjuG4rn2P=d974wBOUtXUUnaZKnx+-G6a8_mSROa+Q@mail.gmail.com/Why?
Yes. On the arm64 platform,
ptep_test_and_clear_young()without flush can cause random
App to crash.ptep_test_and_clear_young()+ flush won't have this kind of crashes though. But after applying commit07509e10dcc7arm64: pgtable: Fixpte_accessible(), on arm64,ptep_test_and_clear_young()without flush won't cause App to crash.Quoted 3 lines
ptep_test_and_clear_young(), with flush, without commit07509e10dcc7: OKptep_test_and_clear_young(), without flush, with commit07509e10dcc7: OKptep_test_and_clear_young(), without flush, without commit07509e10dcc7: CRASHI agree -- my question was rhetorical :)
I was trying to imply this logic:
1. We cleared the A-bit in PTEs withptep_test_and_clear_young()
2. We missed TLB flush for those PTEs on the reclaim path, i.e., case 3 (case 1 & 2 guarantee flushes)
3. We saw crashes, but only occasionallyAssuming TLB cached those PTEs, we would have seen the crashes more often, which contradicts our observation. So the conclusion is TLB didn't cache them most of the time, meaning flushing TLB just for the sake of the A-bit isn't necessary.
Quoted 3 lines
do you think it is safe to totally remove the flush code even for the original
LRU?
Quoted 6 lines
Affirmative, based on not only my words, but 3rd parties':
1. Your (indirect) observation
2. Alexander's benchmark:
https://lore.kernel.org/r/BYAPR12MB271295B398729E07F31082A7CFAA0@BYAPR12MB2712.namprd12.prod.outlook.com/
3. The fundamental hardware limitation in terms of the TLB scalability (Fig. 1): https://www.usenix.org/legacy/events/osdi02/tech/full_papers/navarro/navarro.pdf
4. Intel's commit b13b1d2d8692 ("x86/mm: In the PTE swapout page reclaim case clear the accessed bit instead of flushing the TLB")
Re: [PATCH v11 07/14] mm: multi-gen LRU: exploit locality in rmap
Quoted 83 lines · Yu Zhao
Quoted 74 lines · Yu Zhao
Quoted 51 lines · Barry Song
Quoted 34 lines · Yu Zhao
Quoted 17 lines · Barry Song
Quoted 9 lines · Linus Torvalds
Quoted 6 lines · Barry Song
Given we used to have a flush for clear pte young in LRU, right now we are moving to nop in almost all cases for the flush unless the address becomes young exactly after
look_aroundand beforeptep_clear_flush_young_notify. It means we are actually dropping flush. So the question is, were we overcautious? we actually don't need the flush at all even without mglru?Quoted 4 lines
We stopped flushing the TLB on A bit clears on x86 back in 2014.
See commit
b13b1d2d8692("x86/mm: In the PTE swapout page reclaim case clear the accessed bit instead of flushing the TLB").Quoted 7 lines
This is true for x86, RISC-V, powerpc and S390. but it is not true for most platforms.
There was an attempt to do the same thing in arm64:
https://www.mail-archive.com/[email protected]/msg1793830.html
but arm64 still sent a nosync tlbi and depent on a deferred to dsb :
https://www.mail-archive.com/[email protected]/msg1794484.htmlBarry, you've already answered your own question.
Without commit
07509e10dcc7arm64: pgtable: Fixpte_accessible():#define pte_accessible(mm, pte) \- (
mm_tlb_flush_pending(mm)?pte_present(pte):pte_valid_young(pte))
+ (mm_tlb_flush_pending(mm)?pte_present(pte):pte_valid(pte))Quoted 6 lines
You missed all TLB flushes for PTEs that have gone through
ptep_test_and_clear_young()on the reclaim path. But most of the time, you got away with it, only occasional app crashes:
https://lore.kernel.org/r/CAGsJ_4w6JjuG4rn2P=d974wBOUtXUUnaZKnx+-G6a8_mSROa+Q@mail.gmail.com/Why?
Yes. On the arm64 platform,
ptep_test_and_clear_young()without flush can cause random
App to crash.ptep_test_and_clear_young()+ flush won't have this kind of crashes though. But after applying commit07509e10dcc7arm64: pgtable: Fixpte_accessible(), on arm64,ptep_test_and_clear_young()without flush won't cause App to crash.Quoted 3 lines
ptep_test_and_clear_young(), with flush, without commit07509e10dcc7: OKptep_test_and_clear_young(), without flush, with commit07509e10dcc7: OKptep_test_and_clear_young(), without flush, without commit07509e10dcc7: CRASHI agree -- my question was rhetorical :)
I was trying to imply this logic:
1. We cleared the A-bit in PTEs withptep_test_and_clear_young()
2. We missed TLB flush for those PTEs on the reclaim path, i.e., case 3 (case 1 & 2 guarantee flushes)
3. We saw crashes, but only occasionallyAssuming TLB cached those PTEs, we would have seen the crashes more often, which contradicts our observation. So the conclusion is TLB didn't cache them most of the time, meaning flushing TLB just for the sake of the A-bit isn't necessary.
Quoted 3 lines
do you think it is safe to totally remove the flush code even for the original
LRU?Quoted 6 lines
Affirmative, based on not only my words, but 3rd parties':
1. Your (indirect) observation
2. Alexander's benchmark:
https://lore.kernel.org/r/BYAPR12MB271295B398729E07F31082A7CFAA0@BYAPR12MB2712.namprd12.prod.outlook.com/
3. The fundamental hardware limitation in terms of the TLB scalability (Fig. 1): https://www.usenix.org/legacy/events/osdi02/tech/full_papers/navarro/navarro.pdf
Quoted 2 lines
4. Intel's commit
b13b1d2d8692("x86/mm: In the PTE swapout page reclaim case clear the accessed bit instead of flushing the TLB")
Hi Yu,
I am going to send a RFC based on the above discussion.
mm/rmap.c +1 -1
@@ -830,7 +830,7 @@ static bool folio_referenced_one(struct folio *folio, }if (pvmw.pte) { -if (ptep_clear_flush_young_notify (vma, address, +if (ptep_clear_young_notify (vma, address, pvmw.pte)) {/* * Don't treat a reference through
Thanks
Barry
Re: [PATCH v11 07/14] mm: multi-gen LRU: exploit locality in rmap
Quoted 96 lines · Barry Song
Quoted 83 lines · Yu Zhao
Quoted 74 lines · Yu Zhao
Quoted 51 lines · Barry Song
Quoted 34 lines · Yu Zhao
Quoted 17 lines · Barry Song
Quoted 9 lines · Linus Torvalds
Quoted 6 lines · Barry Song
Given we used to have a flush for clear pte young in LRU, right now we are moving to nop in almost all cases for the flush unless the address becomes young exactly after
look_aroundand beforeptep_clear_flush_young_notify. It means we are actually dropping flush. So the question is, were we overcautious? we actually don't need the flush at all even without mglru?Quoted 4 lines
We stopped flushing the TLB on A bit clears on x86 back in 2014.
See commit
b13b1d2d8692("x86/mm: In the PTE swapout page reclaim case clear the accessed bit instead of flushing the TLB").Quoted 7 lines
This is true for x86, RISC-V, powerpc and S390. but it is not true for most platforms.
There was an attempt to do the same thing in arm64:
https://www.mail-archive.com/[email protected]/msg1793830.html
but arm64 still sent a nosync tlbi and depent on a deferred to dsb :
https://www.mail-archive.com/[email protected]/msg1794484.htmlBarry, you've already answered your own question.
Without commit
07509e10dcc7arm64: pgtable: Fixpte_accessible():#define pte_accessible(mm, pte) \- (
mm_tlb_flush_pending(mm)?pte_present(pte):pte_valid_young(pte))
+ (mm_tlb_flush_pending(mm)?pte_present(pte):pte_valid(pte))Quoted 6 lines
You missed all TLB flushes for PTEs that have gone through
ptep_test_and_clear_young()on the reclaim path. But most of the time, you got away with it, only occasional app crashes:
https://lore.kernel.org/r/CAGsJ_4w6JjuG4rn2P=d974wBOUtXUUnaZKnx+-G6a8_mSROa+Q@mail.gmail.com/Why?
Yes. On the arm64 platform,
ptep_test_and_clear_young()without flush can cause random
App to crash.ptep_test_and_clear_young()+ flush won't have this kind of crashes though. But after applying commit07509e10dcc7arm64: pgtable: Fixpte_accessible(), on arm64,ptep_test_and_clear_young()without flush won't cause App to crash.Quoted 3 lines
ptep_test_and_clear_young(), with flush, without commit07509e10dcc7: OKptep_test_and_clear_young(), without flush, with commit07509e10dcc7: OKptep_test_and_clear_young(), without flush, without commit07509e10dcc7: CRASHI agree -- my question was rhetorical :)
I was trying to imply this logic:
1. We cleared the A-bit in PTEs withptep_test_and_clear_young()
2. We missed TLB flush for those PTEs on the reclaim path, i.e., case 3 (case 1 & 2 guarantee flushes)
3. We saw crashes, but only occasionallyAssuming TLB cached those PTEs, we would have seen the crashes more often, which contradicts our observation. So the conclusion is TLB didn't cache them most of the time, meaning flushing TLB just for the sake of the A-bit isn't necessary.
Quoted 3 lines
do you think it is safe to totally remove the flush code even for the original
LRU?Quoted 6 lines
Affirmative, based on not only my words, but 3rd parties':
1. Your (indirect) observation
2. Alexander's benchmark:
https://lore.kernel.org/r/BYAPR12MB271295B398729E07F31082A7CFAA0@BYAPR12MB2712.namprd12.prod.outlook.com/
3. The fundamental hardware limitation in terms of the TLB scalability (Fig. 1): https://www.usenix.org/legacy/events/osdi02/tech/full_papers/navarro/navarro.pdfQuoted 2 lines
4. Intel's commit
b13b1d2d8692("x86/mm: In the PTE swapout page reclaim case clear the accessed bit instead of flushing the TLB")Hi Yu,
I am going to send a RFC based on the above discussion.mm/rmap.c +0
-0
mm/rmap.c static bool folio_referenced_one(struct folio *folio, · line 836
Quoted 8 lines
mm/rmap.c +1
-1@@ -830,7 +830,7 @@ static bool folio_referenced_one(struct folio *folio, }if (pvmw.pte) { -if (ptep_clear_flush_young_notify (vma, address, +if (ptep_clear_young_notify (vma, address, pvmw.pte)) {/* * Don't treat a reference through
Thanks!
This might make a difference on my 64 core Altra -- I'll test after you post the RFC.
Re: [PATCH v11 07/14] mm: multi-gen LRU: exploit locality in rmap
Quoted 109 lines · Yu Zhao
Quoted 96 lines · Barry Song
Quoted 83 lines · Yu Zhao
Quoted 74 lines · Yu Zhao
Quoted 51 lines · Barry Song
Quoted 34 lines · Yu Zhao
Quoted 17 lines · Barry Song
Quoted 9 lines · Linus Torvalds
Quoted 6 lines · Barry Song
Given we used to have a flush for clear pte young in LRU, right now we are moving to nop in almost all cases for the flush unless the address becomes young exactly after
look_aroundand beforeptep_clear_flush_young_notify. It means we are actually dropping flush. So the question is, were we overcautious? we actually don't need the flush at all even without mglru?Quoted 4 lines
We stopped flushing the TLB on A bit clears on x86 back in 2014.
See commit
b13b1d2d8692("x86/mm: In the PTE swapout page reclaim case clear the accessed bit instead of flushing the TLB").Quoted 7 lines
This is true for x86, RISC-V, powerpc and S390. but it is not true for most platforms.
There was an attempt to do the same thing in arm64:
https://www.mail-archive.com/[email protected]/msg1793830.html
but arm64 still sent a nosync tlbi and depent on a deferred to dsb :
https://www.mail-archive.com/[email protected]/msg1794484.htmlBarry, you've already answered your own question.
Without commit
07509e10dcc7arm64: pgtable: Fixpte_accessible():#define pte_accessible(mm, pte) \- (
mm_tlb_flush_pending(mm)?pte_present(pte):pte_valid_young(pte))
+ (mm_tlb_flush_pending(mm)?pte_present(pte):pte_valid(pte))Quoted 6 lines
You missed all TLB flushes for PTEs that have gone through
ptep_test_and_clear_young()on the reclaim path. But most of the time, you got away with it, only occasional app crashes:
https://lore.kernel.org/r/CAGsJ_4w6JjuG4rn2P=d974wBOUtXUUnaZKnx+-G6a8_mSROa+Q@mail.gmail.com/Why?
Yes. On the arm64 platform,
ptep_test_and_clear_young()without flush can cause random
App to crash.ptep_test_and_clear_young()+ flush won't have this kind of crashes though. But after applying commit07509e10dcc7arm64: pgtable: Fixpte_accessible(), on arm64,ptep_test_and_clear_young()without flush won't cause App to crash.Quoted 3 lines
ptep_test_and_clear_young(), with flush, without commit07509e10dcc7: OKptep_test_and_clear_young(), without flush, with commit07509e10dcc7: OKptep_test_and_clear_young(), without flush, without commit07509e10dcc7: CRASHI agree -- my question was rhetorical :)
I was trying to imply this logic:
1. We cleared the A-bit in PTEs withptep_test_and_clear_young()
2. We missed TLB flush for those PTEs on the reclaim path, i.e., case 3 (case 1 & 2 guarantee flushes)
3. We saw crashes, but only occasionallyAssuming TLB cached those PTEs, we would have seen the crashes more often, which contradicts our observation. So the conclusion is TLB didn't cache them most of the time, meaning flushing TLB just for the sake of the A-bit isn't necessary.
Quoted 3 lines
do you think it is safe to totally remove the flush code even for the original
LRU?Quoted 6 lines
Affirmative, based on not only my words, but 3rd parties':
1. Your (indirect) observation
2. Alexander's benchmark:
https://lore.kernel.org/r/BYAPR12MB271295B398729E07F31082A7CFAA0@BYAPR12MB2712.namprd12.prod.outlook.com/
3. The fundamental hardware limitation in terms of the TLB scalability (Fig. 1): https://www.usenix.org/legacy/events/osdi02/tech/full_papers/navarro/navarro.pdfQuoted 2 lines
4. Intel's commit
b13b1d2d8692("x86/mm: In the PTE swapout page reclaim case clear the accessed bit instead of flushing the TLB")Hi Yu,
I am going to send a RFC based on the above discussion.mm/rmap.c +0
-0mm/rmap.c static bool folio_referenced_one(struct folio *folio, · line 836
Quoted 8 lines
mm/rmap.c +1
-1@@ -830,7 +830,7 @@ static bool folio_referenced_one(struct folio *folio, }if (pvmw.pte) { -if (ptep_clear_flush_young_notify (vma, address, +if (ptep_clear_young_notify (vma, address, pvmw.pte)) {/* * Don't treat a reference through Thanks!
Quoted 2 lines
This might make a difference on my 64 core Altra -- I'll test after you post the RFC.
Also, IIRC, it made no difference on POWER9 because POWER9 flushes TBL regardless which variant is used.
Re: [PATCH v11 07/14] mm: multi-gen LRU: exploit locality in rmap
Quoted 111 lines · Yu Zhao
Quoted 109 lines · Yu Zhao
Quoted 96 lines · Barry Song
Quoted 83 lines · Yu Zhao
Quoted 74 lines · Yu Zhao
Quoted 51 lines · Barry Song
Quoted 34 lines · Yu Zhao
Quoted 17 lines · Barry Song
Quoted 9 lines · Linus Torvalds
Quoted 6 lines · Barry Song
Given we used to have a flush for clear pte young in LRU, right now we are moving to nop in almost all cases for the flush unless the address becomes young exactly after
look_aroundand beforeptep_clear_flush_young_notify. It means we are actually dropping flush. So the question is, were we overcautious? we actually don't need the flush at all even without mglru?Quoted 4 lines
We stopped flushing the TLB on A bit clears on x86 back in 2014.
See commit
b13b1d2d8692("x86/mm: In the PTE swapout page reclaim case clear the accessed bit instead of flushing the TLB").Quoted 7 lines
This is true for x86, RISC-V, powerpc and S390. but it is not true for most platforms.
There was an attempt to do the same thing in arm64:
https://www.mail-archive.com/[email protected]/msg1793830.html
but arm64 still sent a nosync tlbi and depent on a deferred to dsb :
https://www.mail-archive.com/[email protected]/msg1794484.htmlBarry, you've already answered your own question.
Without commit
07509e10dcc7arm64: pgtable: Fixpte_accessible():#define pte_accessible(mm, pte) \- (
mm_tlb_flush_pending(mm)?pte_present(pte):pte_valid_young(pte))
+ (mm_tlb_flush_pending(mm)?pte_present(pte):pte_valid(pte))Quoted 6 lines
You missed all TLB flushes for PTEs that have gone through
ptep_test_and_clear_young()on the reclaim path. But most of the time, you got away with it, only occasional app crashes:
https://lore.kernel.org/r/CAGsJ_4w6JjuG4rn2P=d974wBOUtXUUnaZKnx+-G6a8_mSROa+Q@mail.gmail.com/Why?
Yes. On the arm64 platform,
ptep_test_and_clear_young()without flush can cause random
App to crash.ptep_test_and_clear_young()+ flush won't have this kind of crashes though. But after applying commit07509e10dcc7arm64: pgtable: Fixpte_accessible(), on arm64,ptep_test_and_clear_young()without flush won't cause App to crash.Quoted 3 lines
ptep_test_and_clear_young(), with flush, without commit07509e10dcc7: OKptep_test_and_clear_young(), without flush, with commit07509e10dcc7: OKptep_test_and_clear_young(), without flush, without commit07509e10dcc7: CRASHI agree -- my question was rhetorical :)
I was trying to imply this logic:
1. We cleared the A-bit in PTEs withptep_test_and_clear_young()
2. We missed TLB flush for those PTEs on the reclaim path, i.e., case 3 (case 1 & 2 guarantee flushes)
3. We saw crashes, but only occasionallyAssuming TLB cached those PTEs, we would have seen the crashes more often, which contradicts our observation. So the conclusion is TLB didn't cache them most of the time, meaning flushing TLB just for the sake of the A-bit isn't necessary.
Quoted 3 lines
do you think it is safe to totally remove the flush code even for the original
LRU?Quoted 6 lines
Affirmative, based on not only my words, but 3rd parties':
1. Your (indirect) observation
2. Alexander's benchmark:
https://lore.kernel.org/r/BYAPR12MB271295B398729E07F31082A7CFAA0@BYAPR12MB2712.namprd12.prod.outlook.com/
3. The fundamental hardware limitation in terms of the TLB scalability (Fig. 1): https://www.usenix.org/legacy/events/osdi02/tech/full_papers/navarro/navarro.pdfQuoted 2 lines
4. Intel's commit
b13b1d2d8692("x86/mm: In the PTE swapout page reclaim case clear the accessed bit instead of flushing the TLB")Hi Yu,
I am going to send a RFC based on the above discussion.mm/rmap.c +0
-0mm/rmap.c static bool folio_referenced_one(struct folio *folio, · line 836
Quoted 8 lines
mm/rmap.c +1
-1@@ -830,7 +830,7 @@ static bool folio_referenced_one(struct folio *folio, }if (pvmw.pte) { -if (ptep_clear_flush_young_notify (vma, address, +if (ptep_clear_young_notify (vma, address, pvmw.pte)) {/* * Don't treat a reference through Thanks!
Quoted 5 lines
Quoted 2 lines
This might make a difference on my 64 core Altra -- I'll test after you post the RFC.
Also, IIRC, it made no difference on POWER9 because POWER9 flushes TBL regardless which variant is used.
^^^^^^^ doesn't flush
I just verified this on POWER9. So on top of exhibit 1-4, we got:
5. 3cb1aa7aa3940 ("powerpc/64s: Implement ptep_clear_flush_young
that does not flush TLBs")
Re: [PATCH v11 07/14] mm: multi-gen LRU: exploit locality in rmap
Quoted 116 lines · Yu Zhao
Quoted 114 lines · Yu Zhao
Quoted 111 lines · Yu Zhao
Quoted 96 lines · Barry Song
Quoted 83 lines · Yu Zhao
Quoted 74 lines · Yu Zhao
Quoted 51 lines · Barry Song
Quoted 34 lines · Yu Zhao
Quoted 17 lines · Barry Song
Quoted 9 lines · Linus Torvalds
Quoted 6 lines · Barry Song
Given we used to have a flush for clear pte young in LRU, right now we are moving to nop in almost all cases for the flush unless the address becomes young exactly after
look_aroundand beforeptep_clear_flush_young_notify. It means we are actually dropping flush. So the question is, were we overcautious? we actually don't need the flush at all even without mglru?Quoted 4 lines
We stopped flushing the TLB on A bit clears on x86 back in 2014.
See commit
b13b1d2d8692("x86/mm: In the PTE swapout page reclaim case clear the accessed bit instead of flushing the TLB").Quoted 7 lines
This is true for x86, RISC-V, powerpc and S390. but it is not true for most platforms.
There was an attempt to do the same thing in arm64:
https://www.mail-archive.com/[email protected]/msg1793830.html
but arm64 still sent a nosync tlbi and depent on a deferred to dsb :
https://www.mail-archive.com/[email protected]/msg1794484.htmlBarry, you've already answered your own question.
Without commit
07509e10dcc7arm64: pgtable: Fixpte_accessible():#define pte_accessible(mm, pte) \- (
mm_tlb_flush_pending(mm)?pte_present(pte):pte_valid_young(pte))
+ (mm_tlb_flush_pending(mm)?pte_present(pte):pte_valid(pte))Quoted 6 lines
You missed all TLB flushes for PTEs that have gone through
ptep_test_and_clear_young()on the reclaim path. But most of the time, you got away with it, only occasional app crashes:
https://lore.kernel.org/r/CAGsJ_4w6JjuG4rn2P=d974wBOUtXUUnaZKnx+-G6a8_mSROa+Q@mail.gmail.com/Why?
Yes. On the arm64 platform,
ptep_test_and_clear_young()without flush can cause random
App to crash.ptep_test_and_clear_young()+ flush won't have this kind of crashes though. But after applying commit07509e10dcc7arm64: pgtable: Fixpte_accessible(), on arm64,ptep_test_and_clear_young()without flush won't cause App to crash.Quoted 3 lines
ptep_test_and_clear_young(), with flush, without commit07509e10dcc7: OKptep_test_and_clear_young(), without flush, with commit07509e10dcc7: OKptep_test_and_clear_young(), without flush, without commit07509e10dcc7: CRASHI agree -- my question was rhetorical :)
I was trying to imply this logic:
1. We cleared the A-bit in PTEs withptep_test_and_clear_young()
2. We missed TLB flush for those PTEs on the reclaim path, i.e., case 3 (case 1 & 2 guarantee flushes)
3. We saw crashes, but only occasionallyAssuming TLB cached those PTEs, we would have seen the crashes more often, which contradicts our observation. So the conclusion is TLB didn't cache them most of the time, meaning flushing TLB just for the sake of the A-bit isn't necessary.
Quoted 3 lines
do you think it is safe to totally remove the flush code even for the original
LRU?Quoted 6 lines
Affirmative, based on not only my words, but 3rd parties':
1. Your (indirect) observation
2. Alexander's benchmark:
https://lore.kernel.org/r/BYAPR12MB271295B398729E07F31082A7CFAA0@BYAPR12MB2712.namprd12.prod.outlook.com/
3. The fundamental hardware limitation in terms of the TLB scalability (Fig. 1): https://www.usenix.org/legacy/events/osdi02/tech/full_papers/navarro/navarro.pdfQuoted 2 lines
4. Intel's commit
b13b1d2d8692("x86/mm: In the PTE swapout page reclaim case clear the accessed bit instead of flushing the TLB")Hi Yu,
I am going to send a RFC based on the above discussion.mm/rmap.c +0
-0mm/rmap.c static bool folio_referenced_one(struct folio *folio, · line 836
Quoted 8 lines
mm/rmap.c +1
-1@@ -830,7 +830,7 @@ static bool folio_referenced_one(struct folio *folio, }if (pvmw.pte) { -if (ptep_clear_flush_young_notify (vma, address, +if (ptep_clear_young_notify (vma, address, pvmw.pte)) {/* * Don't treat a reference through Thanks!
This might make a difference on my 64 core Altra -- I'll test after you post the RFC.
Quoted 7 lines
Quoted 2 lines
Also, IIRC, it made no difference on POWER9 because POWER9 flushes TBL regardless which variant is used.
^^^^^^^ doesn't flush
I just verified this on POWER9. So on top of exhibit 1-4, we got:
5.3cb1aa7aa3940("powerpc/64s: Implementptep_clear_flush_young
that does not flush TLBs")
Thanks, Yu. I put a rfc,
https://lore.kernel.org/lkml/[email protected]/
we may clarify everything in that thread :-)
Thanks
Barry
Re: [PATCH v11 07/14] mm: multi-gen LRU: exploit locality in rmap
In reply to Barry SongQuoted 4 lines · Barry Song
Quoted 2 lines · Barry Song
On Wed, May 18, 2022 at 4:49 PM Yu Zhao <[email protected]> wrote:
...
Quoted 6 lines
I can't really explain why we are getting a random app/java vm crash in monkey test by using
ptep_test_and_clear_young()only inlru_gen_look_around()on an armv8-a machine without hardware PTE young support.Moving to
ptep_clear_flush_young()inlook_aroundcan make the random hang disappear according to zhanyuan(Cc-ed).
This sounds too familiar -- let me ask again: was the following commit included during the test?
07509e10dcc7 arm64: pgtable: Fix pte_accessible()
If not, it will cause exactly the problem you described. And what about this one?
e914d8f00391 mm: fix unexpected zeroed page mapping with zram swap
Missing it also causes userspace memory corruption on Android, i.e., random app crashes.
Quoted 15 lines
On x86,
ptep_clear_flush_young()is exactlyptep_test_and_clear_young()after
'commitb13b1d2d8692("x86/mm: In the PTE swapout page reclaim case clear
the accessed bit instead of flushing the TLB")'But on arm64, they are different. according to Will's comments in this thread which
tried to make arm64 same with x86,
https://www.mail-archive.com/[email protected]/msg1793881.html"
This is blindly copied from x86 and isn't true for us: we don't invalidate the TLB on context switch. That means our window for keeping the stale entries around is potentially much bigger and might not be a great idea.
Quoted 9 lines
If we roll a TLB invalidation routine without the trailing DSB, what sort of performance does that get you?
"
We shouldn't thinkptep_clear_flush_young()is safe enough in LRU to clear PTE young? Any comments from Will?Quoted 3 lines
btw,
lru_gen_look_around()has already included 'address', are we doing pte check for 'address' twice here?
Explained in the previous reply. Hope that clarifies things.
Re: [PATCH v11 07/14] mm: multi-gen LRU: exploit locality in rmap
Quoted 21 lines · Yu Zhao
Quoted 4 lines · Barry Song
Quoted 2 lines · Barry Song
On Wed, May 18, 2022 at 4:49 PM Yu Zhao <[email protected]> wrote:
...
Quoted 6 lines
I can't really explain why we are getting a random app/java vm crash in monkey test by using
ptep_test_and_clear_young()only inlru_gen_look_around()on an armv8-a machine without hardware PTE young support.Moving to
ptep_clear_flush_young()inlook_aroundcan make the random hang disappear according to zhanyuan(Cc-ed).This sounds too familiar -- let me ask again: was the following commit included during the test?
07509e10dcc7arm64: pgtable: Fixpte_accessible()
Quoted 8 lines
If not, it will cause exactly the problem you described. And what about this one?
e914d8f00391mm: fix unexpected zeroed page mapping with zram swapMissing it also causes userspace memory corruption on Android, i.e., random app crashes.
According to zhanyuan's testing, we can confirm the above two commits can fix the random android crash.
Thanks
Barry
Re: [PATCH v11 07/14] mm: multi-gen LRU: exploit locality in rmap
In reply to Barry SongQuoted 2 lines · Barry Song
On Wed, May 18, 2022 at 4:49 PM Yu Zhao <[email protected]> wrote:
...
Quoted 35 lines
mm/rmap.c static bool folio_referenced_one(struct folio *folio, · line 831
Quoted 11 lines
mm/rmap.c +6
-0@@ -821,6 +822,12 @@ static bool folio_referenced_one(struct folio *folio, }if (pvmw.pte) { +if (lru_gen_enabled () &&pte_young (*pvmw.pte) && + !(vma->vm_flags & (VM_SEQ_READ |VM_RAND_READ ))) { +lru_gen_look_around (&pvmw); + referenced++; + } + if (ptep_clear_flush_young_notify(vma, address,Hello, Yu.
look_around()is callingptep_test_and_clear_young(pvmw->vma, addr, pte + i)only without flush and notify. for flush, there is a tlb operation for arm64:
static inline int ptep_clear_flush_young(structvm_area_struct*vma,
unsigned long address,pte_t*ptep){ int young = ptep_test_and_clear_young(vma, address, ptep); if (young) { /* * We can elide the trailing DSB here since the worst that can * happen is that a CPU continues to use the young entry in its * TLB and we mistakenly reclaim the associated page. The * window for such an event is bounded by the next * context-switch, which provides a DSB to complete the TLB * invalidation. */ flush_tlb_page_nosync(vma, address); } return young; }
Quoted 1 line
Does it mean the current kernel is over cautious?
Hi Barry,
This is up to individual archs. For x86, ptep_clear_flush_young() is ptep_test_and_clear_young(). For arm64, I'd say yes, based on Figure 1 of Navarro, Juan, et al. "Practical, transparent operating system support for superpages." [1].
int ptep_clear_flush_young(struct vm_area_struct *vma,
unsigned long address, pte_t *ptep)
{
/*
* On x86 CPUs, clearing the accessed bit without a TLB flush
* doesn't cause data corruption. [ It could cause incorrect
* page aging and the (mistaken) reclaim of hot pages, but the
* chance of that should be relatively low. ]
*
* So as a performance optimization don't flush the TLB when
* clearing the accessed bit, it will eventually be flushed by
* a context switch or a VM operation anyway. [ In the rare
* event of it not getting flushed for a long time the delay
* shouldn't really matter because there's no real memory
* pressure for swapout to react to. ]
*/
return ptep_test_and_clear_young(vma, address, ptep);
}
[1] https://www.usenix.org/legacy/events/osdi02/tech/full_papers/navarro/navarro.pdf
Quoted 2 lines
is it
safe to callptep_test_and_clear_young()only?
Yes. Though the h/w A-bit is designed to allow OSes to skip TLB flushes when unmapping, the Linux kernel doesn't do this.
Quoted 2 lines
btw,
lru_gen_look_around()has already included 'address', are we doing pte check for 'address' twice here?
Yes for host MMU but no KVM MMU. ptep_clear_flush_young_notify() goes into the MMU notifier. We don't use the _notify variant in lru_gen_look_around() because GPA space generally exhibits no memory locality.
Thanks.
PATCH v11 8/14 mm: multi-gen LRU: support page table walks
In reply to Yu ZhaoTo further exploit spatial locality, the aging prefers to walk page tables to search for young PTEs and promote hot pages. A kill switch will be added in the next patch to disable this behavior. When disabled, the aging relies on the rmap only.
NB: this behavior has nothing similar with the page table scanning in the 2.4 kernel [1], which searches page tables for old PTEs, adds cold pages to swapcache and unmaps them.
To avoid confusion, the term "iteration" specifically means the traversal of an entire mm_struct list; the term "walk" will be applied to page tables and the rmap, as usual.
An mm_struct list is maintained for each memcg, and an mm_struct follows its owner task to the new memcg when this task is migrated. Given an lruvec, the aging iterates lruvec_memcg()->mm_list and calls walk_page_range() with each mm_struct on this list to promote hot pages before it increments max_seq.
When multiple page table walkers iterate the same list, each of them gets a unique mm_struct; therefore they can run concurrently. Page table walkers ignore any misplaced pages, e.g., if an mm_struct was migrated, pages it left in the previous memcg will not be promoted when its current memcg is under reclaim. Similarly, page table walkers will not promote pages from nodes other than the one under reclaim.
This patch uses the following optimizations when walking page tables:
1. It tracks the usage of mm_struct's between context switches so that page table walkers can skip processes that have been sleeping since the last iteration.
2. It uses generational Bloom filters to record populated branches so that page table walkers can reduce their search space based on the query results, e.g., to skip page tables containing mostly holes or misplaced pages.
3. It takes advantage of the accessed bit in non-leaf PMD entries when CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG=y.
4. It does not zigzag between a PGD table and the same PMD table spanning multiple VMAs. IOW, it finishes all the VMAs within the range of the same PMD table before it returns to a PGD table. This improves the cache performance for workloads that have large numbers of tiny VMAs [2], especially when CONFIG_PGTABLE_LEVELS=5.
Server benchmark results:
Single workload:
fio (buffered I/O): no change
Single workload:
memcached (anon): +[8, 10]%
Ops/sec KB/sec
patch1-7: 1193918.93 46438.15
patch1-8: 1301954.44 50640.27 Configurations:
no change
Client benchmark results:
kswapd profiles:
patch1-7
45.90% lzo1x_1_do_compress (real work)
9.14% page_vma_mapped_walk
6.81% _raw_spin_unlock_irq
2.80% ptep_clear_flush
2.34% __zram_bvec_write
2.29% do_raw_spin_lock
1.84% lru_gen_look_around
1.78% memmove
1.74% obj_malloc
1.50% free_unref_page_list
patch1-8
46.96% lzo1x_1_do_compress (real work)
7.55% page_vma_mapped_walk
5.89% _raw_spin_unlock_irq
3.33% walk_pte_range
2.65% ptep_clear_flush
2.23% __zram_bvec_write
2.08% do_raw_spin_lock
1.83% memmove
1.65% obj_malloc
1.47% free_unref_page_list Configurations:
no change
Thanks to the following developers for their efforts [3].
kernel test robot <[email protected]>
[1] https://lwn.net/Articles/23732/
[2] https://source.android.com/devices/tech/debug/scudo
[3] https://lore.kernel.org/r/[email protected]/
--- fs/exec.c | 2 + include/linux/memcontrol.h | 5 + include/linux/mm_types.h | 77 +++ include/linux/mmzone.h | 61 ++- include/linux/swap.h | 4 + kernel/exit.c | 1 + kernel/fork.c | 9 + kernel/sched/core.c | 1 + mm/memcontrol.c | 25 + mm/vmscan.c | 981 ++++++++++++++++++++++++++++++++++++- 10 files changed, 1150 insertions(+), 16 deletions(-)
fs/exec.c +2 -0
@@ -1011,6 +1011,7 @@ static int exec_mmap(struct mm_struct *mm) active_mm = tsk->active_mm; tsk->active_mm = mm; tsk->mm = mm; +lru_gen_add_mm (mm);/* * This prevents preemption while active_mm is being loaded and * it and mm are being updated, which could cause problems for
@@ -1023,6 +1024,7 @@ static int exec_mmap(struct mm_struct *mm)activate_mm (active_mm, mm);if (IS_ENABLED (CONFIG_ARCH_WANT_IRQS_OFF_ACTIVATE_MM ))local_irq_enable (); +lru_gen_use_mm (mm); tsk->mm->vmacache_seqnum =0 ;vmacache_flush (tsk);task_unlock (tsk);
include/linux/memcontrol.h +5 -0
@@ -344,6 +344,11 @@ struct mem_cgroup {struct deferred_split deferred_split_queue;#endif +#ifdef CONFIG_LRU_GEN +/* per-memcg mm_struct list */ +struct lru_gen_mm_list mm_list; +#endif +struct mem_cgroup_per_node *nodeinfo[]; };
include/linux/mm_types.h +77 -0
101 lines not on this page · raw message
include/linux/mmzone.h +60 -1
85 lines not on this page · raw message
include/linux/swap.h +4 -0
@@ -137,6 +137,10 @@ union swap_header {*/ struct reclaim_state {unsigned long reclaimed_slab; +#ifdef CONFIG_LRU_GEN +/* per-thread mm walk data */ +struct lru_gen_mm_walk *mm_walk; +#endif };#ifdef __KERNEL__
kernel/exit.c +1 -0
@@ -466,6 +466,7 @@ void mm_update_next_owner(struct mm_struct *mm) goto retry; }WRITE_ONCE (mm->owner, c); +lru_gen_migrate_mm (mm);task_unlock (c);put_task_struct (c); }
kernel/fork.c +9 -0
@@ -1149,6 +1149,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, goto fail_nocontext; mm->user_ns =get_user_ns (user_ns); +lru_gen_init_mm (mm);return mm; fail_nocontext:
@@ -1191,6 +1192,7 @@ static inline void __mmput(struct mm_struct *mm) }if (mm->binfmt)module_put (mm->binfmt->module); +lru_gen_del_mm (mm);mmdrop (mm); }
@@ -2660,6 +2662,13 @@ pid_t kernel_clone(struct kernel_clone_args *args)get_task_struct (p); } +if (IS_ENABLED (CONFIG_LRU_GEN ) && !(clone_flags &CLONE_VM )) { +/* lock the task to synchronize with memcg migration */ +task_lock (p); +lru_gen_add_mm (p->mm); +task_unlock (p); + } +wake_up_new_task (p);/* forking complete and child started to run, tell ptracer */
kernel/sched/core.c +1 -0
@@ -5057,6 +5057,7 @@ context_switch(struct rq *rq, struct task_struct *prev,* finish_task_switch()'s mmdrop(). */ switch_mm_irqs_off (prev->active_mm, next->mm, next); +lru_gen_use_mm (next->mm);if (!prev->mm) {// from kernel /* will mmdrop() in finish_task_switch(). */
mm/memcontrol.c +25 -0
@@ -6093,6 +6093,30 @@ static void mem_cgroup_move_task(void) }#endif +#ifdef CONFIG_LRU_GEN +static void mem_cgroup_attach (struct cgroup_taskset *tset) +{ +struct task_struct *task; +struct cgroup_subsys_state *css; + +/* find the first leader if there is any */ +cgroup_taskset_for_each_leader (task, css, tset) +break ; + +if (!task) +return ; + +task_lock (task); +if (task->mm && task->mm->owner == task) +lru_gen_migrate_mm (task->mm); +task_unlock (task); +} +#else +static void mem_cgroup_attach (struct cgroup_taskset *tset) +{ +} +#endif /* CONFIG_LRU_GEN */ +static int seq_puts_memcg_tunable (struct seq_file *m,unsigned long value) {if (value ==PAGE_COUNTER_MAX )
@@ -6438,6 +6462,7 @@ struct cgroup_subsys memory_cgrp_subsys = { .css_reset = mem_cgroup_css_reset, .css_rstat_flush = mem_cgroup_css_rstat_flush, .can_attach = mem_cgroup_can_attach, + .attach = mem_cgroup_attach, .cancel_attach = mem_cgroup_cancel_attach, .post_attach = mem_cgroup_move_task, .dfl_cftypes = memory_files,
mm/vmscan.c +966 -15
1,143 lines not on this page · raw message
Signature · 1 line
2.36.0.550.gb090851708-goog
PATCH v11 9/14 mm: multi-gen LRU: optimize multiple memcgs
In reply to Yu ZhaoWhen multiple memcgs are available, it is possible to make better choices based on generations and tiers and therefore improve the overall performance under global memory pressure. This patch adds a rudimentary optimization to select memcgs that can drop single-use unmapped clean pages first. Doing so reduces the chance of going into the aging path or swapping. These two operations can be costly.
A typical example that benefits from this optimization is a server running mixed types of workloads, e.g., heavy anon workload in one memcg and heavy buffered I/O workload in the other.
Though this optimization can be applied to both kswapd and direct reclaim, it is only added to kswapd to keep the patchset manageable. Later improvements will cover the direct reclaim path.
Server benchmark results:
Mixed workloads:
fio (buffered I/O): +[1, 3]%
IOPS BW
patch1-8: 2154k 8415MiB/s
patch1-9: 2205k 8613MiB/s
memcached (anon): +[132, 136]%
Ops/sec KB/sec
patch1-8: 819618.49 31838.48
patch1-9: 1916516.06 74447.92Mixed workloads:
fio (buffered I/O): +[59, 61]%
IOPS BW
5.18-rc1: 1378k 5385MiB/s
patch1-9: 2205k 8613MiB/s
memcached (anon): +[229, 233]%
Ops/sec KB/sec
5.18-rc1: 578946.00 22489.44
patch1-9: 1916516.06 74447.92Configurations:
(changes since patch 6)
cat mixed.sh
modprobe brd rd_nr=2 rd_size=56623104
swapoff -a
mkswap /dev/ram0
swapon /dev/ram0
mkfs.ext4 /dev/ram1
mount -t ext4 /dev/ram1 /mnt
memtier_benchmark -S /var/run/memcached/memcached.sock \
-P memcache_binary -n allkeys --key-minimum=1 \
--key-maximum=50000000 --key-pattern=P:P -c 1 -t 36 \
--ratio 1:0 --pipeline 8 -d 2000
fio -name=mglru --numjobs=36 --directory=/mnt --size=1408m \
--buffered=1 --ioengine=io_uring --iodepth=128 \
--iodepth_batch_submit=32 --iodepth_batch_complete=32 \
--rw=randread --random_distribution=random --norandommap \
--time_based --ramp_time=10m --runtime=90m --group_reporting &
pid=$!
sleep 200
memtier_benchmark -S /var/run/memcached/memcached.sock \
-P memcache_binary -n allkeys --key-minimum=1 \
--key-maximum=50000000 --key-pattern=R:R -c 1 -t 36 \
--ratio 0:1 --pipeline 8 --randomize --distinct-client-seed
kill -INT $pid
waitClient benchmark results:
no change (CONFIG_MEMCG=n)
--- mm/vmscan.c | 45 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 41 insertions(+), 4 deletions(-)
mm/vmscan.c +41 -4
@@ -128,6 +128,13 @@ struct scan_control {/* Always discard instead of demoting to lower tier memory */ unsigned int no_demotion:1; +#ifdef CONFIG_LRU_GEN +/* help make better choices when multiple memcgs are available */ + unsigned int memcgs_need_aging:1; + unsigned int memcgs_need_swapping:1; + unsigned int memcgs_avoid_swapping:1; +#endif +/* Allocation order */ s8 order;
@@ -4290,6 +4297,22 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc)VM_WARN_ON_ONCE (!current_is_kswapd ()); +/* +* To reduce the chance of going into the aging path or swapping, which +* can be costly, optimistically skip them unless their corresponding +* flags were cleared in the eviction path. This improves the overall +* performance when multiple memcgs are available. +*/ +if (!sc->memcgs_need_aging) { + sc->memcgs_need_aging = true; + sc->memcgs_avoid_swapping = !sc->memcgs_need_swapping; + sc->memcgs_need_swapping = true; +return ; + } + + sc->memcgs_need_swapping = true; + sc->memcgs_avoid_swapping = true; + current->reclaim_state->mm_walk = &pgdat->mm_walk; memcg =mem_cgroup_iter (NULL ,NULL ,NULL );
@@ -4699,7 +4722,8 @@ static int isolate_folios(struct lruvec *lruvec, struct scan_control *sc, int swreturn scanned; } -static int evict_folios (struct lruvec *lruvec,struct scan_control *sc,int swappiness) +static int evict_folios (struct lruvec *lruvec,struct scan_control *sc,int swappiness, +bool *swapped) {int type;int scanned;
@@ -4765,6 +4789,9 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap sc->nr_reclaimed += reclaimed; +if (type ==LRU_GEN_ANON && swapped) + *swapped = true; +return scanned; }
@@ -4793,8 +4820,10 @@ static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc, boolif (!nr_to_scan)return 0 ; -if (!need_aging) +if (!need_aging) { + sc->memcgs_need_aging = false;return nr_to_scan; + }/* leave the work to lru_gen_age_node() */ if (current_is_kswapd ())
@@ -4816,6 +4845,8 @@ static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc {struct blk_plug plug;long scanned =0 ; +bool swapped = false; +unsigned long reclaimed = sc->nr_reclaimed;struct pglist_data *pgdat =lruvec_pgdat (lruvec);lru_add_drain ();
@@ -4841,13 +4872,19 @@ static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *scif (!nr_to_scan)break ; - delta =evict_folios (lruvec, sc, swappiness); + delta =evict_folios (lruvec, sc, swappiness, &swapped);if (!delta)break ; +if (sc->memcgs_avoid_swapping && swappiness <200 && swapped) +break ; + scanned += delta; -if (scanned >= nr_to_scan) +if (scanned >= nr_to_scan) { +if (!swapped && sc->nr_reclaimed - reclaimed >=MIN_LRU_BATCH ) + sc->memcgs_need_swapping = false;break ; + }cond_resched (); }
Signature · 1 line
2.36.0.550.gb090851708-goog
PATCH v11 10/14 mm: multi-gen LRU: kill switch
In reply to Yu ZhaoAdd /sys/kernel/mm/lru_gen/enabled as a kill switch. Components that can be disabled include:
0x0001: the multi-gen LRU core
0x0002: walking page table, when arch_has_hw_pte_young() returns
true
0x0004: clearing the accessed bit in non-leaf PMD entries, when
CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG=y
[yYnN]: apply to all the components above
E.g.,
echo y >/sys/kernel/mm/lru_gen/enabled
cat /sys/kernel/mm/lru_gen/enabled
0x0007
echo 5 >/sys/kernel/mm/lru_gen/enabled
cat /sys/kernel/mm/lru_gen/enabled
0x0005
NB: the page table walks happen on the scale of seconds under heavy memory pressure, in which case the mmap_lock contention is a lesser concern, compared with the LRU lock contention and the I/O congestion. So far the only well-known case of the mmap_lock contention happens on Android, due to Scudo [1] which allocates several thousand VMAs for merely a few hundred MBs. The SPF and the Maple Tree also have provided their own assessments [2][3]. However, if walking page tables does worsen the mmap_lock contention, the kill switch can be used to disable it. In this case the multi-gen LRU will suffer a minor performance degradation, as shown previously.
Clearing the accessed bit in non-leaf PMD entries can also be disabled, since this behavior was not tested on x86 varieties other than Intel and AMD.
[1] https://source.android.com/devices/tech/debug/scudo
[2] https://lore.kernel.org/r/[email protected]/
[3] https://lore.kernel.org/r/[email protected]/
--- include/linux/cgroup.h | 15 +- include/linux/mm_inline.h | 12 +- include/linux/mmzone.h | 9 ++ kernel/cgroup/cgroup-internal.h | 1 - mm/Kconfig | 6 + mm/vmscan.c | 238 +++++++++++++++++++++++++++++++- 6 files changed, 272 insertions(+), 9 deletions(-)
include/linux/cgroup.h +14 -1
@@ -432,6 +432,18 @@ static inline void cgroup_put(struct cgroup *cgrp)css_put (&cgrp->self); } +extern struct mutex cgroup_mutex; + +static inline void cgroup_lock (void ) +{ +mutex_lock (&cgroup_mutex); +} + +static inline void cgroup_unlock (void ) +{ +mutex_unlock (&cgroup_mutex); +} +/** * task_css_set_check - obtain a task's css_set with extra access conditions * @task: the task to obtain css_set for
@@ -446,7 +458,6 @@ static inline void cgroup_put(struct cgroup *cgrp)* as locks used during the cgroup_subsys::attach() methods. */ #ifdef CONFIG_PROVE_RCU -extern struct mutex cgroup_mutex;extern spinlock_t css_set_lock;#define task_css_set_check (task, __c) \ rcu_dereference_check((task)->cgroups, \
@@ -708,6 +719,8 @@ struct cgroup;static inline u64 cgroup_id (const struct cgroup *cgrp) {return 1 ; }static inline void css_get (struct cgroup_subsys_state *css) {}static inline void css_put (struct cgroup_subsys_state *css) {} +static inline void cgroup_lock (void ) {} +static inline void cgroup_unlock (void ) {}static inline int cgroup_attach_task_all (struct task_struct *from,struct task_struct *t) {return 0 ; } static inline int cgroupstats_build(struct cgroupstats *stats,
include/linux/mm_inline.h +10 -2
@@ -106,7 +106,15 @@ static __always_inline enum lru_list folio_lru_list(struct folio *folio)static inline bool lru_gen_enabled (void ) { -return true; +#ifdef CONFIG_LRU_GEN_ENABLED +DECLARE_STATIC_KEY_TRUE (lru_gen_caps[NR_LRU_GEN_CAPS ]); + +return static_branch_likely (&lru_gen_caps[LRU_GEN_CORE ]); +#else +DECLARE_STATIC_KEY_FALSE (lru_gen_caps[NR_LRU_GEN_CAPS ]); + +return static_branch_unlikely (&lru_gen_caps[LRU_GEN_CORE ]); +#endif }static inline bool lru_gen_in_fault (void )
@@ -219,7 +227,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,VM_WARN_ON_ONCE_FOLIO (gen !=-1 , folio); -if (folio_test_unevictable (folio)) +if (folio_test_unevictable (folio) || !lrugen->enabled)return false;/* * There are three common cases for this page:
include/linux/mmzone.h +9 -0
@@ -386,6 +386,13 @@ enum { LRU_GEN_FILE, }; +enum { +LRU_GEN_CORE , +LRU_GEN_MM_WALK , +LRU_GEN_NONLEAF_YOUNG , +NR_LRU_GEN_CAPS +}; +#define MIN_LRU_BATCH BITS_PER_LONG#define MAX_LRU_BATCH (MIN_LRU_BATCH * 128)
@@ -427,6 +434,8 @@ struct lru_gen_struct {/* can be modified without holding the LRU lock */ atomic_long_t evicted[NR_HIST_GENS ][ANON_AND_FILE ][MAX_NR_TIERS ];atomic_long_t refaulted[NR_HIST_GENS ][ANON_AND_FILE ][MAX_NR_TIERS ]; +/* whether the multi-gen LRU is enabled */ +bool enabled; };enum {
kernel/cgroup/cgroup-internal.h +0 -1
@@ -165,7 +165,6 @@ struct cgroup_mgctx {#define DEFINE_CGROUP_MGCTX (name) \ struct cgroup_mgctx name = CGROUP_MGCTX_INIT(name) -extern struct mutex cgroup_mutex;extern spinlock_t css_set_lock;extern struct cgroup_subsys *cgroup_subsys[];extern struct list_head cgroup_roots;
mm/Kconfig +6 -0
@@ -918,6 +918,12 @@ config LRU_GEN help A high performance LRU implementation to overcommit memory. +config LRU_GEN_ENABLED + bool "Enable by default" + depends on LRU_GEN + help + This option enables the multi-gen LRU by default. + config LRU_GEN_STATS bool "Full stats for debugging" depends on LRU_GEN
mm/vmscan.c +233 -5
293 lines not on this page · raw message
Signature · 1 line
2.36.0.550.gb090851708-goog
PATCH v11 11/14 mm: multi-gen LRU: thrashing prevention
In reply to Yu ZhaoAdd /sys/kernel/mm/lru_gen/min_ttl_ms for thrashing prevention, as requested by many desktop users [1].
When set to value N, it prevents the working set of N milliseconds from getting evicted. The OOM killer is triggered if this working set cannot be kept in memory. Based on the average human detectable lag (~100ms), N=1000 usually eliminates intolerable lags due to thrashing. Larger values like N=3000 make lags less noticeable at the risk of premature OOM kills.
Compared with the size-based approach, e.g., [2], this time-based approach has the following advantages:
1. It is easier to configure because it is agnostic to applications and memory sizes.
2. It is more reliable because it is directly wired to the OOM killer.
[1] https://lore.kernel.org/r/Ydza%[email protected]/
[2] https://lore.kernel.org/r/[email protected]/
--- include/linux/mmzone.h | 2 ++ mm/vmscan.c | 68 +++++++++++++++++++++++++++++++++++++++--- 2 files changed, 66 insertions(+), 4 deletions(-)
include/linux/mmzone.h +2 -0
@@ -421,6 +421,8 @@ struct lru_gen_struct {unsigned long max_seq;/* the eviction increments the oldest generation numbers */ unsigned long min_seq[ANON_AND_FILE ]; +/* the birth time of each generation in jiffies */ +unsigned long timestamps[MAX_NR_GENS ];/* the multi-gen LRU lists */ struct list_head lists[MAX_NR_GENS ][ANON_AND_FILE ][MAX_NR_ZONES ];/* the sizes of the above lists */
mm/vmscan.c +64 -4
138 lines not on this page · raw message
Signature · 1 line
2.36.0.550.gb090851708-goog
PATCH v11 12/14 mm: multi-gen LRU: debugfs interface
In reply to Yu ZhaoAdd /sys/kernel/debug/lru_gen for working set estimation and proactive reclaim. These techniques are commonly used to optimize job scheduling (bin packing) in data centers [1][2].
Compared with the page table-based approach and the PFN-based approach, this lruvec-based approach has the following advantages:
1. It offers better choices because it is aware of memcgs, NUMA nodes, shared mappings and unmapped page cache.
2. It is more scalable because it is O(nr_hot_pages), whereas the PFN-based approach is O(nr_total_pages).
Add /sys/kernel/debug/lru_gen_full for debugging.
[1] https://dl.acm.org/doi/10.1145/3297858.3304053
[2] https://dl.acm.org/doi/10.1145/3503222.3507731
--- include/linux/nodemask.h | 1 + mm/vmscan.c | 403 ++++++++++++++++++++++++++++++++++++++- 2 files changed, 394 insertions(+), 10 deletions(-)
include/linux/nodemask.h +1 -0
@@ -486,6 +486,7 @@ static inline int num_node_state(enum node_states state)#define first_online_node 0#define first_memory_node 0#define next_online_node (nid) (MAX_NUMNODES) +#define next_memory_node (nid) (MAX_NUMNODES)#define nr_node_ids 1U#define nr_online_nodes 1U
mm/vmscan.c +393 -10
481 lines not on this page · raw message
Signature · 1 line
2.36.0.550.gb090851708-goog
PATCH v11 13/14 mm: multi-gen LRU: admin guide
In reply to Yu ZhaoAdd an admin guide.
--- Documentation/admin-guide/mm/index.rst | 1 + Documentation/admin-guide/mm/multigen_lru.rst | 156 ++++++++++++++++++ mm/Kconfig | 3 +- 3 files changed, 159 insertions(+), 1 deletion(-) create mode 100644 Documentation/admin-guide/mm/multigen_lru.rst
Documentation/admin-guide/mm/index.rst +1 -0
@@ -32,6 +32,7 @@ the Linux memory management. idle_page_tracking ksm memory-hotplug + multigen_lru nommu-mmap numa_memory_policy numaperf
Documentation/admin-guide/mm/multigen_lru.rst new file +156 -0
156 lines not on this page · raw message
mm/Kconfig +2 -1
@@ -916,7 +916,8 @@ config LRU_GEN # make sure folio->flags has enough spare bits depends on 64BIT || !SPARSEMEM || SPARSEMEM_VMEMMAP help - A high performance LRU implementation to overcommit memory. + A high performance LRU implementation to overcommit memory. See + Documentation/admin-guide/mm/multigen_lru.rst for details. config LRU_GEN_ENABLED bool "Enable by default"
Signature · 1 line
2.36.0.550.gb090851708-goog
PATCH v11 14/14 mm: multi-gen LRU: design doc
In reply to Yu ZhaoAdd a design doc.
--- Documentation/vm/index.rst | 1 + Documentation/vm/multigen_lru.rst | 159 ++++++++++++++++++++++++++++++ 2 files changed, 160 insertions(+) create mode 100644 Documentation/vm/multigen_lru.rst
Documentation/vm/index.rst +1 -0
@@ -25,6 +25,7 @@ algorithms. If you are looking for advice on simply allocating memory, see the ksm memory-model mmu_notifier + multigen_lru numa overcommit-accounting page_migration
Documentation/vm/multigen_lru.rst new file +159 -0
159 lines not on this page · raw message
Signature · 1 line
2.36.0.550.gb090851708-goog
Quoted 6 lines · Yu Zhao
TLDR
====
The current page reclaim is too expensive in terms of CPU usage and it often makes poor choices about what to evict. This patchset offers an alternative solution that is performant, versatile and straightforward.
Where's the changelog since v10?
Signature · 1 line
Jens Axboe
Quoted 10 lines · Jens Axboe
Quoted 6 lines · Yu Zhao
TLDR
====
The current page reclaim is too expensive in terms of CPU usage and it often makes poor choices about what to evict. This patchset offers an alternative solution that is performant, versatile and straightforward.Where's the changelog since v10?
Apologies for my laziness.
The changes are mainly nits, e.g., small refactorings, additional comments, etc.; relatively major ones are:
* VM_BUG_ON() -> VM_WARN_ON_ONCE()
* Removed depends on !MAXSMP
There are no bug fixes or structural changes in v11. I do have a bug fix [1] queued for the coming v12, which I have been benchmarking on top of 5.19-rc1. So far I see no improvements or regressions, compared with v10 on top of 5.18-rc1.
[1] https://forum.armbian.com/topic/20018-mglru-patches-to-bring-down-kswapd-cpu-usage