Multigenerational LRU Framework
Subsystems (21)
v7 cover Multigenerational LRU Framework
What's new
==========
1) Addressed all the comments received on the mailing list and in the meeting with the stakeholders (will note on individual patches).
2) Measured the performance improvements for each patch between 5-8 (reported in the commit messages).
TLDR
====
The current page reclaim is too expensive in terms of CPU usage and it often makes poor choices about what to evict. This patchset offers an alternative solution that is performant, versatile and straightforward.
Patchset overview
=================
The design and implementation overview was moved to patch 12 so that people can finish reading this cover letter.
1. mm: x86, arm64: add arch_has_hw_pte_young()
2. mm: x86: add CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG
Using hardware optimizations when trying to clear the accessed bit in many PTEs.
3. mm/vmscan.c: refactor shrink_node()
A minor refactor.
4. mm: multigenerational LRU: groundwork
Adding the basic data structure and the functions that insert/remove pages to/from the multigenerational LRU (MGLRU) lists.
5. mm: multigenerational LRU: minimal implementation
A minimal (functional) implementation without any optimizations.
6. mm: multigenerational LRU: exploit locality in rmap Improving the efficiency when using the rmap.
7. mm: multigenerational LRU: support page table walks Adding the (optional) page table scanning.
8. mm: multigenerational LRU: optimize multiple memcgs Optimizing the overall performance for multiple memcgs running mixed types of workloads.
9. mm: multigenerational LRU: runtime switch Adding a runtime switch to enable or disable MGLRU.
10. mm: multigenerational LRU: thrashing prevention
11. mm: multigenerational LRU: debugfs interface
Providing userspace with additional features like thrashing prevention, working set estimation and proactive reclaim.
12. mm: multigenerational LRU: documentation
Adding a design doc and an admin guide.
Benchmark results
=================
Independent lab results
-----------------------
Based on the popularity of searches [01] and the memory usage in Google's public cloud, the most popular open-source memory-hungry applications, in alphabetical order, are:
Apache Cassandra Memcached
Apache Hadoop MongoDB
Apache Spark PostgreSQL
MariaDB (MySQL) RedisAn independent lab evaluated MGLRU with the most widely used benchmark suites for the above applications. They posted 960 data points along with kernel metrics and perf profiles collected over more than 500 hours of total benchmark time. Their final reports show that, with 95% confidence intervals (CIs), the above applications all performed significantly better for at least part of their benchmark matrices.
On 5.14:
1. Apache Spark [02] took 95% CIs [9.28, 11.19]% and [12.20, 14.93]% less wall time to sort three billion random integers, respectively, under the medium- and the high-concurrency conditions, when overcommitting memory. There were no statistically significant changes in wall time for the rest of the benchmark matrix.
2. MariaDB [03] achieved 95% CIs [5.24, 10.71]% and [20.22, 25.97]% more transactions per minute (TPM), respectively, under the medium- and the high-concurrency conditions, when overcommitting memory. There were no statistically significant changes in TPM for the rest of the benchmark matrix.
3. Memcached [04] achieved 95% CIs [23.54, 32.25]%, [20.76, 41.61]% and [21.59, 30.02]% more operations per second (OPS), respectively, for sequential access, random access and Gaussian (distribution) access, when THP=always; 95% CIs [13.85, 15.97]% and [23.94, 29.92]% more OPS, respectively, for random access and Gaussian access, when THP=never. There were no statistically significant changes in OPS for the rest of the benchmark matrix.
4. MongoDB [05] achieved 95% CIs [2.23, 3.44]%, [6.97, 9.73]% and [2.16, 3.55]% more operations per second (OPS), respectively, for exponential (distribution) access, random access and Zipfian (distribution) access, when underutilizing memory; 95% CIs [8.83, 10.03]%, [21.12, 23.14]% and [5.53, 6.46]% more OPS, respectively, for exponential access, random access and Zipfian access, when overcommitting memory.
On 5.15:
5. Apache Cassandra [06] achieved 95% CIs [1.06, 4.10]%, [1.94, 5.43]% and [4.11, 7.50]% more operations per second (OPS), respectively, for exponential (distribution) access, random access and Zipfian (distribution) access, when swap was off; 95% CIs [0.50, 2.60]%, [6.51, 8.77]% and [3.29, 6.75]% more OPS, respectively, for exponential access, random access and Zipfian access, when swap was on.
6. Apache Hadoop [07] took 95% CIs [5.31, 9.69]% and [2.02, 7.86]% less average wall time to finish twelve parallel TeraSort jobs, respectively, under the medium- and the high-concurrency conditions, when swap was on. There were no statistically significant changes in average wall time for the rest of the benchmark matrix.
7. PostgreSQL [08] achieved 95% CI [1.75, 6.42]% more transactions per minute (TPM) under the high-concurrency condition, when swap was off; 95% CIs [12.82, 18.69]% and [22.70, 46.86]% more TPM, respectively, under the medium- and the high-concurrency conditions, when swap was on. There were no statistically significant changes in TPM for the rest of the benchmark matrix.
8. Redis [09] achieved 95% CIs [0.58, 5.94]%, [6.55, 14.58]% and [11.47, 19.36]% more total operations per second (OPS), respectively, for sequential access, random access and Gaussian (distribution) access, when THP=always; 95% CIs [1.27, 3.54]%, [10.11, 14.81]% and [8.75, 13.64]% more total OPS, respectively, for sequential access, random access and Gaussian access, when THP=never.
Our lab results
---------------
To supplement the above results, we ran the following benchmark suites on 5.16-rc7 and found no regressions [10]. (These synthetic benchmarks are popular among MM developers, but we prefer large-scale A/B experiments to validate improvements.)
fs_fio_bench_hdd_mq pft
fs_lmbench pgsql-hammerdb
fs_parallelio redis
fs_postmark stream
hackbench sysbenchthread
kernbench tpcc_spark
memcached unixbench
multichase vm-scalability
mutilate will-it-scalenginx
[01] https://trends.google.com
[02] https://lore.kernel.org/lkml/[email protected]/
[03] https://lore.kernel.org/lkml/[email protected]/
[04] https://lore.kernel.org/lkml/[email protected]/
[05] https://lore.kernel.org/lkml/[email protected]/
[06] https://lore.kernel.org/lkml/[email protected]/
[07] https://lore.kernel.org/lkml/[email protected]/
[08] https://lore.kernel.org/lkml/[email protected]/
[09] https://lore.kernel.org/lkml/[email protected]/
[10] https://lore.kernel.org/lkml/[email protected]/
Read-world applications
=======================
Third-party testimonials
------------------------
Konstantin wrote [11]:
I have Archlinux with 8G RAM + zswap + swap. While developing, I have lots of apps opened such as multiple LSP-servers for different langs, chats, two browsers, etc... Usually, my system gets quickly to a point of SWAP-storms, where I have to kill LSP-servers, restart browsers to free memory, etc, otherwise the system lags heavily and is barely usable.
1.5 day ago I migrated from 5.11.15 kernel to 5.12 + the LRU patchset, and I started up by opening lots of apps to create memory pressure, and worked for a day like this. Till now I had *not a single SWAP-storm*, and mind you I got 3.4G in SWAP. I was never getting to the point of 3G in SWAP before without a single SWAP-storm.
An anonymous user wrote [12]:
Using that v5 for some time and confirm that difference under heavy load and memory pressure is significant.
Shuang wrote [13]:
With the MGLRU, fio achieved 95% CIs [38.95, 40.26]%, [4.12, 6.64]% and [9.26, 10.36]% higher throughput, respectively, for random access, Zipfian (distribution) access and Gaussian (distribution) access, when the average number of jobs per CPU is 1; 95% CIs [42.32, 49.15]%, [9.44, 9.89]% and [20.99, 22.86]% higher throughput, respectively, for random access, Zipfian access and Gaussian access, when the average number of jobs per CPU is 2.
Daniel wrote [14]:
With memcached allocating ~100GB of byte-addressable Optante, performance improvement in terms of throughput (measured as queries per second) was about 10% for a series of workloads.
Large-scale deployments
-----------------------
The downstream kernels that have been using MGLRU include:
1. Android ARCVM [15]
2. Arch Linux Zen [16]
3. Chrome OS [17]
4. Liquorix [18]
5. post-factum [19]
6. XanMod [20]
We've rolled out MGLRU to tens of millions of Chrome OS users and about a million Android users. Google's fleetwide profiling [21] shows an overall 40% decrease in kswapd CPU usage, in addition to improvements in other UX metrics, e.g., an 85% decrease in the number of low-memory kills at the 75th percentile and an 18% decrease in rendering latency at the 50th percentile.
[11] https://lore.kernel.org/lkml/[email protected]/
[12] https://phoronix.com/forums/forum/software/general-linux-open-source/1301258-mglru-is-a-very-enticing-enhancement-for-linux-in-2022?p=1301275#post1301275
[13] https://lore.kernel.org/lkml/[email protected]/
[14] https://lore.kernel.org/linux-mm/CA+4-3vksGvKd18FgRinxhqHetBS1hQekJE2gwco8Ja-bJWKtFw@mail.gmail.com/
[15] https://chromium.googlesource.com/chromiumos/third_party/kernel
[16] https://archlinux.org
[17] https://chromium.org
[18] https://liquorix.net
[19] https://gitlab.com/post-factum/pf-kernel
[20] https://xanmod.org
[21] https://research.google/pubs/pub44271/
Summery
=======
The facts are:
1. The independent lab results and the real-world applications indicate substantial improvements; there are no known regressions.
2. Thrashing prevention, working set estimation and proactive reclaim work out of the box; there are no equivalent solutions.
3. There is a lot of new code; nobody has demonstrated smaller changes with similar effects.
Our options, accordingly, are:
1. Given the amount of evidence, the reported improvements will likely materialize for a wide range of workloads.
2. Gauging the interest from the past discussions [22][23][24], the new features will likely be put to use for both personal computers and data centers.
3. Based on Google's track record, the new code will likely be well maintained in the long term. It'd be more difficult if not impossible to achieve similar effects on top of the existing design.
[22] https://lore.kernel.org/lkml/[email protected]/
[23] https://lore.kernel.org/lkml/[email protected]/
[24] https://lore.kernel.org/lkml/[email protected]/
Yu Zhao (12):
mm: x86, arm64: add arch_has_hw_pte_young()
mm: x86: add CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG mm/vmscan.c: refactor shrink_node()
mm: multigenerational LRU: groundwork
mm: multigenerational LRU: minimal implementation
mm: multigenerational LRU: exploit locality in rmap
mm: multigenerational LRU: support page table walks
mm: multigenerational LRU: optimize multiple memcgs
mm: multigenerational LRU: runtime switch
mm: multigenerational LRU: thrashing prevention
mm: multigenerational LRU: debugfs interface
mm: multigenerational LRU: documentation
Documentation/admin-guide/mm/index.rst | 1 + Documentation/admin-guide/mm/multigen_lru.rst | 121 + Documentation/vm/index.rst | 1 + Documentation/vm/multigen_lru.rst | 152 + arch/Kconfig | 9 + arch/arm64/include/asm/pgtable.h | 14 +- arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 9 +- arch/x86/mm/pgtable.c | 5 +- fs/exec.c | 2 + fs/fuse/dev.c | 3 +- include/linux/cgroup.h | 15 +- include/linux/memcontrol.h | 36 + include/linux/mm.h | 8 + include/linux/mm_inline.h | 214 ++ include/linux/mm_types.h | 78 + include/linux/mmzone.h | 182 ++ include/linux/nodemask.h | 1 + include/linux/page-flags-layout.h | 19 +- include/linux/page-flags.h | 4 +- include/linux/pgtable.h | 17 +- include/linux/sched.h | 4 + include/linux/swap.h | 5 + kernel/bounds.c | 3 + kernel/cgroup/cgroup-internal.h | 1 - kernel/exit.c | 1 + kernel/fork.c | 9 + kernel/sched/core.c | 1 + mm/Kconfig | 50 + mm/huge_memory.c | 3 +- mm/memcontrol.c | 27 + mm/memory.c | 39 +- mm/mm_init.c | 6 +- mm/page_alloc.c | 1 + mm/rmap.c | 7 + mm/swap.c | 55 +- mm/vmscan.c | 2831 ++++++++++++++++- mm/workingset.c | 119 +- 38 files changed, 3908 insertions(+), 146 deletions(-) create mode 100644 Documentation/admin-guide/mm/multigen_lru.rst create mode 100644 Documentation/vm/multigen_lru.rst
Signature · 1 line
2.35.0.263.gb82422642f-goog
PATCH v7 1/12 mm: x86, arm64: add arch_has_hw_pte_young()
Some architectures automatically set the accessed bit in PTEs, e.g., x86 and arm64 v8.2. On architectures that don't have this capability, clearing the accessed bit in a PTE usually triggers a page fault following the TLB miss of this PTE (to emulate the accessed bit).
Being aware of this capability can help make better decisions, e.g., whether to spread the work out over a period of time to reduce bursty page faults when trying to clear the accessed bit in many PTEs.
Note that theoretically this capability can be unreliable, e.g., hotplugged CPUs might be different from builtin ones. Therefore it shouldn't be used in architecture-independent code that involves correctness, e.g., to determine whether TLB flushes are required (in combination with the accessed bit).
--- arch/arm64/include/asm/pgtable.h | 14 ++------------ arch/x86/include/asm/pgtable.h | 6 +++--- include/linux/pgtable.h | 13 +++++++++++++ mm/memory.c | 14 +------------- 4 files changed, 19 insertions(+), 28 deletions(-)
arch/arm64/include/asm/pgtable.h +2 -12
@@ -999,23 +999,13 @@ static inline void update_mmu_cache(struct vm_area_struct *vma,* page after fork() + CoW for pfn mappings. We don't always have a * hardware-managed access flag on arm64. */ -static inline bool arch_faults_on_old_pte (void ) -{ -WARN_ON (preemptible ()); - -return !cpu_has_hw_af (); -} -#define arch_faults_on_old_pte arch_faults_on_old_pte +#define arch_has_hw_pte_young cpu_has_hw_af/* * Experimentally, it's cheap to set the access flag in hardware and we * benefit from prefaulting mappings as 'old' to start with. */ -static inline bool arch_wants_old_prefaulted_pte (void ) -{ -return !arch_faults_on_old_pte (); -} -#define arch_wants_old_prefaulted_pte arch_wants_old_prefaulted_pte +#define arch_wants_old_prefaulted_pte cpu_has_hw_afstatic inline pgprot_t arch_filter_pgprot (pgprot_t prot) {
arch/x86/include/asm/pgtable.h +3 -3
@@ -1423,10 +1423,10 @@ static inline bool arch_has_pfn_modify_check(void) return boot_cpu_has_bug(X86_BUG_L1TF); } -#define arch_faults_on_old_pte arch_faults_on_old_pte -static inline bool arch_faults_on_old_pte (void ) +#define arch_has_hw_pte_young arch_has_hw_pte_young +static inline bool arch_has_hw_pte_young (void ) { -return false; +return true; }#endif /* __ASSEMBLY__ */
include/linux/pgtable.h +13 -0
@@ -259,6 +259,19 @@ static inline int pmdp_clear_flush_young(struct vm_area_struct *vma,#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif +#ifndef arch_has_hw_pte_young +/* +* Return whether the accessed bit is supported by the local CPU. +* +* This stub assumes accessing through an old PTE triggers a page fault. +* Architectures that automatically set the access bit should overwrite it. +*/ +static inline bool arch_has_hw_pte_young (void ) +{ +return false; +} +#endif +#ifndef __HAVE_ARCH_PTEP_CLEARstatic inline void ptep_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep)
mm/memory.c +1 -13
@@ -122,18 +122,6 @@ int randomize_va_space __read_mostly =2 ;#endif -#ifndef arch_faults_on_old_pte -static inline bool arch_faults_on_old_pte (void ) -{ -/* -* Those arches which don't have hw access flag feature need to -* implement their own helper. By default, "true" means pagefault -* will be hit on old pte. -*/ -return true; -} -#endif -#ifndef arch_wants_old_prefaulted_ptestatic inline bool arch_wants_old_prefaulted_pte (void ) {
@@ -2778,7 +2766,7 @@ static inline bool cow_user_page(struct page *dst, struct page *src,* On architectures with software "accessed" bits, we would * take a double page fault, so mark it accessed here. */ -if (arch_faults_on_old_pte () && !pte_young (vmf->orig_pte)) { +if (!arch_has_hw_pte_young () && !pte_young (vmf->orig_pte)) {pte_t entry; vmf->pte =pte_offset_map_lock (mm, vmf->pmd, addr, &vmf->ptl);
Signature · 1 line
2.35.0.263.gb82422642f-goog
Re: [PATCH v7 01/12] mm: x86, arm64: add arch_has_hw_pte_young()
On Tue, Feb 08, 2022 at 01:18:51AM -0700, Yu Zhao wrote:
<snipped>
Quoted 6 lines
arch/arm64/include/asm/pgtable.h +0
-0@@ -999,23 +999,13 @@ static inline void update_mmu_cache(struct vm_area_struct *vma,* page after fork() + CoW for pfn mappings. We don't always have a
arch/arm64/include/asm/pgtable.h static inline void update_mmu_cache(struct vm_area_struct *vma, · line 1002
Quoted 10 lines
arch/arm64/include/asm/pgtable.h +1
-7* hardware-managed access flag on arm64. */ -static inline bool arch_faults_on_old_pte (void ) -{ -WARN_ON (preemptible ()); - -return !cpu_has_hw_af (); -} -#define arch_faults_on_old_pte arch_faults_on_old_pte +#define arch_has_hw_pte_young cpu_has_hw_af
Reworked arch_has_hw_pte_young() for arm64 according to:
https://lore.kernel.org/linux-mm/20220111141901.GA10338@willie-the-truck/
<snipped>
Re: [PATCH v7 01/12] mm: x86, arm64: add arch_has_hw_pte_young()
In reply to Yu ZhaoQuoted 27 lines · Yu Zhao
Some architectures automatically set the accessed bit in PTEs, e.g., x86 and arm64 v8.2. On architectures that don't have this capability, clearing the accessed bit in a PTE usually triggers a page fault following the TLB miss of this PTE (to emulate the accessed bit).
Being aware of this capability can help make better decisions, e.g., whether to spread the work out over a period of time to reduce bursty page faults when trying to clear the accessed bit in many PTEs.
Note that theoretically this capability can be unreliable, e.g., hotplugged CPUs might be different from builtin ones. Therefore it shouldn't be used in architecture-independent code that involves correctness, e.g., to determine whether TLB flushes are required (in combination with the accessed bit).
Signed-off-by Yu Zhao <[email protected]>Acked-by Brian Geffon <[email protected]>Acked-by Steven Barrett <[email protected]>Acked-by Suleiman Souhlal <[email protected]>Tested-by Daniel Byrne <[email protected]>Tested-by Donald Carr <[email protected]>Tested-by Holger Hoffstätte <[email protected]>Tested-by Konstantin Kharlamov <[email protected]>Tested-by Shuang Zhai <[email protected]>Tested-by Sofia Trinh <[email protected]>
Quoted 6 lines
--- arch/arm64/include/asm/pgtable.h | 14 ++------------ arch/x86/include/asm/pgtable.h | 6 +++--- include/linux/pgtable.h | 13 +++++++++++++ mm/memory.c | 14 +------------- 4 files changed, 19 insertions(+), 28 deletions(-)
For the arm64 bit:
Will
PATCH v7 2/12 mm: x86: add CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG
In reply to Yu ZhaoSome architectures support the accessed bit in non-leaf PMD entries, e.g., x86 sets the accessed bit in a non-leaf PMD entry when using it as part of linear address translation [1]. Page table walkers that clear the accessed bit may use this capability to reduce their search space.
Note that:
1. Although an inline function is preferable, this capability is added as a configuration option for the consistency with the existing macros.
2. Due to the little interest in other varieties, this capability was only tested on Intel and AMD CPUs.
[1]: Intel 64 and IA-32 Architectures Software Developer's Manual
Volume 3 (June 2021), section 4.8
--- arch/Kconfig | 9 +++++++++ arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 3 ++- arch/x86/mm/pgtable.c | 5 ++++- include/linux/pgtable.h | 4 ++-- 5 files changed, 18 insertions(+), 4 deletions(-)
arch/Kconfig +9 -0
@@ -1322,6 +1322,15 @@ config DYNAMIC_SIGFRAME config HAVE_ARCH_NODE_DEV_GROUP bool +config ARCH_HAS_NONLEAF_PMD_YOUNG + bool + depends on PGTABLE_LEVELS > 2 + help + Architectures that select this option are capable of setting the + accessed bit in non-leaf PMD entries when using them as part of linear + address translations. Page table walkers that clear the accessed bit + may use this capability to reduce their search space. + source "kernel/gcov/Kconfig" source "scripts/gcc-plugins/Kconfig"
arch/x86/Kconfig +1 -0
@@ -85,6 +85,7 @@ config X86 select ARCH_HAS_PMEM_API if X86_64 select ARCH_HAS_PTE_DEVMAP if X86_64 select ARCH_HAS_PTE_SPECIAL + select ARCH_HAS_NONLEAF_PMD_YOUNG select ARCH_HAS_UACCESS_FLUSHCACHE if X86_64 select ARCH_HAS_COPY_MC if X86_64 select ARCH_HAS_SET_MEMORY
arch/x86/include/asm/pgtable.h +2 -1
@@ -819,7 +819,8 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)static inline int pmd_bad (pmd_t pmd) { -return (pmd_flags (pmd) & ~_PAGE_USER) != _KERNPG_TABLE; +return (pmd_flags (pmd) & ~(_PAGE_USER | _PAGE_ACCESSED)) != + (_KERNPG_TABLE & ~_PAGE_ACCESSED); }static inline unsigned long pages_to_mb (unsigned long npg)
arch/x86/mm/pgtable.c +4 -1
@@ -550,7 +550,7 @@ int ptep_test_and_clear_young(struct vm_area_struct *vma,return ret; } -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#if defined(CONFIG_TRANSPARENT_HUGEPAGE ) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG )int pmdp_test_and_clear_young (struct vm_area_struct *vma,unsigned long addr,pmd_t *pmdp) {
@@ -562,6 +562,9 @@ int pmdp_test_and_clear_young(struct vm_area_struct *vma,return ret; } +#endif + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE int pudp_test_and_clear_young (struct vm_area_struct *vma,unsigned long addr,pud_t *pudp) {
include/linux/pgtable.h +2 -2
@@ -212,7 +212,7 @@ static inline int ptep_test_and_clear_young(struct vm_area_struct *vma,#endif #ifndef __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#if defined(CONFIG_TRANSPARENT_HUGEPAGE ) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG )static inline int pmdp_test_and_clear_young (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp)
@@ -233,7 +233,7 @@ static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma,BUILD_BUG ();return 0 ; } -#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +#endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG */ #endif #ifndef __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH
Signature · 1 line
2.35.0.263.gb82422642f-goog
Re: [PATCH v7 02/12] mm: x86: add CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG
Quoted 6 lines · Yu Zhao
Some architectures support the accessed bit in non-leaf PMD entries, e.g., x86 sets the accessed bit in a non-leaf PMD entry when using it as part of linear address translation [1]. Page table walkers that clear the accessed bit may use this capability to reduce their search space.
Quoted 6 lines
Note that:
1. Although an inline function is preferable, this capability is added as a configuration option for the consistency with the existing macros.
2. Due to the little interest in other varieties, this capability was only tested on Intel and AMD CPUs.
Clarified ARCH_HAS_NONLEAF_PMD_YOUNG for x86 as requested here:
https://lore.kernel.org/linux-mm/CAHk-=wgvOqj6LUhNp8V5ddT8eZyYdFDzMZE73KgPggOnc28VWg@mail.gmail.com/
<snipped>
arch/x86/Kconfig config X86 · line 88
Quoted 9 lines
arch/x86/Kconfig +1
-0@@ -85,6 +85,7 @@ config X86 select ARCH_HAS_PMEM_API if X86_64 select ARCH_HAS_PTE_DEVMAP if X86_64 select ARCH_HAS_PTE_SPECIAL + select ARCH_HAS_NONLEAF_PMD_YOUNG
And enabled it for both 32-bit and 64-bit.
<snipped>
PATCH v7 3/12 mm/vmscan.c: refactor shrink_node()
In reply to Yu ZhaoThis patch refactors shrink_node() to improve readability for the upcoming changes to mm/vmscan.c.
--- mm/vmscan.c | 198 +++++++++++++++++++++++++++------------------------- 1 file changed, 104 insertions(+), 94 deletions(-)
mm/vmscan.c +104 -94
@@ -2716,6 +2716,109 @@ enum scan_balance { SCAN_FILE, }; +static void prepare_scan_count (pg_data_t *pgdat,struct scan_control *sc) +{ +unsigned long file; +struct lruvec *target_lruvec; + + target_lruvec =mem_cgroup_lruvec (sc->target_mem_cgroup, pgdat); + +/* +* Flush the memory cgroup stats, so that we read accurate per-memcg +* lruvec stats for heuristics. +*/ +mem_cgroup_flush_stats (); + +/* +* Determine the scan balance between anon and file LRUs. +*/ +spin_lock_irq (&target_lruvec->lru_lock); + sc->anon_cost = target_lruvec->anon_cost; + sc->file_cost = target_lruvec->file_cost; +spin_unlock_irq (&target_lruvec->lru_lock); + +/* +* Target desirable inactive:active list ratios for the anon +* and file LRU lists. +*/ +if (!sc->force_deactivate) { +unsigned long refaults; + + refaults =lruvec_page_state (target_lruvec, +WORKINGSET_ACTIVATE_ANON ); +if (refaults != target_lruvec->refaults[0 ] || +inactive_is_low (target_lruvec,LRU_INACTIVE_ANON )) + sc->may_deactivate |=DEACTIVATE_ANON ; +else + sc->may_deactivate &= ~DEACTIVATE_ANON ; + +/* +* When refaults are being observed, it means a new +* workingset is being established. Deactivate to get +* rid of any stale active pages quickly. +*/ + refaults =lruvec_page_state (target_lruvec, +WORKINGSET_ACTIVATE_FILE ); +if (refaults != target_lruvec->refaults[1 ] || +inactive_is_low (target_lruvec,LRU_INACTIVE_FILE )) + sc->may_deactivate |=DEACTIVATE_FILE ; +else + sc->may_deactivate &= ~DEACTIVATE_FILE ; + }else + sc->may_deactivate =DEACTIVATE_ANON |DEACTIVATE_FILE ; + +/* +* If we have plenty of inactive file pages that aren't +* thrashing, try to reclaim those first before touching +* anonymous pages. +*/ + file =lruvec_page_state (target_lruvec,NR_INACTIVE_FILE ); +if (file >> sc->priority && !(sc->may_deactivate &DEACTIVATE_FILE )) + sc->cache_trim_mode =1 ; +else + sc->cache_trim_mode =0 ; + +/* +* Prevent the reclaimer from falling into the cache trap: as +* cache pages start out inactive, every cache fault will tip +* the scan balance towards the file LRU. And as the file LRU +* shrinks, so does the window for rotation from references. +* This means we have a runaway feedback loop where a tiny +* thrashing file LRU becomes infinitely more attractive than +* anon pages. Try to detect this based on file LRU size. +*/ +if (!cgroup_reclaim (sc)) { +unsigned long total_high_wmark =0 ; +unsigned long free, anon; +int z; + + free =sum_zone_node_page_state (pgdat->node_id,NR_FREE_PAGES ); + file =node_page_state (pgdat,NR_ACTIVE_FILE ) + +node_page_state (pgdat,NR_INACTIVE_FILE ); + +for (z =0 ; z <MAX_NR_ZONES ; z++) { +struct zone *zone = &pgdat->node_zones[z]; + +if (!managed_zone (zone)) +continue ; + + total_high_wmark +=high_wmark_pages (zone); + } + +/* +* Consider anon: if that's low too, this isn't a +* runaway file reclaim problem, but rather just +* extreme pressure. Reclaim as per usual then. +*/ + anon =node_page_state (pgdat,NR_INACTIVE_ANON ); + + sc->file_is_tiny = + file + free <= total_high_wmark && + !(sc->may_deactivate &DEACTIVATE_ANON ) && + anon >> sc->priority; + } +} +/* * Determine how aggressively the anon and file LRU lists should be * scanned. The relative value of each set of LRU lists is determined
@@ -3186,109 +3289,16 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc)unsigned long nr_reclaimed, nr_scanned;struct lruvec *target_lruvec;bool reclaimable = false; -unsigned long file; target_lruvec =mem_cgroup_lruvec (sc->target_mem_cgroup, pgdat); again: -/* -* Flush the memory cgroup stats, so that we read accurate per-memcg -* lruvec stats for heuristics. -*/ -mem_cgroup_flush_stats (); -memset (&sc->nr,0 ,sizeof (sc->nr)); nr_reclaimed = sc->nr_reclaimed; nr_scanned = sc->nr_scanned; -/* -* Determine the scan balance between anon and file LRUs. -*/ -spin_lock_irq (&target_lruvec->lru_lock); - sc->anon_cost = target_lruvec->anon_cost; - sc->file_cost = target_lruvec->file_cost; -spin_unlock_irq (&target_lruvec->lru_lock); - -/* -* Target desirable inactive:active list ratios for the anon -* and file LRU lists. -*/ -if (!sc->force_deactivate) { -unsigned long refaults; - - refaults =lruvec_page_state (target_lruvec, -WORKINGSET_ACTIVATE_ANON ); -if (refaults != target_lruvec->refaults[0 ] || -inactive_is_low (target_lruvec,LRU_INACTIVE_ANON )) - sc->may_deactivate |=DEACTIVATE_ANON ; -else - sc->may_deactivate &= ~DEACTIVATE_ANON ; - -/* -* When refaults are being observed, it means a new -* workingset is being established. Deactivate to get -* rid of any stale active pages quickly. -*/ - refaults =lruvec_page_state (target_lruvec, -WORKINGSET_ACTIVATE_FILE ); -if (refaults != target_lruvec->refaults[1 ] || -inactive_is_low (target_lruvec,LRU_INACTIVE_FILE )) - sc->may_deactivate |=DEACTIVATE_FILE ; -else - sc->may_deactivate &= ~DEACTIVATE_FILE ; - }else - sc->may_deactivate =DEACTIVATE_ANON |DEACTIVATE_FILE ; - -/* -* If we have plenty of inactive file pages that aren't -* thrashing, try to reclaim those first before touching -* anonymous pages. -*/ - file =lruvec_page_state (target_lruvec,NR_INACTIVE_FILE ); -if (file >> sc->priority && !(sc->may_deactivate &DEACTIVATE_FILE )) - sc->cache_trim_mode =1 ; -else - sc->cache_trim_mode =0 ; - -/* -* Prevent the reclaimer from falling into the cache trap: as -* cache pages start out inactive, every cache fault will tip -* the scan balance towards the file LRU. And as the file LRU -* shrinks, so does the window for rotation from references. -* This means we have a runaway feedback loop where a tiny -* thrashing file LRU becomes infinitely more attractive than -* anon pages. Try to detect this based on file LRU size. -*/ -if (!cgroup_reclaim (sc)) { -unsigned long total_high_wmark =0 ; -unsigned long free, anon; -int z; - - free =sum_zone_node_page_state (pgdat->node_id,NR_FREE_PAGES ); - file =node_page_state (pgdat,NR_ACTIVE_FILE ) + -node_page_state (pgdat,NR_INACTIVE_FILE ); - -for (z =0 ; z <MAX_NR_ZONES ; z++) { -struct zone *zone = &pgdat->node_zones[z]; -if (!managed_zone (zone)) -continue ; - - total_high_wmark +=high_wmark_pages (zone); - } - -/* -* Consider anon: if that's low too, this isn't a -* runaway file reclaim problem, but rather just -* extreme pressure. Reclaim as per usual then. -*/ - anon =node_page_state (pgdat,NR_INACTIVE_ANON ); - - sc->file_is_tiny = - file + free <= total_high_wmark && - !(sc->may_deactivate &DEACTIVATE_ANON ) && - anon >> sc->priority; - } +prepare_scan_count (pgdat, sc);shrink_node_memcgs (pgdat, sc);
Signature · 1 line
2.35.0.263.gb82422642f-goog
PATCH v7 4/12 mm: multigenerational LRU: groundwork
In reply to Yu ZhaoEvictable pages are divided into multiple generations for each lruvec. The youngest generation number is stored in lrugen->max_seq for both anon and file types as they're aged on an equal footing. The oldest generation numbers are stored in lrugen->min_seq[] separately for anon and file types as clean file pages can be evicted regardless of swap constraints. These three variables are monotonically increasing.
Generation numbers are truncated into order_base_2(MAX_NR_GENS+1) bits in order to fit into the gen counter in folio->flags. Each truncated generation number is an index to lrugen->lists[]. The sliding window technique is used to track at least MIN_NR_GENS and at most MAX_NR_GENS generations. The gen counter stores (seq%MAX_NR_GENS)+1 while a page is on one of lrugen->lists[]. Otherwise it stores 0.
There are two conceptually independent processes (as in the manufacturing process): "the aging", which produces young generations, and "the eviction", which consumes old generations. They form a closed-loop system, i.e., "the page reclaim". Both processes can be invoked from userspace for the purposes of working set estimation and proactive reclaim. These features are required to optimize job scheduling (bin packing) in data centers. The variable size of the sliding window is designed for such use cases [1][2].
To avoid confusions, the terms "hot" and "cold" will be applied to the multigenerational LRU, as a new convention; the terms "active" and "inactive" will be applied to the active/inactive LRU, as usual.
The protection of hot pages and the selection of cold pages are based on page access channels and patterns. There are two access channels:
one through page tables and the other through file descriptors. The protection of the former channel is by design stronger because:
1) The uncertainty in determining the access patterns of the former channel is higher due to the approximation of the accessed bit.
2) The cost of evicting the former channel is higher due to the TLB flushes required and the likelihood of encountering the dirty bit.
3) The penalty of underprotecting the former channel is higher because applications usually don't prepare themselves for major page faults like they do for blocked I/O. E.g., GUI applications commonly use dedicated I/O threads to avoid blocking the rendering threads.
There are also two access patterns: one with temporal locality and the other without. For the reasons listed above, the former channel is assumed to follow the former pattern unless VM_SEQ_READ or VM_RAND_READ is present, and the latter channel is assumed to follow the latter pattern unless outlying refaults have been observed.
The next patch will address the "outlying refaults". A few macros, i.e., LRU_REFS_*, used there are added in this patch to make the patchset less diffy.
A page is added to the youngest generation on faulting. The aging needs to check the accessed bit at least twice before handing this page over to the eviction. The first check takes care of the accessed bit set on the initial fault; the second check makes sure this page hasn't been used since then. This process, AKA second chance, requires a minimum of two generations, hence MIN_NR_GENS.
[1] https://research.google/pubs/pub48551/
[2] https://www.cs.cmu.edu/~dskarlat/publications/tmo_asplos22.pdf
--- fs/fuse/dev.c | 3 +- include/linux/mm.h | 2 + include/linux/mm_inline.h | 191 ++++++++++++++++++++++++++++++ include/linux/mmzone.h | 76 ++++++++++++ include/linux/page-flags-layout.h | 19 ++- include/linux/page-flags.h | 4 +- include/linux/sched.h | 4 + kernel/bounds.c | 3 + mm/huge_memory.c | 3 +- mm/memcontrol.c | 2 + mm/memory.c | 25 ++++ mm/mm_init.c | 6 +- mm/page_alloc.c | 1 + mm/swap.c | 5 + mm/vmscan.c | 85 +++++++++++++ 15 files changed, 418 insertions(+), 11 deletions(-)
fs/fuse/dev.c +2 -1
@@ -785,7 +785,8 @@ static int fuse_check_page(struct page *page) 1 << PG_active | 1 << PG_workingset | 1 << PG_reclaim | - 1 << PG_waiters))) { + 1 << PG_waiters | + LRU_GEN_MASK | LRU_REFS_MASK))) {dump_page (page,"fuse: trying to steal weird page" );return 1 ; }
include/linux/mm.h +2 -0
@@ -1032,6 +1032,8 @@ vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf);#define ZONES_PGOFF (NODES_PGOFF - ZONES_WIDTH)#define LAST_CPUPID_PGOFF (ZONES_PGOFF - LAST_CPUPID_WIDTH)#define KASAN_TAG_PGOFF (LAST_CPUPID_PGOFF - KASAN_TAG_WIDTH) +#define LRU_GEN_PGOFF (KASAN_TAG_PGOFF - LRU_GEN_WIDTH) +#define LRU_REFS_PGOFF (LRU_GEN_PGOFF - LRU_REFS_WIDTH)/* * Define the bit shifts to access each section. For non-existent
include/linux/mm_inline.h +191 -0
214 lines not on this page · raw message
include/linux/mmzone.h +76 -0
88 lines not on this page · raw message
include/linux/page-flags-layout.h +14 -5
@@ -26,6 +26,14 @@#define ZONES_WIDTH ZONES_SHIFT +#ifdef CONFIG_LRU_GEN +/* LRU_GEN_WIDTH is generated from order_base_2(CONFIG_NR_LRU_GENS + 1). */ +#define LRU_REFS_WIDTH (CONFIG_TIERS_PER_GEN - 2) +#else +#define LRU_GEN_WIDTH 0 +#define LRU_REFS_WIDTH 0 +#endif /* CONFIG_LRU_GEN */ +#ifdef CONFIG_SPARSEMEM #include <asm/sparsemem.h> #define SECTIONS_SHIFT (MAX_PHYSMEM_BITS - SECTION_SIZE_BITS)
@@ -55,7 +63,8 @@#define SECTIONS_WIDTH 0#endif -#if ZONES_WIDTH +SECTIONS_WIDTH +NODES_SHIFT <=BITS_PER_LONG -NR_PAGEFLAGS +#if ZONES_WIDTH +LRU_GEN_WIDTH +LRU_REFS_WIDTH +SECTIONS_WIDTH +NODES_SHIFT \ + <=BITS_PER_LONG -NR_PAGEFLAGS #define NODES_WIDTH NODES_SHIFT#elif defined(CONFIG_SPARSEMEM_VMEMMAP )#error "Vmemmap: No space for nodes field in page flags"
@@ -89,8 +98,8 @@#define LAST_CPUPID_SHIFT 0#endif -#if ZONES_WIDTH +SECTIONS_WIDTH +NODES_WIDTH +KASAN_TAG_WIDTH +LAST_CPUPID_SHIFT \ - <=BITS_PER_LONG -NR_PAGEFLAGS +#if ZONES_WIDTH +LRU_GEN_WIDTH +LRU_REFS_WIDTH +SECTIONS_WIDTH +NODES_WIDTH + \ +KASAN_TAG_WIDTH +LAST_CPUPID_SHIFT <=BITS_PER_LONG -NR_PAGEFLAGS #define LAST_CPUPID_WIDTH LAST_CPUPID_SHIFT#else #define LAST_CPUPID_WIDTH 0
@@ -100,8 +109,8 @@#define LAST_CPUPID_NOT_IN_PAGE_FLAGS #endif -#if ZONES_WIDTH +SECTIONS_WIDTH +NODES_WIDTH +KASAN_TAG_WIDTH +LAST_CPUPID_WIDTH \ - >BITS_PER_LONG -NR_PAGEFLAGS +#if ZONES_WIDTH +LRU_GEN_WIDTH +LRU_REFS_WIDTH +SECTIONS_WIDTH +NODES_WIDTH + \ +KASAN_TAG_WIDTH +LAST_CPUPID_WIDTH >BITS_PER_LONG -NR_PAGEFLAGS #error "Not enough bits in page flags"#endif
include/linux/page-flags.h +2 -2
@@ -935,7 +935,7 @@ __PAGEFLAG(Isolated, isolated, PF_ANY);1UL << PG_private |1UL << PG_private_2 | \1UL << PG_writeback |1UL << PG_reserved | \1UL << PG_slab |1UL << PG_active | \ - 1UL << PG_unevictable | __PG_MLOCKED) + 1UL << PG_unevictable | __PG_MLOCKED | LRU_GEN_MASK)/* * Flags checked when a page is prepped for return by the page allocator.
@@ -946,7 +946,7 @@ __PAGEFLAG(Isolated, isolated, PF_ANY);* alloc-free cycle to prevent from reusing the page. */ #define PAGE_FLAGS_CHECK_AT_PREP \ - (PAGEFLAGS_MASK & ~__PG_HWPOISON) + ((PAGEFLAGS_MASK & ~__PG_HWPOISON) | LRU_GEN_MASK | LRU_REFS_MASK)#define PAGE_FLAGS_PRIVATE \ (1UL << PG_private | 1UL << PG_private_2)
include/linux/sched.h +4 -0
@@ -914,6 +914,10 @@ struct task_struct {#ifdef CONFIG_MEMCG unsigned in_user_fault:1;#endif +#ifdef CONFIG_LRU_GEN +/* whether the LRU algorithm may apply to this access */ + unsigned in_lru_fault:1; +#endif #ifdef CONFIG_COMPAT_BRK unsigned brk_randomized:1;#endif
kernel/bounds.c +3 -0
@@ -22,6 +22,9 @@ int main(void)DEFINE (NR_CPUS_BITS ,ilog2 (CONFIG_NR_CPUS ));#endif DEFINE (SPINLOCK_SIZE ,sizeof (spinlock_t)); +#ifdef CONFIG_LRU_GEN +DEFINE (LRU_GEN_WIDTH ,order_base_2 (CONFIG_NR_LRU_GENS +1 )); +#endif /* End of constants */ return 0 ;
mm/huge_memory.c +2 -1
@@ -2364,7 +2364,8 @@ static void __split_huge_page_tail(struct page *head, int tail,#ifdef CONFIG_64BIT (1L << PG_arch_2) | #endif - (1L << PG_dirty))); + (1L << PG_dirty) | + LRU_GEN_MASK | LRU_REFS_MASK));/* ->mapping in first tail page is compound_mapcount */ VM_BUG_ON_PAGE(tail > 2 && page_tail->mapping != TAIL_MAPPING,
mm/memcontrol.c +2 -0
@@ -5121,6 +5121,7 @@ static void __mem_cgroup_free(struct mem_cgroup *memcg)static void mem_cgroup_free (struct mem_cgroup *memcg) { +lru_gen_free_memcg (memcg);memcg_wb_domain_exit (memcg);__mem_cgroup_free (memcg); }
@@ -5180,6 +5181,7 @@ static struct mem_cgroup *mem_cgroup_alloc(void) memcg->deferred_split_queue.split_queue_len =0 ;#endif idr_replace (&mem_cgroup_idr, memcg, memcg->id.id); +lru_gen_init_memcg (memcg);return memcg; fail:mem_cgroup_id_remove (memcg);
mm/memory.c +25 -0
@@ -4754,6 +4754,27 @@ static inline void mm_account_fault(struct pt_regs *regs,perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MIN ,1 , regs, address); } +#ifdef CONFIG_LRU_GEN +static void lru_gen_enter_fault (struct vm_area_struct *vma) +{ +/* the LRU algorithm doesn't apply to sequential or random reads */ + current->in_lru_fault = !(vma->vm_flags & (VM_SEQ_READ |VM_RAND_READ )); +} + +static void lru_gen_exit_fault (void ) +{ + current->in_lru_fault = false; +} +#else +static void lru_gen_enter_fault (struct vm_area_struct *vma) +{ +} + +static void lru_gen_exit_fault (void ) +{ +} +#endif /* CONFIG_LRU_GEN */ +/* * By the time we get here, we already hold the mm semaphore *
@@ -4785,11 +4806,15 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,if (flags &FAULT_FLAG_USER )mem_cgroup_enter_user_fault (); +lru_gen_enter_fault (vma); +if (unlikely (is_vm_hugetlb_page (vma))) ret =hugetlb_fault (vma->vm_mm, vma, address, flags);else ret =__handle_mm_fault (vma, address, flags); +lru_gen_exit_fault (); +if (flags &FAULT_FLAG_USER ) {mem_cgroup_exit_user_fault ();/*
mm/mm_init.c +4 -2
@@ -65,14 +65,16 @@ void __init mminit_verify_pageflags_layout(void) shift =8 *sizeof (unsigned long ); width = shift -SECTIONS_WIDTH -NODES_WIDTH -ZONES_WIDTH - -LAST_CPUPID_SHIFT -KASAN_TAG_WIDTH ; + -LAST_CPUPID_SHIFT -KASAN_TAG_WIDTH -LRU_GEN_WIDTH -LRU_REFS_WIDTH ;mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_widths" , -"Section %d Node %d Zone %d Lastcpupid %d Kasantag %d Flags %d\n" , +"Section %d Node %d Zone %d Lastcpupid %d Kasantag %d Gen %d Tier %d Flags %d\n" ,SECTIONS_WIDTH ,NODES_WIDTH ,ZONES_WIDTH ,LAST_CPUPID_WIDTH ,KASAN_TAG_WIDTH , +LRU_GEN_WIDTH , +LRU_REFS_WIDTH ,NR_PAGEFLAGS ); mminit_dprintk(MMINIT_TRACE, "pageflags_layout_shifts", "Section %d Node %d Zone %d Lastcpupid %d Kasantag %d\n",
mm/page_alloc.c +1 -0
@@ -7480,6 +7480,7 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat)pgdat_page_ext_init (pgdat);lruvec_init (&pgdat->__lruvec); +lru_gen_init_state (NULL , &pgdat->__lruvec); } static void __meminit zone_init_internals(struct zone *zone, enum zone_type idx, int nid,
mm/swap.c +5 -0
@@ -462,6 +462,11 @@ void folio_add_lru(struct folio *folio)VM_BUG_ON_FOLIO (folio_test_active (folio) &&folio_test_unevictable (folio), folio);VM_BUG_ON_FOLIO (folio_test_lru (folio), folio); +/* see the comment in lru_gen_add_folio() */ +if (lru_gen_enabled () && !folio_test_unevictable (folio) && +lru_gen_in_fault () && !(current->flags &PF_MEMALLOC )) +folio_set_active (folio); +folio_get (folio);local_lock (&lru_pvecs.lock); pvec =this_cpu_ptr (&lru_pvecs.lru_add);
mm/vmscan.c +85 -0
91 lines not on this page · raw message
Signature · 1 line
2.35.0.263.gb82422642f-goog
Re: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
On Tue, Feb 08, 2022 at 01:18:54AM -0700, Yu Zhao wrote:
<snipped>
Quoted 18 lines
mm/memory.c +10
-0@@ -4754,6 +4754,27 @@ static inline void mm_account_fault(struct pt_regs *regs,perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MIN ,1 , regs, address); } +#ifdef CONFIG_LRU_GEN +static void lru_gen_enter_fault (struct vm_area_struct *vma) +{ +/* the LRU algorithm doesn't apply to sequential or random reads */ + current->in_lru_fault = !(vma->vm_flags & (VM_SEQ_READ |VM_RAND_READ )); +} + +static void lru_gen_exit_fault (void ) +{ + current->in_lru_fault = false;
mm/memory.c static inline void mm_account_fault(struct pt_regs *regs, · line 4776
Quoted 10 lines
mm/memory.c +10
-0+} +#else +static void lru_gen_enter_fault (struct vm_area_struct *vma) +{ +} + +static void lru_gen_exit_fault (void ) +{ +} +#endif /* CONFIG_LRU_GEN */
Moved task_enter_lru_fault() from mm.h to memory.c as requested here:
https://lore.kernel.org/linux-mm/CAHk-=wib5-tUrf2=zYL9hjCqqFykZmTr_-vMAvSo48boCA+-Wg@mail.gmail.com/
Also renamed it to lru_gen_enter_fault().
<snipped>
Johannes Weiner
maintainerRe: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
In reply to Yu ZhaoHello Yu,
Quoted 20 lines · Yu Zhao
include/linux/mm_inline.h +16
-0@@ -92,11 +92,196 @@ static __always_inline enum lru_list folio_lru_list(struct folio *folio)return lru; } +#ifdef CONFIG_LRU_GEN + +static inline bool lru_gen_enabled (void ) +{ +return true; +} + +static inline bool lru_gen_in_fault (void ) +{ +return current->in_lru_fault; +} + +static inline int lru_gen_from_seq (unsigned long seq) +{ +return seq %MAX_NR_GENS ; +}
include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 120
Quoted 10 lines
include/linux/mm_inline.h +10
-0+ +static inline bool lru_gen_is_active (struct lruvec *lruvec,int gen) +{ +unsigned long max_seq = lruvec->lrugen.max_seq; + +VM_BUG_ON (gen >=MAX_NR_GENS ); + +/* see the comment on MIN_NR_GENS */ +return gen ==lru_gen_from_seq (max_seq) || gen ==lru_gen_from_seq (max_seq -1 ); +}
I'm still reading the series, so correct me if I'm wrong: the "active" set is split into two generations for the sole purpose of the second-chance policy for fresh faults, right?
If so, it'd be better to have the comment here instead of down by MIN_NR_GENS. This is the place that defines what "active" is, so this is where the reader asks what it means and what it implies. The definition of MIN_NR_GENS can be briefer: "need at least two for second chance, see lru_gen_is_active() for details".
include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 132
Quoted 11 lines
include/linux/mm_inline.h +11
-0+static inline void lru_gen_update_size (struct lruvec *lruvec,enum lru_list lru, +int zone,long delta) +{ +struct pglist_data *pgdat =lruvec_pgdat (lruvec); + +lockdep_assert_held (&lruvec->lru_lock); +WARN_ON_ONCE (delta != (int )delta); + +__mod_lruvec_state (lruvec,NR_LRU_BASE + lru, delta); +__mod_zone_page_state (&pgdat->node_zones[zone],NR_ZONE_LRU_BASE + lru, delta); +}
This is a duplicate of update_lru_size(), please use that instead.
Yeah technically you don't need the mem_cgroup_update_lru_size() but that's not worth sweating over, better to keep it simple.
include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 135
Quoted 2 lines
include/linux/mm_inline.h +2
-0+static inline void lru_gen_balance_size (struct lruvec *lruvec,struct folio *folio, +int old_gen,int new_gen)
lru_gen_update_lru_sizes() for this one would be more descriptive imo and in line with update_lru_size() that it's built on.
include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 145
Quoted 10 lines
include/linux/mm_inline.h +10
-0+{ +int type =folio_is_file_lru (folio); +int zone =folio_zonenum (folio); +int delta =folio_nr_pages (folio); +enum lru_list lru = type *LRU_INACTIVE_FILE ; +struct lru_gen_struct *lrugen = &lruvec->lrugen; + +VM_BUG_ON (old_gen !=-1 && old_gen >=MAX_NR_GENS ); +VM_BUG_ON (new_gen !=-1 && new_gen >=MAX_NR_GENS ); +VM_BUG_ON (old_gen ==-1 && new_gen ==-1 );
Could be a bit easier to read quickly with high-level descriptions:
include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 153
Quoted 7 lines
include/linux/mm_inline.h +7
-0+if (old_gen >=0 ) +WRITE_ONCE (lrugen->nr_pages[old_gen][type][zone], + lrugen->nr_pages[old_gen][type][zone] - delta); +if (new_gen >=0 ) +WRITE_ONCE (lrugen->nr_pages[new_gen][type][zone], + lrugen->nr_pages[new_gen][type][zone] + delta); +
/* Addition */
include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 160
Quoted 7 lines
include/linux/mm_inline.h +7
-0+if (old_gen <0 ) { +if (lru_gen_is_active (lruvec, new_gen)) + lru +=LRU_ACTIVE ; +lru_gen_update_size (lruvec, lru, zone, delta); +return ; + } +
/* Removal */
include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 167
Quoted 7 lines
include/linux/mm_inline.h +7
-0+if (new_gen <0 ) { +if (lru_gen_is_active (lruvec, old_gen)) + lru +=LRU_ACTIVE ; +lru_gen_update_size (lruvec, lru, zone, -delta); +return ; + } +
/* Promotion */
include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 173
Quoted 6 lines
include/linux/mm_inline.h +6
-0+if (!lru_gen_is_active (lruvec, old_gen) &&lru_gen_is_active (lruvec, new_gen)) { +lru_gen_update_size (lruvec, lru, zone, -delta); +lru_gen_update_size (lruvec, lru +LRU_ACTIVE , zone, delta); + } + +/* Promotion is legit while a page is on an LRU list, but demotion isn't. */
/* Demotion happens during aging when pages are isolated, never on-LRU */
Review static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 172
Quoted 2 lines
(quoted diff) +2
-0+ VM_BUG_ON(lru_gen_is_active(lruvec, old_gen) && !lru_gen_is_active(lruvec, new_gen)); +}
On that note, please move introduction of the promotion and demotion bits to the next patch. They aren't used here yet, and I spent some time jumping around patches to verify the promotion callers and confirm the validy of the BUG_ON.
include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 186
Quoted 10 lines
include/linux/mm_inline.h +10
-0+static inline bool lru_gen_add_folio (struct lruvec *lruvec,struct folio *folio,bool reclaiming) +{ +int gen; +unsigned long old_flags, new_flags; +int type =folio_is_file_lru (folio); +int zone =folio_zonenum (folio); +struct lru_gen_struct *lrugen = &lruvec->lrugen; + +if (folio_test_unevictable (folio) || !lrugen->enabled) +return false;
These two checks should be in the callsite and the function should return void. Otherwise you can't understand the callsite without drilling down into lrugen code, even if lrugen is disabled.
folio_add_lru() gets it right.
include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 190
Quoted 4 lines
include/linux/mm_inline.h +4
-0+/* +* There are three common cases for this page: +* 1) If it shouldn't be evicted, e.g., it was just faulted in, add it +* to the youngest generation.
"shouldn't be evicted" makes it sound like mlock. But they should just be evicted last, right? Maybe:
/*
* Pages start in different generations depending on
* advance knowledge we have about their hotness and
* evictability:
*
* 1. Already active pages start out youngest. This can be
* fresh faults, or refaults of previously hot pages.
* 2. Cold pages that require writeback before becoming
* evictable start on the second oldest generation.
* 3. Everything else (clean, cold) starts old. */
On that note, I think #1 is reintroducing a problem we have fixed before, which is trashing the workingset with a flood of use-once mmapped pages. It's the classic scenario where LFU beats LRU.
Mapped streaming IO isn't very common, but it does happen. See these commits:
dfc8d636cdb95f7b792d5ba8c9f3b295809c125d31c0569c3b0b6cc8a867ac6665ca081553f7984c645747462435d84c6c6a64269ed49cc3015f753d
From the changelog:
The used-once mapped file page detection patchset.
It is meant to help workloads with large amounts of shortly used file mappings, like rtorrent hashing a file or git when dealing with loose objects (git gc on a bigger site?).
Right now, the VM activates referenced mapped file pages on first encounter on the inactive list and it takes a full memory cycle to reclaim them again. When those pages dominate memory, the system no longer has a meaningful notion of 'working set' and is required to give up the active list to make reclaim progress. Obviously, this results in rather bad scanning latencies and the wrong pages being reclaimed.
This patch makes the VM be more careful about activating mapped file pages in the first place. The minimum granted lifetime without another memory access becomes an inactive list cycle instead of the full memory cycle, which is more natural given the mentioned loads.
Translating this to multigen, it seems fresh faults should really start on the second oldest rather than on the youngest generation, to get a second chance but without jeopardizing the workingset if they don't take it.
Quoted 4 lines
include/linux/mm_inline.h +4
-0+* 2) If it can't be evicted immediately, i.e., it's an anon page and +* not in swapcache, or a dirty page pending writeback, add it to the +* second oldest generation. +* 3) If it may be evicted immediately, e.g., it's a clean page, add it
include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 204
Quoted 10 lines
include/linux/mm_inline.h +10
-0+* to the oldest generation. +*/ +if (folio_test_active (folio)) + gen =lru_gen_from_seq (lrugen->max_seq); +else if ((!type && !folio_test_swapcache (folio)) || + (folio_test_reclaim (folio) && + (folio_test_dirty (folio) ||folio_test_writeback (folio)))) + gen =lru_gen_from_seq (lrugen->min_seq[type] +1 ); +else + gen =lru_gen_from_seq (lrugen->min_seq[type]);
Condition #2 is not quite clear to me, and the comment is incomplete:
The code does put dirty/writeback pages on the oldest gen as long as they haven't been marked for immediate reclaim by the scanner yet. HOWEVER, once the scanner does see those pages and sets PG_reclaim, it will also activate them to move them out of the way until writeback finishes (see shrink_page_list()) - at which point we'll trigger #1. So that second part of #2 appears unreachable.
It could be a good exercise to describe how cache pages move through the generations, similar to the comment on lru_deactivate_file_fn(). It's a good example of intent vs implementation.
On another note, "!type" meaning "anon" is a bit rough. Please follow the "bool file" convention used elsewhere.
include/linux/mm_inline.h void lruvec_add_folio_tail(struct lruvec *lruvec, struct folio *folio) · line 303
Quoted 7 lines
include/linux/mm_inline.h +3
-0@@ -113,6 +298,9 @@ void lruvec_add_folio_tail(struct lruvec *lruvec, struct folio *folio) {enum lru_list lru =folio_lru_list (folio); +if (lru_gen_add_folio (lruvec, folio, true)) +return ; +
bool parameters are notoriously hard to follow in the callsite. Can you please add lru_gen_add_folio_tail() instead and have them use a common helper?
Quoted 17 lines
include/linux/mm_inline.h +3
-0@@ -127,6 +315,9 @@ static __always_inline void add_page_to_lru_list_tail(struct page *page,static __always_inline void lruvec_del_folio(struct lruvec *lruvec, struct folio *folio) { +if (lru_gen_del_folio (lruvec, folio, false)) +return ; +list_del (&folio->lru);update_lru_size (lruvec,folio_lru_list (folio),folio_zonenum (folio), -folio_nr_pages (folio));include/linux/mmzone.h +0
-0@@ -303,6 +303,78 @@ enum lruvec_flags {*/ };
include/linux/mmzone.h enum lruvec_flags { · line 314
Quoted 10 lines
include/linux/mmzone.h +9
-0+struct lruvec ; + +#define LRU_GEN_MASK ((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF) +#define LRU_REFS_MASK ((BIT(LRU_REFS_WIDTH) - 1) << LRU_REFS_PGOFF) + +#ifdef CONFIG_LRU_GEN + +#define MIN_LRU_BATCH BITS_PER_LONG +#define MAX_LRU_BATCH (MIN_LRU_BATCH * 128)
Those two aren't used in this patch, so it's hard to say whether they are chosen correctly.
Quoted 12 lines
include/linux/mmzone.h +12
-0+* Evictable pages are divided into multiple generations. The youngest and the +* oldest generation numbers, max_seq and min_seq, are monotonically increasing. +* They form a sliding window of a variable size [MIN_NR_GENS, MAX_NR_GENS]. An +* offset within MAX_NR_GENS, gen, indexes the LRU list of the corresponding +* generation. The gen counter in folio->flags stores gen+1 while a page is on +* one of lrugen->lists[]. Otherwise it stores 0. +* +* A page is added to the youngest generation on faulting. The aging needs to +* check the accessed bit at least twice before handing this page over to the +* eviction. The first check takes care of the accessed bit set on the initial +* fault; the second check makes sure this page hasn't been used since then. +* This process, AKA second chance, requires a minimum of two generations,
include/linux/mmzone.h enum lruvec_flags { · line 338
Quoted 10 lines
include/linux/mmzone.h +10
-0+* hence MIN_NR_GENS. And to be compatible with the active/inactive LRU, these +* two generations are mapped to the active; the rest of generations, if they +* exist, are mapped to the inactive. PG_active is always cleared while a page +* is on one of lrugen->lists[] so that demotion, which happens consequently +* when the aging produces a new generation, needs not to worry about it. +*/ +#define MIN_NR_GENS 2U +#define MAX_NR_GENS ((unsigned int)CONFIG_NR_LRU_GENS) + +struct lru_gen_struct {
struct lrugen?
In fact, "lrugen" for the general function and variable namespace might be better, the _ doesn't seem to pull its weight.
CONFIG_LRUGEN
struct lrugenlrugen_foo()
etc.
include/linux/mmzone.h enum lruvec_flags { · line 342
Quoted 4 lines
include/linux/mmzone.h +4
-0+/* the aging increments the youngest generation number */ +unsigned long max_seq; +/* the eviction increments the oldest generation numbers */ +unsigned long min_seq[ANON_AND_FILE ];
The singular max_seq vs the split min_seq raises questions. Please add a comment that explains or points to an explanation.
include/linux/mmzone.h enum lruvec_flags { · line 344
Quoted 2 lines
include/linux/mmzone.h +2
-0+/* the birth time of each generation in jiffies */ +unsigned long timestamps[MAX_NR_GENS ];
This isn't in use until the thrashing-based OOM killing patch.
include/linux/mmzone.h enum lruvec_flags { · line 350
Quoted 6 lines
include/linux/mmzone.h +6
-0+/* the multigenerational LRU lists */ +struct list_head lists[MAX_NR_GENS ][ANON_AND_FILE ][MAX_NR_ZONES ]; +/* the sizes of the above lists */ +unsigned long nr_pages[MAX_NR_GENS ][ANON_AND_FILE ][MAX_NR_ZONES ]; +/* whether the multigenerational LRU is enabled */ +bool enabled;
Not (really) in use until the runtime switch. Best to keep everybody checking the global flag for now, and have the runtime switch patch introduce this flag and switch necessary callsites over.
include/linux/mmzone.h enum lruvec_flags { · line 353
Quoted 1 line
include/linux/mmzone.h +1
-0+void lru_gen_init_state (struct mem_cgroup *memcg,struct lruvec *lruvec);
"state" is what we usually init :) How about lrugen_init_lruvec()?
You can drop the memcg parameter and use lruvec_memcg().
include/linux/mmzone.h enum lruvec_flags { · line 357
Quoted 3 lines
include/linux/mmzone.h +3
-0+#ifdef CONFIG_MEMCG +void lru_gen_init_memcg (struct mem_cgroup *memcg); +void lru_gen_free_memcg (struct mem_cgroup *memcg);
This should be either init+exit, or alloc+free.
Thanks
Re: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
On Thu, Feb 10, 2022 at 03:41:57PM -0500, Johannes Weiner wrote:
Thanks for reviewing.
Quoted 5 lines
Quoted 5 lines
include/linux/mm_inline.h +5
-0+static inline bool lru_gen_is_active (struct lruvec *lruvec,int gen) +{ +unsigned long max_seq = lruvec->lrugen.max_seq; + +VM_BUG_ON (gen >=MAX_NR_GENS );
Quoted 8 lines
include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 120
Quoted 4 lines
include/linux/mm_inline.h +4
-0+ +/* see the comment on MIN_NR_GENS */ +return gen ==lru_gen_from_seq (max_seq) || gen ==lru_gen_from_seq (max_seq -1 ); +}I'm still reading the series, so correct me if I'm wrong: the "active" set is split into two generations for the sole purpose of the second-chance policy for fresh faults, right?
To be precise, the active/inactive notion on top of generations is just for ABI compatibility, e.g., the counters in /proc/vmstat. Otherwise, this function wouldn't be needed.
Quoted 5 lines
If so, it'd be better to have the comment here instead of down by
MIN_NR_GENS. This is the place that defines what "active" is, so this is where the reader asks what it means and what it implies. The definition ofMIN_NR_GENScan be briefer: "need at least two for second chance, seelru_gen_is_active()for details".
This could be understood this way. It'd be more appropriate to see this function as an auxiliary and MIN_NR_GENS as something fundamental. Therefore the former should refer to the latter. Specifically, the "see the comment on MIN_NR_GENS" refers to this part:
And to be compatible with the active/inactive LRU, these two generations are mapped to the active; the rest of generations, if they exist, are mapped to the inactive.
Quoted 14 lines
include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 132
Quoted 11 lines
include/linux/mm_inline.h +11
-0+static inline void lru_gen_update_size (struct lruvec *lruvec,enum lru_list lru, +int zone,long delta) +{ +struct pglist_data *pgdat =lruvec_pgdat (lruvec); + +lockdep_assert_held (&lruvec->lru_lock); +WARN_ON_ONCE (delta != (int )delta); + +__mod_lruvec_state (lruvec,NR_LRU_BASE + lru, delta); +__mod_zone_page_state (&pgdat->node_zones[zone],NR_ZONE_LRU_BASE + lru, delta); +}This is a duplicate of
update_lru_size(), please use that instead.
Quoted 2 lines
Yeah technically you don't need the
mem_cgroup_update_lru_size()but that's not worth sweating over, better to keep it simple.
I agree we don't need the mem_cgroup_update_lru_size() -- let me spell out why:
this function is not needed here because it updates the counters used only by the active/inactive lru code, i.e., get_scan_count().
However, we can't reuse update_lru_size() because MGLRU can trip the WARN_ONCE() in mem_cgroup_update_lru_size().
Unlike lru_zone_size[], lrugen->nr_pages[] is eventually consistent. To move a page to a different generation, the gen counter in page->flags is updated first, which doesn't require the LRU lock. The second step, i.e., the update of lrugen->nr_pages[], requires the LRU lock, and it usually isn't done immediately due to batching. Meanwhile, if this page is, for example, isolated, nr_pages[] becomes temporarily unbalanced. And this trips the WARN_ONCE().
<snipped>
Quoted 9 lines
/* Promotion */include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 173
Quoted 6 lines
include/linux/mm_inline.h +6
-0+if (!lru_gen_is_active (lruvec, old_gen) &&lru_gen_is_active (lruvec, new_gen)) { +lru_gen_update_size (lruvec, lru, zone, -delta); +lru_gen_update_size (lruvec, lru +LRU_ACTIVE , zone, delta); + } + +/* Promotion is legit while a page is on an LRU list, but demotion isn't. */ /* Demotion happens during aging when pages are isolated, never on-LRU */
Quoted 7 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ VM_BUG_ON(lru_gen_is_active(lruvec, old_gen) && !lru_gen_is_active(lruvec, new_gen)); +}On that note, please move introduction of the promotion and demotion bits to the next patch. They aren't used here yet, and I spent some time jumping around patches to verify the promotion callers and confirm the validy of the
BUG_ON.
Will do.
Quoted 6 lines
Quoted 6 lines
include/linux/mm_inline.h +6
-0+static inline bool lru_gen_add_folio (struct lruvec *lruvec,struct folio *folio,bool reclaiming) +{ +int gen; +unsigned long old_flags, new_flags; +int type =folio_is_file_lru (folio); +int zone =folio_zonenum (folio);
Quoted 8 lines
include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 186
Quoted 4 lines
include/linux/mm_inline.h +4
-0+struct lru_gen_struct *lrugen = &lruvec->lrugen; + +if (folio_test_unevictable (folio) || !lrugen->enabled) +return false;These two checks should be in the callsite and the function should return void. Otherwise you can't understand the callsite without drilling down into lrugen code, even if lrugen is disabled.
I agree it's a bit of nuisance this way. The alternative is we'd need ifdef or another helper at the call sites because lrugen->enabled is specific to lrugen.
Quoted 8 lines
include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 190
Quoted 4 lines
include/linux/mm_inline.h +4
-0+/* +* There are three common cases for this page: +* 1) If it shouldn't be evicted, e.g., it was just faulted in, add it +* to the youngest generation. "shouldn't be evicted" makes it sound like mlock. But they should just be evicted last, right? Maybe:
Quoted 11 lines
/*
* Pages start in different generations depending on
* advance knowledge we have about their hotness and
* evictability:
*
* 1. Already active pages start out youngest. This can be
* fresh faults, or refaults of previously hot pages.
* 2. Cold pages that require writeback before becoming
* evictable start on the second oldest generation.
* 3. Everything else (clean, cold) starts old. */
Will do.
Quoted 32 lines
On that note, I think #1 is reintroducing a problem we have fixed before, which is trashing the workingset with a flood of use-once mmapped pages. It's the classic scenario where LFU beats LRU.
Mapped streaming IO isn't very common, but it does happen. See these commits:
dfc8d636cdb95f7b792d5ba8c9f3b295809c125d31c0569c3b0b6cc8a867ac6665ca081553f7984c645747462435d84c6c6a64269ed49cc3015f753dFrom the changelog:
The used-once mapped file page detection patchset.
It is meant to help workloads with large amounts of shortly used file mappings, like rtorrent hashing a file or git when dealing with loose objects (git gc on a bigger site?).
Right now, the VM activates referenced mapped file pages on first encounter on the inactive list and it takes a full memory cycle to reclaim them again. When those pages dominate memory, the system no longer has a meaningful notion of 'working set' and is required to give up the active list to make reclaim progress. Obviously, this results in rather bad scanning latencies and the wrong pages being reclaimed.
This patch makes the VM be more careful about activating mapped file pages in the first place. The minimum granted lifetime without another memory access becomes an inactive list cycle instead of the full memory cycle, which is more natural given the mentioned loads.
Quoted 4 lines
Translating this to multigen, it seems fresh faults should really start on the second oldest rather than on the youngest generation, to get a second chance but without jeopardizing the workingset if they don't take it.
This is a good point, and I had worked on a similar idea but failed to measure its benefits. In addition to placing mmapped file pages in older generations, I also tried placing refaulted anon pages in older generations. My conclusion was that the initial LRU positions of NFU pages are not a bottleneck for workloads I've tested. The efficiency of testing/clearing the accessed bit is.
And some applications are smart enough to leverage MADV_SEQUENTIAL. In this case, MGLRU does place mmapped file pages in the oldest generation.
I have an oversimplified script that uses memcached to mimic a non-streaming workload and fio a (mmapped) streaming workload:
1. With MADV_SEQUENTIAL, the non-streaming workload is about 5 times faster when using MGLRU. Somehow the baseline (rc3) swapped a lot. (It shouldn't, and I haven't figured out why.)
2. Without MADV_SEQUENTIAL, the non-streaming workload is about 1 time faster when using MGLRU. Both MGLRU and the baseline swapped a lot.
MADV_SEQUENTIAL non-streaming ops/sec (memcached)
rc3 yes 292k rc3 no 203k rc3+v7 yes 1967k rc3+v7 no 436k
cat mmap.sh
modprobe brd rd_nr=2 rd_size=56623104
mkswap /dev/ram0
swapon /dev/ram0
mkfs.ext4 /dev/ram1
mount -t ext4 /dev/ram1 /mnt
memtier_benchmark -S /var/run/memcached/memcached.sock -P memcache_binary \
-n allkeys --key-minimum=1 --key-maximum=50000000 --key-pattern=P:P -c 1 \
-t 36 --ratio 1:0 --pipeline 8 -d 2000
# streaming workload: --fadvise_hint=0 disables MADV_SEQUENTIAL
fio -name=mglru --numjobs=12 --directory=/mnt --size=4224m --buffered=1 \
--ioengine=mmap --iodepth=128 --iodepth_batch_submit=32 \
--iodepth_batch_complete=32 --rw=read --time_based --ramp_time=10m \
--runtime=180m --group_reporting &
pid=$!sleep 200
# non-streaming workload
memtier_benchmark -S /var/run/memcached/memcached.sock -P memcache_binary \
-n allkeys --key-minimum=1 --key-maximum=50000000 --key-pattern=R:R \
-c 1 -t 36 --ratio 0:1 --pipeline 8 --randomize --distinct-client-seed kill -INT $pid
wait
Quoted 11 lines
Quoted 11 lines
include/linux/mm_inline.h +11
-0+* 2) If it can't be evicted immediately, i.e., it's an anon page and +* not in swapcache, or a dirty page pending writeback, add it to the +* second oldest generation. +* 3) If it may be evicted immediately, e.g., it's a clean page, add it +* to the oldest generation. +*/ +if (folio_test_active (folio)) + gen =lru_gen_from_seq (lrugen->max_seq); +else if ((!type && !folio_test_swapcache (folio)) || + (folio_test_reclaim (folio) && + (folio_test_dirty (folio) ||folio_test_writeback (folio))))
Quoted 8 lines
include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 204
Quoted 3 lines
include/linux/mm_inline.h +3
-0+ gen =lru_gen_from_seq (lrugen->min_seq[type] +1 ); + else + gen =lru_gen_from_seq (lrugen->min_seq[type]);Condition #2 is not quite clear to me, and the comment is incomplete:
The code does put dirty/writeback pages on the oldest gen as long as they haven't been marked for immediate reclaim by the scanner yet.
Right.
Quoted 4 lines
HOWEVER, once the scanner does see those pages and sets
PG_reclaim, it will also activate them to move them out of the way until writeback finishes (seeshrink_page_list()) - at which point we'll trigger #1. So that second part of #2 appears unreachable.
Yes, dirty file pages go to #1; dirty pages in swapcache go to #2. (Ideally we want dirty file pages go to #2 too. IMO, the code would
be cleaner that way.)
Quoted 3 lines
It could be a good exercise to describe how cache pages move through the generations, similar to the comment on
lru_deactivate_file_fn(). It's a good example of intent vs implementation.
Will do.
Quoted 2 lines
On another note, "!type" meaning "anon" is a bit rough. Please follow the "bool file" convention used elsewhere.
Originally I used "file", e.g., in v2:
https://lore.kernel.org/linux-mm/[email protected]/
But I was told to renamed it since "file" usually means file. Let me rename it back to "file", unless somebody still objects.
Quoted 11 lines
include/linux/mm_inline.h void lruvec_add_folio_tail(struct lruvec *lruvec, struct folio *folio) · line 303
Quoted 7 lines
include/linux/mm_inline.h +3
-0@@ -113,6 +298,9 @@ void lruvec_add_folio_tail(struct lruvec *lruvec, struct folio *folio) {enum lru_list lru =folio_lru_list (folio); +if (lru_gen_add_folio (lruvec, folio, true)) +return ; +bool parameters are notoriously hard to follow in the callsite. Can you please add
lru_gen_add_folio_tail()instead and have them use a common helper?
I'm not sure -- there are several places like this one. My question is whether we want to do it throughout this patchset. We'd end up with many helpers and duplicate code. E.g., in this file alone, we have two functions taking bool parameters:
lru_gen_add_folio(..., bool reclaiming) lru_gen_del_folio(..., bool reclaiming)
I can't say they are very readable; at least they are very compact right now. My concern is that we might loose the latter without having enough of the former.
Perhaps this is something that we could revisit after you've finished reviewing the entire patchset?
Quoted 18 lines
Quoted 18 lines
include/linux/mm_inline.h +3
-0@@ -127,6 +315,9 @@ static __always_inline void add_page_to_lru_list_tail(struct page *page,static __always_inline void lruvec_del_folio(struct lruvec *lruvec, struct folio *folio) { +if (lru_gen_del_folio (lruvec, folio, false)) +return ; +list_del (&folio->lru);update_lru_size (lruvec,folio_lru_list (folio),folio_zonenum (folio), -folio_nr_pages (folio));include/linux/mmzone.h +0
-0@@ -303,6 +303,78 @@ enum lruvec_flags {*/ };
Quoted 12 lines
include/linux/mmzone.h enum lruvec_flags { · line 314
Quoted 9 lines
include/linux/mmzone.h +9
-0+struct lruvec ; + +#define LRU_GEN_MASK ((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF) +#define LRU_REFS_MASK ((BIT(LRU_REFS_WIDTH) - 1) << LRU_REFS_PGOFF) + +#ifdef CONFIG_LRU_GEN + +#define MIN_LRU_BATCH BITS_PER_LONG +#define MAX_LRU_BATCH (MIN_LRU_BATCH * 128)Those two aren't used in this patch, so it's hard to say whether they are chosen correctly.
Right. They slipped during the v6/v7 refactoring. Will move them to the next patch.
Quoted 25 lines
Quoted 12 lines
include/linux/mmzone.h +12
-0+* Evictable pages are divided into multiple generations. The youngest and the +* oldest generation numbers, max_seq and min_seq, are monotonically increasing. +* They form a sliding window of a variable size [MIN_NR_GENS, MAX_NR_GENS]. An +* offset within MAX_NR_GENS, gen, indexes the LRU list of the corresponding +* generation. The gen counter in folio->flags stores gen+1 while a page is on +* one of lrugen->lists[]. Otherwise it stores 0. +* +* A page is added to the youngest generation on faulting. The aging needs to +* check the accessed bit at least twice before handing this page over to the +* eviction. The first check takes care of the accessed bit set on the initial +* fault; the second check makes sure this page hasn't been used since then. +* This process, AKA second chance, requires a minimum of two generations, include/linux/mmzone.h enum lruvec_flags { · line 338
Quoted 10 lines
include/linux/mmzone.h +10
-0+* hence MIN_NR_GENS. And to be compatible with the active/inactive LRU, these +* two generations are mapped to the active; the rest of generations, if they +* exist, are mapped to the inactive. PG_active is always cleared while a page +* is on one of lrugen->lists[] so that demotion, which happens consequently +* when the aging produces a new generation, needs not to worry about it. +*/ +#define MIN_NR_GENS 2U +#define MAX_NR_GENS ((unsigned int)CONFIG_NR_LRU_GENS) + +struct lru_gen_struct {struct lrugen?
Quoted 7 lines
In fact, "lrugen" for the general function and variable namespace might be better, the _ doesn't seem to pull its weight.
CONFIG_LRUGEN
struct lrugenlrugen_foo()
etc.
No strong opinion here. I usually add underscores to functions and types so that grep doesn't end up with tons of local variables.
Quoted 7 lines
include/linux/mmzone.h enum lruvec_flags { · line 342
Quoted 4 lines
include/linux/mmzone.h +4
-0+/* the aging increments the youngest generation number */ +unsigned long max_seq; +/* the eviction increments the oldest generation numbers */ +unsigned long min_seq[ANON_AND_FILE ];The singular
max_seqvs the splitmin_seqraises questions. Please add a comment that explains or points to an explanation.
Will do.
Quoted 4 lines
include/linux/mmzone.h enum lruvec_flags { · line 344
Quoted 2 lines
include/linux/mmzone.h +2
-0+/* the birth time of each generation in jiffies */ +unsigned long timestamps[MAX_NR_GENS ];This isn't in use until the thrashing-based OOM killing patch.
Will move it there.
Quoted 10 lines
include/linux/mmzone.h enum lruvec_flags { · line 350
Quoted 6 lines
include/linux/mmzone.h +6
-0+/* the multigenerational LRU lists */ +struct list_head lists[MAX_NR_GENS ][ANON_AND_FILE ][MAX_NR_ZONES ]; +/* the sizes of the above lists */ +unsigned long nr_pages[MAX_NR_GENS ][ANON_AND_FILE ][MAX_NR_ZONES ]; +/* whether the multigenerational LRU is enabled */ +bool enabled;Not (really) in use until the runtime switch. Best to keep everybody checking the global flag for now, and have the runtime switch patch introduce this flag and switch necessary callsites over.
Will do.
Quoted 3 lines
include/linux/mmzone.h enum lruvec_flags { · line 353
Quoted 1 line
include/linux/mmzone.h +1
-0+void lru_gen_init_state (struct mem_cgroup *memcg,struct lruvec *lruvec);"state" is what we usually init :) How about
lrugen_init_lruvec()?
Same story as "file", lol -- this used to be lru_gen_init_lruvec():
https://lore.kernel.org/linux-mm/[email protected]/
Naming is hard. Hopefully we can finalize it this time.
Quoted 1 line
You can drop the memcg parameter and use
lruvec_memcg().
lruvec_memcg() isn't available yet when pgdat_init_internals() calls this function because mem_cgroup_disabled() is initialized afterward.
Quoted 5 lines
include/linux/mmzone.h enum lruvec_flags { · line 357
Quoted 3 lines
include/linux/mmzone.h +3
-0+#ifdef CONFIG_MEMCG +void lru_gen_init_memcg (struct mem_cgroup *memcg); +void lru_gen_free_memcg (struct mem_cgroup *memcg);This should be either init+exit, or alloc+free.
Will do.
Johannes Weiner
maintainerRe: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
Hi Yu,
Quoted 15 lines · Yu Zhao
Quoted 5 lines · Johannes Weiner
Quoted 5 lines
(quoted diff) +5
-0+static inline bool lru_gen_is_active(struct lruvec *lruvec, int gen) +{ + unsigned long max_seq = lruvec->lrugen.max_seq; + + VM_BUG_ON(gen >= MAX_NR_GENS);Quoted 8 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+ + /* see the comment on MIN_NR_GENS */ + return gen == lru_gen_from_seq(max_seq) || gen == lru_gen_from_seq(max_seq - 1); +}I'm still reading the series, so correct me if I'm wrong: the "active" set is split into two generations for the sole purpose of the second-chance policy for fresh faults, right?
Quoted 3 lines
To be precise, the active/inactive notion on top of generations is just for ABI compatibility, e.g., the counters in
/proc/vmstat. Otherwise, this function wouldn't be needed.
Ah! would you mind adding this as a comment to the function?
But AFAICS there is the lru_gen_del_folio() callsite that maps it to the PG_active flag - which in turn gets used by add_folio() to place the thing back on the max_seq generation. So I suppose there is a secondary purpose of the function for remembering the page's rough age for non-reclaim isolation. It would be good to capture that as well in a comment on the function.
Quoted 6 lines
Quoted 5 lines
If so, it'd be better to have the comment here instead of down by
MIN_NR_GENS. This is the place that defines what "active" is, so this is where the reader asks what it means and what it implies. The definition ofMIN_NR_GENScan be briefer: "need at least two for second chance, seelru_gen_is_active()for details".
Quoted 7 lines
This could be understood this way. It'd be more appropriate to see this function as an auxiliary and
MIN_NR_GENSas something fundamental. Therefore the former should refer to the latter. Specifically, the "see the comment onMIN_NR_GENS" refers to this part:
And to be compatible with the active/inactive LRU, these two generations are mapped to the active; the rest of generations, if they exist, are mapped to the inactive.
I agree, thanks for enlightening me.
Quoted 25 lines
Quoted 14 lines
Review
Quoted 11 lines
(quoted diff) +11
-0+static inline void lru_gen_update_size(struct lruvec *lruvec, enum lru_list lru, + int zone, long delta) +{ + struct pglist_data *pgdat = lruvec_pgdat(lruvec); + + lockdep_assert_held(&lruvec->lru_lock); + WARN_ON_ONCE(delta != (int)delta); + + __mod_lruvec_state(lruvec, NR_LRU_BASE + lru, delta); + __mod_zone_page_state(&pgdat->node_zones[zone], NR_ZONE_LRU_BASE + lru, delta); +}This is a duplicate of
update_lru_size(), please use that instead.Quoted 2 lines
Yeah technically you don't need the
mem_cgroup_update_lru_size()but that's not worth sweating over, better to keep it simple.I agree we don't need the
mem_cgroup_update_lru_size()-- let me spell out why:
this function is not needed here because it updates the counters used only by the active/inactive lru code, i.e.,get_scan_count().However, we can't reuse
update_lru_size()because MGLRU can trip theWARN_ONCE()inmem_cgroup_update_lru_size().
Quoted 7 lines
Unlike lru_zone_size[], lrugen->nr_pages[] is eventually consistent. To move a page to a different generation, the gen counter in
page->flagsis updated first, which doesn't require the LRU lock. The second step, i.e., the update of lrugen->nr_pages[], requires the LRU lock, and it usually isn't done immediately due to batching. Meanwhile, if this page is, for example, isolated, nr_pages[] becomes temporarily unbalanced. And this trips theWARN_ONCE().
Good insight.
But in that case, I'd still think it's better to use update_lru_size() and gate the memcg update on lrugen-enabled, with a short comment saying that lrugen has its own per-cgroup counts already. It's just a bit too error prone to duplicate the stat updates.
Even better would be:
static __always_inline
void lruvec_add_folio(struct lruvec *lruvec, struct folio *folio)
{
enum lru_list lru = folio_lru_list(folio);
update_lru_size(lruvec, lru, folio_zonenum(folio),
folio_nr_pages(folio));
if (lrugen_enabled(lruvec))
lrugen_add_folio(lruvec, folio);
else
list_add(&folio->lru, &lruvec->lists[lru]);
}But it does mean you'd have to handle unevictable pages. I'm reviewing from the position that mglru is going to supplant the existing reclaim algorithm in the long term, though, so being more comprehensive and eliminating special cases where possible is all-positive, IMO.
Up to you. I'd only insist on reusing update_lru_size() at least.
Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
(quoted diff) +6
-0+static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming) +{ + int gen; + unsigned long old_flags, new_flags; + int type = folio_is_file_lru(folio); + int zone = folio_zonenum(folio);Quoted 8 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+ struct lru_gen_struct *lrugen = &lruvec->lrugen; + + if (folio_test_unevictable(folio) || !lrugen->enabled) + return false;These two checks should be in the callsite and the function should return void. Otherwise you can't understand the callsite without drilling down into lrugen code, even if lrugen is disabled.
Quoted 3 lines
I agree it's a bit of nuisance this way. The alternative is we'd need ifdef or another helper at the call sites because
lrugen->enabledis specific to lrugen.
Coming from memcg, my experience has been that when you have a compile time-optional MM extension like this, you'll sooner or later need a config-independent helper to gate callbacks in generic code. So I think it's a good idea to add one now.
One of these?
lruvec_on_lrugen()lruvec_using_lrugen()lruvec_lrugen_enabled()
lruvec_has_generations() :-)
Quoted 37 lines
Quoted 32 lines
On that note, I think #1 is reintroducing a problem we have fixed before, which is trashing the workingset with a flood of use-once mmapped pages. It's the classic scenario where LFU beats LRU.
Mapped streaming IO isn't very common, but it does happen. See these commits:
dfc8d636cdb95f7b792d5ba8c9f3b295809c125d31c0569c3b0b6cc8a867ac6665ca081553f7984c645747462435d84c6c6a64269ed49cc3015f753dFrom the changelog:
The used-once mapped file page detection patchset.
It is meant to help workloads with large amounts of shortly used file mappings, like rtorrent hashing a file or git when dealing with loose objects (git gc on a bigger site?).
Right now, the VM activates referenced mapped file pages on first encounter on the inactive list and it takes a full memory cycle to reclaim them again. When those pages dominate memory, the system no longer has a meaningful notion of 'working set' and is required to give up the active list to make reclaim progress. Obviously, this results in rather bad scanning latencies and the wrong pages being reclaimed.
This patch makes the VM be more careful about activating mapped file pages in the first place. The minimum granted lifetime without another memory access becomes an inactive list cycle instead of the full memory cycle, which is more natural given the mentioned loads.
Quoted 4 lines
Translating this to multigen, it seems fresh faults should really start on the second oldest rather than on the youngest generation, to get a second chance but without jeopardizing the workingset if they don't take it.
Quoted 6 lines
This is a good point, and I had worked on a similar idea but failed to measure its benefits. In addition to placing mmapped file pages in older generations, I also tried placing refaulted anon pages in older generations. My conclusion was that the initial LRU positions of NFU pages are not a bottleneck for workloads I've tested. The efficiency of testing/clearing the accessed bit is.
The concern isn't the scan overhead, but jankiness from the workingset being flooded out by streaming IO.
The concrete usecase at the time was a torrent client hashing a downloaded file and thereby kicking out the desktop environment, which caused jankiness. The hashing didn't benefit from caching - the file wouldn't have fit into RAM anyway - so this was pointless to boot.
Essentially, the tradeoff is this:
1) If you treat new pages as hot, you accelerate workingset transitions, but on the flipside you risk unnecessary refaults in running applications when those new pages are one-off.
2) If you take new pages with a grain of salt, you protect existing applications better from one-off floods, but risk refaults in NEW application while they're trying to start up.
There are two arguments for why 2) is preferable:
1) Users are tolerant of cache misses when applications first launch, much less so after they've been running for hours.
2) Workingset transitions (and associated jankiness) are bounded by the amount of RAM you need to repopulate. But streaming IO is bounded by storage, and datasets are routinely several times the amount of RAM. Uncacheable sets in excess of RAM can produce an infinite stream of "new" references; not protecting the workingset from that means longer or even sustained jankiness.
Quoted 3 lines
And some applications are smart enough to leverage
MADV_SEQUENTIAL. In this case, MGLRU does place mmapped file pages in the oldest generation.
Yes, it makes sense to optimize when MADV_SEQUENTIAL is requested. But that hint isn't reliably there, so it matters that we don't do poorly when it's missing.
Quoted 2 lines
I have an oversimplified script that uses memcached to mimic a non-streaming workload and fio a (mmapped) streaming workload:
Looking at the paramters and observed behavior, let me say up front that this looks like a useful benchmark, but doesn't capture the scenario I was talking about above.
For one, the presence of swapping in both kernels suggests that the "streaming IO" component actually has repeat access that could benefit from caching. Second, I would expect memcache is accessing its memory frequently and consistently, and so could withstand workingset challenges from streaming IO better than, say, a desktop environment.
More on that below.
Quoted 3 lines
1. With
MADV_SEQUENTIAL, the non-streaming workload is about 5 times faster when using MGLRU. Somehow the baseline (rc3) swapped a lot. (It shouldn't, and I haven't figured out why.)
Baseline swaps when there are cache refaults. This is regardless of the hint: you may say you're accessing these pages sequentially, but the refaults say you're reusing them, with a frequency that suggests they might be cacheable. So it tries to cache them.
I'd be curious if that results in fio being faster, or whether it's all just pointless thrashing. Can you share the fio results too?
We could patch baseline to prioritize MADV_SEQUENTIAL more, but...
Quoted 3 lines
2. Without
MADV_SEQUENTIAL, the non-streaming workload is about 1 time faster when using MGLRU. Both MGLRU and the baseline swapped a lot.
...in practice I think this scenario will matter to a lot more users.
I would again be interested in the fio results.
Quoted 15 lines
MADV_SEQUENTIALnon-streaming ops/sec (memcached)rc3 yes 292k rc3 no 203k rc3+v7 yes 1967k rc3+v7 no 436kcat
mmap.sh
modprobe brd rd_nr=2 rd_size=56623104mkswap
/dev/ram0
swapon/dev/ram0mkfs.ext4
/dev/ram1
mount -t ext4/dev/ram1/mnt
Quoted 9 lines
memtier_benchmark -S /var/run/memcached/memcached.sock -P memcache_binary \
-n allkeys --key-minimum=1 --key-maximum=50000000 --key-pattern=P:P -c 1 \
-t 36 --ratio 1:0 --pipeline 8 -d 2000
# streaming workload: --fadvise_hint=0 disables MADV_SEQUENTIAL
fio -name=mglru --numjobs=12 --directory=/mnt --size=4224m --buffered=1 \
--ioengine=mmap --iodepth=128 --iodepth_batch_submit=32 \
--iodepth_batch_complete=32 --rw=read --time_based --ramp_time=10m \
--runtime=180m --group_reporting &As per above, I think this would be closer to a cacheable workingset than a streaming IO pattern. It depends on total RAM of course, but size=4G and time_based should loop around pretty quickly.
Would you mind rerunning with files larger than RAM, to avoid repeat accesses (or at least only repeat with large distances)?
Depending on how hot memcache runs, it may or may not be able to hold onto its workingset. Testing interactivity is notoriously hard, but using a smaller, intermittent workload is probably more representative of overall responsiveness. Let fio ramp until memory is full, then do perf stat -r 10 /bin/sh -c 'git shortlog v5.0.. >/dev/null; sleep 1'
I'll try to reproduce this again too. Back then, that workload gave me a very janky desktop experience, and the patch very obvious relief.
Quoted 19 lines
Quoted 11 lines
Review void lruvec_add_folio_tail(struct lruvec *lruvec, struct folio *folio) · line 303
Quoted 7 lines
(quoted diff) +3
-0@@ -113,6 +298,9 @@ void lruvec_add_folio_tail(struct lruvec *lruvec, struct folio *folio) { enum lru_list lru = folio_lru_list(folio); + if (lru_gen_add_folio(lruvec, folio, true)) + return; +bool parameters are notoriously hard to follow in the callsite. Can you please add
lru_gen_add_folio_tail()instead and have them use a common helper?I'm not sure -- there are several places like this one. My question is whether we want to do it throughout this patchset. We'd end up with many helpers and duplicate code. E.g., in this file alone, we have two functions taking bool parameters:
lru_gen_add_folio(..., bool reclaiming)lru_gen_del_folio(..., bool reclaiming)
Quoted 6 lines
I can't say they are very readable; at least they are very compact right now. My concern is that we might loose the latter without having enough of the former.
Perhaps this is something that we could revisit after you've finished reviewing the entire patchset?
Sure, fair enough.
Quoted 8 lines
Quoted 3 lines
Review void lruvec_add_folio_tail(struct lruvec *lruvec, struct folio *folio) · line 304
Quoted 1 line
(quoted diff) +1
-0+void lru_gen_init_state(struct mem_cgroup *memcg, struct lruvec *lruvec);"state" is what we usually init :) How about
lrugen_init_lruvec()?Same story as "file", lol -- this used to be
lru_gen_init_lruvec():
https://lore.kernel.org/linux-mm/[email protected]/Naming is hard. Hopefully we can finalize it this time.
Was that internal feedback? The revisions show this function went through several names, but I don't see reviews requesting those. If they weren't public I'm gonna pretend they didn't happen ;-)
Quoted 4 lines
Quoted 1 line
You can drop the memcg parameter and use
lruvec_memcg().
lruvec_memcg()isn't available yet whenpgdat_init_internals()calls this function becausemem_cgroup_disabled()is initialized afterward.
Good catch. That'll container_of() into garbage. However, we have to assume that somebody's going to try that simplification again, so we should set up the code now to prevent issues.
cgroup_disable parsing is self-contained, so we can pull it ahead in the init sequence. How about this?
kernel/cgroup/cgroup.c +2 -2
@@ -6464,9 +6464,9 @@ static int __init cgroup_disable(char *str)break ; } } -return 1 ; +return 0 ; } -__setup ("cgroup_disable=" , cgroup_disable); +early_param ("cgroup_disable" , cgroup_disable);void __init __weak enable_debug_cgroup (void ) { }
Thanks!
Re: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
Quoted 18 lines · Johannes Weiner
Hi Yu,
Quoted 15 lines · Yu Zhao
Quoted 13 lines · Johannes Weiner
Review
Quoted 9 lines
(quoted diff) +9
-0+static inline bool lru_gen_is_active(struct lruvec *lruvec, int gen) +{ + unsigned long max_seq = lruvec->lrugen.max_seq; + + VM_BUG_ON(gen >= MAX_NR_GENS); + + /* see the comment on MIN_NR_GENS */ + return gen == lru_gen_from_seq(max_seq) || gen == lru_gen_from_seq(max_seq - 1); +}I'm still reading the series, so correct me if I'm wrong: the "active" set is split into two generations for the sole purpose of the second-chance policy for fresh faults, right?
Quoted 5 lines
Quoted 3 lines
To be precise, the active/inactive notion on top of generations is just for ABI compatibility, e.g., the counters in
/proc/vmstat. Otherwise, this function wouldn't be needed.Ah! would you mind adding this as a comment to the function?
Will do.
Quoted 5 lines
But AFAICS there is the
lru_gen_del_folio()callsite that maps it to thePG_activeflag - which in turn gets used byadd_folio()to place the thing back on themax_seqgeneration. So I suppose there is a secondary purpose of the function for remembering the page's rough age for non-reclaim isolation.>
Yes, e.g., migration.
Quoted 1 line
It would be good to capture that as well in a comment on the function.
Will do.
Quoted 55 lines
Quoted 25 lines
Quoted 14 lines
Review
Quoted 11 lines
(quoted diff) +11
-0+static inline void lru_gen_update_size(struct lruvec *lruvec, enum lru_list lru, + int zone, long delta) +{ + struct pglist_data *pgdat = lruvec_pgdat(lruvec); + + lockdep_assert_held(&lruvec->lru_lock); + WARN_ON_ONCE(delta != (int)delta); + + __mod_lruvec_state(lruvec, NR_LRU_BASE + lru, delta); + __mod_zone_page_state(&pgdat->node_zones[zone], NR_ZONE_LRU_BASE + lru, delta); +}This is a duplicate of
update_lru_size(), please use that instead.Quoted 2 lines
Yeah technically you don't need the
mem_cgroup_update_lru_size()but that's not worth sweating over, better to keep it simple.I agree we don't need the
mem_cgroup_update_lru_size()-- let me spell out why:
this function is not needed here because it updates the counters used only by the active/inactive lru code, i.e.,get_scan_count().However, we can't reuse
update_lru_size()because MGLRU can trip theWARN_ONCE()inmem_cgroup_update_lru_size().Quoted 7 lines
Unlike lru_zone_size[], lrugen->nr_pages[] is eventually consistent. To move a page to a different generation, the gen counter in
page->flagsis updated first, which doesn't require the LRU lock. The second step, i.e., the update of lrugen->nr_pages[], requires the LRU lock, and it usually isn't done immediately due to batching. Meanwhile, if this page is, for example, isolated, nr_pages[] becomes temporarily unbalanced. And this trips theWARN_ONCE().Good insight.
But in that case, I'd still think it's better to use
update_lru_size()and gate the memcg update on lrugen-enabled, with a short comment saying that lrugen has its own per-cgroup counts already. It's just a bit too error prone to duplicate the stat updates.Even better would be:
static
__always_inline
voidlruvec_add_folio(struct lruvec *lruvec, struct folio *folio){ enum lru_list lru = folio_lru_list(folio); update_lru_size(lruvec, lru, folio_zonenum(folio), folio_nr_pages(folio)); if (lrugen_enabled(lruvec)) lrugen_add_folio(lruvec, folio); else list_add(&folio->lru, &lruvec->lists[lru]); }
Quoted 6 lines
But it does mean you'd have to handle unevictable pages. I'm reviewing from the position that mglru is going to supplant the existing reclaim algorithm in the long term, though, so being more comprehensive and eliminating special cases where possible is all-positive, IMO.
Up to you. I'd only insist on reusing
update_lru_size()at least.
Will do.
Quoted 19 lines
Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
(quoted diff) +6
-0+static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming) +{ + int gen; + unsigned long old_flags, new_flags; + int type = folio_is_file_lru(folio); + int zone = folio_zonenum(folio);Quoted 8 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+ struct lru_gen_struct *lrugen = &lruvec->lrugen; + + if (folio_test_unevictable(folio) || !lrugen->enabled) + return false;These two checks should be in the callsite and the function should return void. Otherwise you can't understand the callsite without drilling down into lrugen code, even if lrugen is disabled.
Quoted 3 lines
I agree it's a bit of nuisance this way. The alternative is we'd need ifdef or another helper at the call sites because
lrugen->enabledis specific to lrugen.
Quoted 8 lines
Coming from memcg, my experience has been that when you have a compile time-optional MM extension like this, you'll sooner or later need a config-independent helper to gate callbacks in generic code. So I think it's a good idea to add one now.
One of these?
lruvec_on_lrugen()
SGTM.
Personally I'd reuse lru_gen_enabled(), by passing NULL/lruvec. But my guess is you wouldn't like it.
Quoted 49 lines
lruvec_using_lrugen()lruvec_lrugen_enabled()
lruvec_has_generations():-)Quoted 37 lines
Quoted 32 lines
On that note, I think #1 is reintroducing a problem we have fixed before, which is trashing the workingset with a flood of use-once mmapped pages. It's the classic scenario where LFU beats LRU.
Mapped streaming IO isn't very common, but it does happen. See these commits:
dfc8d636cdb95f7b792d5ba8c9f3b295809c125d31c0569c3b0b6cc8a867ac6665ca081553f7984c645747462435d84c6c6a64269ed49cc3015f753dFrom the changelog:
The used-once mapped file page detection patchset.
It is meant to help workloads with large amounts of shortly used file mappings, like rtorrent hashing a file or git when dealing with loose objects (git gc on a bigger site?).
Right now, the VM activates referenced mapped file pages on first encounter on the inactive list and it takes a full memory cycle to reclaim them again. When those pages dominate memory, the system no longer has a meaningful notion of 'working set' and is required to give up the active list to make reclaim progress. Obviously, this results in rather bad scanning latencies and the wrong pages being reclaimed.
This patch makes the VM be more careful about activating mapped file pages in the first place. The minimum granted lifetime without another memory access becomes an inactive list cycle instead of the full memory cycle, which is more natural given the mentioned loads.
Quoted 4 lines
Translating this to multigen, it seems fresh faults should really start on the second oldest rather than on the youngest generation, to get a second chance but without jeopardizing the workingset if they don't take it.
Quoted 6 lines
This is a good point, and I had worked on a similar idea but failed to measure its benefits. In addition to placing mmapped file pages in older generations, I also tried placing refaulted anon pages in older generations. My conclusion was that the initial LRU positions of NFU pages are not a bottleneck for workloads I've tested. The efficiency of testing/clearing the accessed bit is.
Quoted 2 lines
The concern isn't the scan overhead, but jankiness from the workingset being flooded out by streaming IO.
Yes, MGLRU uses a different approach to solve this problem, and for its approach, the scan overhead is the concern.
MGLRU detects (defines) the working set by scanning the entire memory for each generation, and it counters the flooding by accelerating the creation of generations. IOW, all mapped pages have an equal chance to get scanned, no matter which generation they are in. This is a design difference compared with the active/inactive LRU, which tries to scans the active/inactive lists less/more frequently.
Quoted 7 lines
The concrete usecase at the time was a torrent client hashing a downloaded file and thereby kicking out the desktop environment, which caused jankiness. The hashing didn't benefit from caching - the file wouldn't have fit into RAM anyway - so this was pointless to boot.
Essentially, the tradeoff is this:
Quoted 7 lines
1) If you treat new pages as hot, you accelerate workingset transitions, but on the flipside you risk unnecessary refaults in running applications when those new pages are one-off.
2) If you take new pages with a grain of salt, you protect existing applications better from one-off floods, but risk refaults in NEW application while they're trying to start up.
Agreed.
Quoted 4 lines
There are two arguments for why 2) is preferable:
1) Users are tolerant of cache misses when applications first launch, much less so after they've been running for hours.
Our CUJs (Critical User Journeys) respectfully disagree :)
They are built on the observation that once users have moved onto another tab/app, they are more likely to stay with the new tab/app rather than go back to the old ones. Speaking for myself, this is generally the case.
Quoted 6 lines
2) Workingset transitions (and associated jankiness) are bounded by the amount of RAM you need to repopulate. But streaming IO is bounded by storage, and datasets are routinely several times the amount of RAM. Uncacheable sets in excess of RAM can produce an infinite stream of "new" references; not protecting the workingset from that means longer or even sustained jankiness.
I'd argue the opposite -- we shouldn't risk refaulting fresh hot pages just to accommodate this concrete yet minor use case, especially considering torrent has been given the means (MADV_SEQUENTIAL) to help itself.
I appreciate all your points here. The bottom line is we agree this is a trade off. For what disagree about, we could be both right -- it comes down to what workloads we care about *more*.
To move forward, I propose we look at it from a non-technical POV:
would we want to offer users an alternative trade off so that they can have greater flexibility?
Quoted 7 lines
Quoted 3 lines
And some applications are smart enough to leverage
MADV_SEQUENTIAL. In this case, MGLRU does place mmapped file pages in the oldest generation.Yes, it makes sense to optimize when
MADV_SEQUENTIALis requested. But that hint isn't reliably there, so it matters that we don't do poorly when it's missing.
Agreed.
Quoted 7 lines
Quoted 2 lines
I have an oversimplified script that uses memcached to mimic a non-streaming workload and fio a (mmapped) streaming workload:
Looking at the paramters and observed behavior, let me say up front that this looks like a useful benchmark, but doesn't capture the scenario I was talking about above.
Quoted 5 lines
For one, the presence of swapping in both kernels suggests that the "streaming IO" component actually has repeat access that could benefit from caching. Second, I would expect memcache is accessing its memory frequently and consistently, and so could withstand workingset challenges from streaming IO better than, say, a desktop environment.
The fio workload is a real streaming workload, but the memcached workload might have been too large to be a typical desktop workload.
More below.
Quoted 6 lines
More on that below.
Quoted 3 lines
1. With
MADV_SEQUENTIAL, the non-streaming workload is about 5 times faster when using MGLRU. Somehow the baseline (rc3) swapped a lot. (It shouldn't, and I haven't figured out why.)
Quoted 7 lines
Baseline swaps when there are cache refaults. This is regardless of the hint: you may say you're accessing these pages sequentially, but the refaults say you're reusing them, with a frequency that suggests they might be cacheable. So it tries to cache them.
I'd be curious if that results in fio being faster, or whether it's all just pointless thrashing. Can you share the fio results too?
More below.
Quoted 7 lines
We could patch baseline to prioritize
MADV_SEQUENTIALmore, but...Quoted 3 lines
2. Without
MADV_SEQUENTIAL, the non-streaming workload is about 1 time faster when using MGLRU. Both MGLRU and the baseline swapped a lot....in practice I think this scenario will matter to a lot more users.
I strongly feel we should prioritize what's advertised on a man page over an unspecified (performance) behavior.
Quoted 27 lines
I would again be interested in the fio results.
Quoted 15 lines
MADV_SEQUENTIALnon-streaming ops/sec (memcached)rc3 yes 292k rc3 no 203k rc3+v7 yes 1967k rc3+v7 no 436kcat
mmap.sh
modprobe brd rd_nr=2 rd_size=56623104mkswap
/dev/ram0
swapon/dev/ram0mkfs.ext4
/dev/ram1
mount -t ext4/dev/ram1/mntQuoted 9 lines
memtier_benchmark -S /var/run/memcached/memcached.sock -P memcache_binary \ -n allkeys --key-minimum=1 --key-maximum=50000000 --key-pattern=P:P -c 1 \ -t 36 --ratio 1:0 --pipeline 8 -d 2000 # streaming workload: --fadvise_hint=0 disables MADV_SEQUENTIAL fio -name=mglru --numjobs=12 --directory=/mnt --size=4224m --buffered=1 \ --ioengine=mmap --iodepth=128 --iodepth_batch_submit=32 \ --iodepth_batch_complete=32 --rw=read --time_based --ramp_time=10m \ --runtime=180m --group_reporting &
Quoted 3 lines
As per above, I think this would be closer to a cacheable workingset than a streaming IO pattern. It depends on total RAM of course, but size=4G and
time_basedshould loop around pretty quickly.
The file size here shouldn't matter since fio is smart enough to invalidate page cache before it rewinds (for sequential access):
https://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-invalidate
https://github.com/axboe/fio/blob/master/filesetup.c#L602
I think the problem might have been the memory size for memcached was too large (100GB) to be all hot (limited by memory bandwidth).
Quoted 2 lines
Would you mind rerunning with files larger than RAM, to avoid repeat accesses (or at least only repeat with large distances)?
Retested with the same free memory (120GB) for 40GB memcached and 200GB fio.
MADV_SEQUENTIAL FADV_DONTNEED memcached fio rc4 no yes 4716k 232k rc4+v7 no yes 4307k 265k delta -9% +14%
MGLRU lost with memcached but won with fio for the same reason: it doesn't have any heuristics to detect the streaming characteristic (and therefore lost with memcached) but relies on faster scanning (and therefore won with fio) to keep the working set in memory.
The baseline didn't swap this time (MGLRU did slightly), but it lost with fio because it had to walk the rmap for each page in the entire 200GB VMA, at least once, even for this streaming workload.
This reflects the design difference I mentioned earlier.
cat test.sh
modprobe brd rd_nr=1 rd_size=268435456
mkfs.ext4 /dev/ram0
mount -t ext4 /dev/ram0 /mnt
fallocate -l 40g /mnt/swapfile
mkswap /mnt/swapfile
swapon /mnt/swapfile
fio -name=mglru --numjobs=1 --directory=/mnt --size=204800m \
--buffered=1 --ioengine=mmap --fadvise_hint=0 --iodepth=128 \
--iodepth_batch_submit=32 --iodepth_batch_complete=32 \
--rw=read --time_based --ramp_time=10m --runtime=180m \
--group_reporting &
pid=$!sleep 600
# load objects
memtier_benchmark -S /var/run/memcached/memcached.sock \
-P memcache_binary -n allkeys --key-minimum=1 \
--key-maximum=20000000 --key-pattern=P:P -c 1 -t 36 \
--ratio 1:0 --pipeline 8 -d 2000
# read objects
memtier_benchmark -S /var/run/memcached/memcached.sock \
-P memcache_binary -n allkeys --key-minimum=1 \
--key-maximum=20000000 --key-pattern=R:R -c 1 -t 36 \
--ratio 0:1 --pipeline 8 --randomize --distinct-client-seedkill -INT $pid
Quoted 2 lines
Depending on how hot memcache runs, it may or may not be able to hold onto its workingset.
Agreed.
Quoted 4 lines
Testing interactivity is notoriously hard, but
using a smaller, intermittent workload is probably more representative of overall responsiveness. Let fio ramp until memory is full, then do perf stat -r 10/bin/sh-c 'git shortlog v5.0.. >/dev/null; sleep 1'
I'll also check with the downstream maintainers to see if they have heard any complaints about streaming workloads negatively impacting user experience.
Quoted 2 lines
I'll try to reproduce this again too. Back then, that workload gave me a very janky desktop experience, and the patch very obvious relief.
SGTM.
Quoted 32 lines
Quoted 19 lines
Quoted 11 lines
Review void lruvec_add_folio_tail(struct lruvec *lruvec, struct folio *folio) · line 303
Quoted 7 lines
(quoted diff) +3
-0@@ -113,6 +298,9 @@ void lruvec_add_folio_tail(struct lruvec *lruvec, struct folio *folio) { enum lru_list lru = folio_lru_list(folio); + if (lru_gen_add_folio(lruvec, folio, true)) + return; +bool parameters are notoriously hard to follow in the callsite. Can you please add
lru_gen_add_folio_tail()instead and have them use a common helper?I'm not sure -- there are several places like this one. My question is whether we want to do it throughout this patchset. We'd end up with many helpers and duplicate code. E.g., in this file alone, we have two functions taking bool parameters:
lru_gen_add_folio(..., bool reclaiming)lru_gen_del_folio(..., bool reclaiming)Quoted 6 lines
I can't say they are very readable; at least they are very compact right now. My concern is that we might loose the latter without having enough of the former.
Perhaps this is something that we could revisit after you've finished reviewing the entire patchset?
Sure, fair enough.
Quoted 4 lines
Quoted 3 lines
Review void lruvec_add_folio_tail(struct lruvec *lruvec, struct folio *folio) · line 304
Quoted 1 line
(quoted diff) +1
-0+void lru_gen_init_state(struct mem_cgroup *memcg, struct lruvec *lruvec);"state" is what we usually init :) How about
lrugen_init_lruvec()?
Quoted 8 lines
Quoted 4 lines
Same story as "file", lol -- this used to be
lru_gen_init_lruvec():
https://lore.kernel.org/linux-mm/[email protected]/Naming is hard. Hopefully we can finalize it this time.
Was that internal feedback? The revisions show this function went through several names, but I don't see reviews requesting those. If they weren't public I'm gonna pretend they didn't happen ;-)
Indeed. I lost track.
Quoted 17 lines
Quoted 4 lines
Quoted 1 line
You can drop the memcg parameter and use
lruvec_memcg().
lruvec_memcg()isn't available yet whenpgdat_init_internals()calls this function becausemem_cgroup_disabled()is initialized afterward.Good catch. That'll
container_of()into garbage. However, we have to assume that somebody's going to try that simplification again, so we should set up the code now to prevent issues.
cgroup_disableparsing is self-contained, so we can pull it ahead in the init sequence. How about this?kernel/cgroup/cgroup.c +0
-0
kernel/cgroup/cgroup.c static int __init cgroup_disable(char *str) · line 6469
Quoted 8 lines
kernel/cgroup/cgroup.c +2
-2@@ -6464,9 +6464,9 @@ static int __init cgroup_disable(char *str)break ; } } -return 1 ; +return 0 ; } -__setup ("cgroup_disable=" , cgroup_disable); +early_param ("cgroup_disable" , cgroup_disable);
I think early_param() is still after pgdat_init_internals(), no?
Thanks!
Re: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
.
Quoted 294 lines · Yu Zhao
Quoted 18 lines · Johannes Weiner
Hi Yu,
Quoted 15 lines · Yu Zhao
Quoted 13 lines · Johannes Weiner
Review
Quoted 9 lines
(quoted diff) +9
-0+static inline bool lru_gen_is_active(struct lruvec *lruvec, int gen) +{ + unsigned long max_seq = lruvec->lrugen.max_seq; + + VM_BUG_ON(gen >= MAX_NR_GENS); + + /* see the comment on MIN_NR_GENS */ + return gen == lru_gen_from_seq(max_seq) || gen == lru_gen_from_seq(max_seq - 1); +}I'm still reading the series, so correct me if I'm wrong: the "active" set is split into two generations for the sole purpose of the second-chance policy for fresh faults, right?
Quoted 5 lines
Quoted 3 lines
To be precise, the active/inactive notion on top of generations is just for ABI compatibility, e.g., the counters in
/proc/vmstat. Otherwise, this function wouldn't be needed.Ah! would you mind adding this as a comment to the function?
Will do.
Quoted 5 lines
But AFAICS there is the
lru_gen_del_folio()callsite that maps it to thePG_activeflag - which in turn gets used byadd_folio()to place the thing back on themax_seqgeneration. So I suppose there is a secondary purpose of the function for remembering the page's rough age for non-reclaim isolation.>Yes, e.g., migration.
Quoted 1 line
It would be good to capture that as well in a comment on the function.
Will do.
Quoted 55 lines
Quoted 25 lines
Quoted 14 lines
Review
Quoted 11 lines
(quoted diff) +11
-0+static inline void lru_gen_update_size(struct lruvec *lruvec, enum lru_list lru, + int zone, long delta) +{ + struct pglist_data *pgdat = lruvec_pgdat(lruvec); + + lockdep_assert_held(&lruvec->lru_lock); + WARN_ON_ONCE(delta != (int)delta); + + __mod_lruvec_state(lruvec, NR_LRU_BASE + lru, delta); + __mod_zone_page_state(&pgdat->node_zones[zone], NR_ZONE_LRU_BASE + lru, delta); +}This is a duplicate of
update_lru_size(), please use that instead.Quoted 2 lines
Yeah technically you don't need the
mem_cgroup_update_lru_size()but that's not worth sweating over, better to keep it simple.I agree we don't need the
mem_cgroup_update_lru_size()-- let me spell out why:
this function is not needed here because it updates the counters used only by the active/inactive lru code, i.e.,get_scan_count().However, we can't reuse
update_lru_size()because MGLRU can trip theWARN_ONCE()inmem_cgroup_update_lru_size().Quoted 7 lines
Unlike lru_zone_size[], lrugen->nr_pages[] is eventually consistent. To move a page to a different generation, the gen counter in
page->flagsis updated first, which doesn't require the LRU lock. The second step, i.e., the update of lrugen->nr_pages[], requires the LRU lock, and it usually isn't done immediately due to batching. Meanwhile, if this page is, for example, isolated, nr_pages[] becomes temporarily unbalanced. And this trips theWARN_ONCE().Good insight.
But in that case, I'd still think it's better to use
update_lru_size()and gate the memcg update on lrugen-enabled, with a short comment saying that lrugen has its own per-cgroup counts already. It's just a bit too error prone to duplicate the stat updates.Even better would be:
static
__always_inline
voidlruvec_add_folio(struct lruvec *lruvec, struct folio *folio){ enum lru_list lru = folio_lru_list(folio); update_lru_size(lruvec, lru, folio_zonenum(folio), folio_nr_pages(folio)); if (lrugen_enabled(lruvec)) lrugen_add_folio(lruvec, folio); else list_add(&folio->lru, &lruvec->lists[lru]); }Quoted 6 lines
But it does mean you'd have to handle unevictable pages. I'm reviewing from the position that mglru is going to supplant the existing reclaim algorithm in the long term, though, so being more comprehensive and eliminating special cases where possible is all-positive, IMO.
Up to you. I'd only insist on reusing
update_lru_size()at least.Will do.
Quoted 19 lines
Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
(quoted diff) +6
-0+static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming) +{ + int gen; + unsigned long old_flags, new_flags; + int type = folio_is_file_lru(folio); + int zone = folio_zonenum(folio);Quoted 8 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+ struct lru_gen_struct *lrugen = &lruvec->lrugen; + + if (folio_test_unevictable(folio) || !lrugen->enabled) + return false;These two checks should be in the callsite and the function should return void. Otherwise you can't understand the callsite without drilling down into lrugen code, even if lrugen is disabled.
Quoted 3 lines
I agree it's a bit of nuisance this way. The alternative is we'd need ifdef or another helper at the call sites because
lrugen->enabledis specific to lrugen.Quoted 8 lines
Coming from memcg, my experience has been that when you have a compile time-optional MM extension like this, you'll sooner or later need a config-independent helper to gate callbacks in generic code. So I think it's a good idea to add one now.
One of these?
lruvec_on_lrugen()SGTM.
Personally I'd reuse
lru_gen_enabled(), by passing NULL/lruvec. But my guess is you wouldn't like it.Quoted 49 lines
lruvec_using_lrugen()lruvec_lrugen_enabled()
lruvec_has_generations():-)Quoted 37 lines
Quoted 32 lines
On that note, I think #1 is reintroducing a problem we have fixed before, which is trashing the workingset with a flood of use-once mmapped pages. It's the classic scenario where LFU beats LRU.
Mapped streaming IO isn't very common, but it does happen. See these commits:
dfc8d636cdb95f7b792d5ba8c9f3b295809c125d31c0569c3b0b6cc8a867ac6665ca081553f7984c645747462435d84c6c6a64269ed49cc3015f753dFrom the changelog:
The used-once mapped file page detection patchset.
It is meant to help workloads with large amounts of shortly used file mappings, like rtorrent hashing a file or git when dealing with loose objects (git gc on a bigger site?).
Right now, the VM activates referenced mapped file pages on first encounter on the inactive list and it takes a full memory cycle to reclaim them again. When those pages dominate memory, the system no longer has a meaningful notion of 'working set' and is required to give up the active list to make reclaim progress. Obviously, this results in rather bad scanning latencies and the wrong pages being reclaimed.
This patch makes the VM be more careful about activating mapped file pages in the first place. The minimum granted lifetime without another memory access becomes an inactive list cycle instead of the full memory cycle, which is more natural given the mentioned loads.
Quoted 4 lines
Translating this to multigen, it seems fresh faults should really start on the second oldest rather than on the youngest generation, to get a second chance but without jeopardizing the workingset if they don't take it.
Quoted 6 lines
This is a good point, and I had worked on a similar idea but failed to measure its benefits. In addition to placing mmapped file pages in older generations, I also tried placing refaulted anon pages in older generations. My conclusion was that the initial LRU positions of NFU pages are not a bottleneck for workloads I've tested. The efficiency of testing/clearing the accessed bit is.
Quoted 2 lines
The concern isn't the scan overhead, but jankiness from the workingset being flooded out by streaming IO.
Yes, MGLRU uses a different approach to solve this problem, and for its approach, the scan overhead is the concern.
MGLRU detects (defines) the working set by scanning the entire memory for each generation, and it counters the flooding by accelerating the creation of generations. IOW, all mapped pages have an equal chance to get scanned, no matter which generation they are in. This is a design difference compared with the active/inactive LRU, which tries to scans the active/inactive lists less/more frequently.
Quoted 7 lines
The concrete usecase at the time was a torrent client hashing a downloaded file and thereby kicking out the desktop environment, which caused jankiness. The hashing didn't benefit from caching - the file wouldn't have fit into RAM anyway - so this was pointless to boot.
Essentially, the tradeoff is this:
Quoted 7 lines
1) If you treat new pages as hot, you accelerate workingset transitions, but on the flipside you risk unnecessary refaults in running applications when those new pages are one-off.
2) If you take new pages with a grain of salt, you protect existing applications better from one-off floods, but risk refaults in NEW application while they're trying to start up.
Agreed.
Quoted 4 lines
There are two arguments for why 2) is preferable:
1) Users are tolerant of cache misses when applications first launch, much less so after they've been running for hours.
Our CUJs (Critical User Journeys) respectfully disagree :)
They are built on the observation that once users have moved onto another tab/app, they are more likely to stay with the new tab/app rather than go back to the old ones. Speaking for myself, this is generally the case.
Quoted 6 lines
2) Workingset transitions (and associated jankiness) are bounded by the amount of RAM you need to repopulate. But streaming IO is bounded by storage, and datasets are routinely several times the amount of RAM. Uncacheable sets in excess of RAM can produce an infinite stream of "new" references; not protecting the workingset from that means longer or even sustained jankiness.
I'd argue the opposite -- we shouldn't risk refaulting fresh hot pages just to accommodate this concrete yet minor use case, especially considering torrent has been given the means (
MADV_SEQUENTIAL) to help itself.I appreciate all your points here. The bottom line is we agree this is a trade off. For what disagree about, we could be both right -- it comes down to what workloads we care about *more*.
To move forward, I propose we look at it from a non-technical POV:
would we want to offer users an alternative trade off so that they can have greater flexibility?Quoted 7 lines
Quoted 3 lines
And some applications are smart enough to leverage
MADV_SEQUENTIAL. In this case, MGLRU does place mmapped file pages in the oldest generation.Yes, it makes sense to optimize when
MADV_SEQUENTIALis requested. But that hint isn't reliably there, so it matters that we don't do poorly when it's missing.Agreed.
Quoted 7 lines
Quoted 2 lines
I have an oversimplified script that uses memcached to mimic a non-streaming workload and fio a (mmapped) streaming workload:
Looking at the paramters and observed behavior, let me say up front that this looks like a useful benchmark, but doesn't capture the scenario I was talking about above.
Quoted 5 lines
For one, the presence of swapping in both kernels suggests that the "streaming IO" component actually has repeat access that could benefit from caching. Second, I would expect memcache is accessing its memory frequently and consistently, and so could withstand workingset challenges from streaming IO better than, say, a desktop environment.
The fio workload is a real streaming workload, but the memcached workload might have been too large to be a typical desktop workload.
More below.
Quoted 6 lines
More on that below.
Quoted 3 lines
1. With
MADV_SEQUENTIAL, the non-streaming workload is about 5 times faster when using MGLRU. Somehow the baseline (rc3) swapped a lot. (It shouldn't, and I haven't figured out why.)Quoted 7 lines
Baseline swaps when there are cache refaults. This is regardless of the hint: you may say you're accessing these pages sequentially, but the refaults say you're reusing them, with a frequency that suggests they might be cacheable. So it tries to cache them.
I'd be curious if that results in fio being faster, or whether it's all just pointless thrashing. Can you share the fio results too?
More below.
Quoted 2 lines
We could patch baseline to prioritize
MADV_SEQUENTIALmore, but...
Quoted 16 lines
Quoted 5 lines
Quoted 3 lines
2. Without
MADV_SEQUENTIAL, the non-streaming workload is about 1 time faster when using MGLRU. Both MGLRU and the baseline swapped a lot....in practice I think this scenario will matter to a lot more users.
I strongly feel we should prioritize what's advertised on a man page over an unspecified (performance) behavior.
Quoted 7 lines
I would again be interested in the fio results.
Quoted 5 lines
MADV_SEQUENTIALnon-streaming ops/sec (memcached)rc3 yes 292k rc3 no 203k rc3+v7 yes 1967k rc3+v7 no 436k
Appending a few notes on the baseline results:
1. Apparently FADV_DONTNEED rejects mmaped pages -- I found no reasons from the man page or the original commit why it should. I propose we remove the page_mapped() check in lru_deactivate_file_fn(). Adding Minchan to see how he thinks about this.
2. For MADV_SEQUENTIAL, I made workingset_refault() check (VM_SEQ_READ | VM_RAND_READ) and return if they are set, and the performance got a lot better (x3).
3. In page_referenced_one(), I think we should also exclude VM_RAND_READ in addition to VM_SEQ_READ.
Quoted 177 lines
Quoted 19 lines
Quoted 9 lines
cat
mmap.sh
modprobe brd rd_nr=2 rd_size=56623104mkswap
/dev/ram0
swapon/dev/ram0mkfs.ext4
/dev/ram1
mount -t ext4/dev/ram1/mntQuoted 9 lines
memtier_benchmark -S /var/run/memcached/memcached.sock -P memcache_binary \ -n allkeys --key-minimum=1 --key-maximum=50000000 --key-pattern=P:P -c 1 \ -t 36 --ratio 1:0 --pipeline 8 -d 2000 # streaming workload: --fadvise_hint=0 disables MADV_SEQUENTIAL fio -name=mglru --numjobs=12 --directory=/mnt --size=4224m --buffered=1 \ --ioengine=mmap --iodepth=128 --iodepth_batch_submit=32 \ --iodepth_batch_complete=32 --rw=read --time_based --ramp_time=10m \ --runtime=180m --group_reporting &Quoted 3 lines
As per above, I think this would be closer to a cacheable workingset than a streaming IO pattern. It depends on total RAM of course, but size=4G and
time_basedshould loop around pretty quickly.The file size here shouldn't matter since fio is smart enough to invalidate page cache before it rewinds (for sequential access):
https://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-invalidate
https://github.com/axboe/fio/blob/master/filesetup.c#L602I think the problem might have been the memory size for memcached was too large (100GB) to be all hot (limited by memory bandwidth).
Quoted 2 lines
Would you mind rerunning with files larger than RAM, to avoid repeat accesses (or at least only repeat with large distances)?
Retested with the same free memory (120GB) for 40GB memcached and 200GB fio.
MADV_SEQUENTIAL FADV_DONTNEED memcached fio rc4 no yes 4716k 232k rc4+v7 no yes 4307k 265k delta -9% +14%MGLRU lost with memcached but won with fio for the same reason: it doesn't have any heuristics to detect the streaming characteristic (and therefore lost with memcached) but relies on faster scanning (and therefore won with fio) to keep the working set in memory.
The baseline didn't swap this time (MGLRU did slightly), but it lost with fio because it had to walk the rmap for each page in the entire 200GB VMA, at least once, even for this streaming workload.
This reflects the design difference I mentioned earlier.
cat
test.sh
modprobe brd rd_nr=1 rd_size=268435456mkfs.ext4
/dev/ram0
mount -t ext4/dev/ram0/mntfallocate -l 40g
/mnt/swapfile
mkswap/mnt/swapfile
swapon/mnt/swapfilefio -name=mglru --numjobs=1 --directory=/mnt --size=204800m \ --buffered=1 --ioengine=mmap --fadvise_hint=0 --iodepth=128 \ --iodepth_batch_submit=32 --iodepth_batch_complete=32 \ --rw=read --time_based --ramp_time=10m --runtime=180m \ --group_reporting & pid=$!sleep 600
# load objects memtier_benchmark -S /var/run/memcached/memcached.sock \ -P memcache_binary -n allkeys --key-minimum=1 \ --key-maximum=20000000 --key-pattern=P:P -c 1 -t 36 \ --ratio 1:0 --pipeline 8 -d 2000 # read objects memtier_benchmark -S /var/run/memcached/memcached.sock \ -P memcache_binary -n allkeys --key-minimum=1 \ --key-maximum=20000000 --key-pattern=R:R -c 1 -t 36 \ --ratio 0:1 --pipeline 8 --randomize --distinct-client-seedkill -INT $pid
Quoted 2 lines
Depending on how hot memcache runs, it may or may not be able to hold onto its workingset.
Agreed.
Quoted 4 lines
Testing interactivity is notoriously hard, but
using a smaller, intermittent workload is probably more representative of overall responsiveness. Let fio ramp until memory is full, then do perf stat -r 10/bin/sh-c 'git shortlog v5.0.. >/dev/null; sleep 1'I'll also check with the downstream maintainers to see if they have heard any complaints about streaming workloads negatively impacting user experience.
Quoted 2 lines
I'll try to reproduce this again too. Back then, that workload gave me a very janky desktop experience, and the patch very obvious relief.
SGTM.
Quoted 32 lines
Quoted 19 lines
Quoted 11 lines
Review void lruvec_add_folio_tail(struct lruvec *lruvec, struct folio *folio) · line 303
Quoted 7 lines
(quoted diff) +3
-0@@ -113,6 +298,9 @@ void lruvec_add_folio_tail(struct lruvec *lruvec, struct folio *folio) { enum lru_list lru = folio_lru_list(folio); + if (lru_gen_add_folio(lruvec, folio, true)) + return; +bool parameters are notoriously hard to follow in the callsite. Can you please add
lru_gen_add_folio_tail()instead and have them use a common helper?I'm not sure -- there are several places like this one. My question is whether we want to do it throughout this patchset. We'd end up with many helpers and duplicate code. E.g., in this file alone, we have two functions taking bool parameters:
lru_gen_add_folio(..., bool reclaiming)lru_gen_del_folio(..., bool reclaiming)Quoted 6 lines
I can't say they are very readable; at least they are very compact right now. My concern is that we might loose the latter without having enough of the former.
Perhaps this is something that we could revisit after you've finished reviewing the entire patchset?
Sure, fair enough.
Quoted 4 lines
Quoted 3 lines
Review void lruvec_add_folio_tail(struct lruvec *lruvec, struct folio *folio) · line 304
Quoted 1 line
(quoted diff) +1
-0+void lru_gen_init_state(struct mem_cgroup *memcg, struct lruvec *lruvec);"state" is what we usually init :) How about
lrugen_init_lruvec()?Quoted 8 lines
Quoted 4 lines
Same story as "file", lol -- this used to be
lru_gen_init_lruvec():
https://lore.kernel.org/linux-mm/[email protected]/Naming is hard. Hopefully we can finalize it this time.
Was that internal feedback? The revisions show this function went through several names, but I don't see reviews requesting those. If they weren't public I'm gonna pretend they didn't happen ;-)
Indeed. I lost track.
Quoted 17 lines
Quoted 4 lines
Quoted 1 line
You can drop the memcg parameter and use
lruvec_memcg().
lruvec_memcg()isn't available yet whenpgdat_init_internals()calls this function becausemem_cgroup_disabled()is initialized afterward.Good catch. That'll
container_of()into garbage. However, we have to assume that somebody's going to try that simplification again, so we should set up the code now to prevent issues.
cgroup_disableparsing is self-contained, so we can pull it ahead in the init sequence. How about this?kernel/cgroup/cgroup.c +0
-0kernel/cgroup/cgroup.c static int __init cgroup_disable(char *str) · line 6469
Quoted 8 lines
kernel/cgroup/cgroup.c +2
-2@@ -6464,9 +6464,9 @@ static int __init cgroup_disable(char *str)break ; } } -return 1 ; +return 0 ; } -__setup ("cgroup_disable=" , cgroup_disable); +early_param ("cgroup_disable" , cgroup_disable);I think
early_param()is still afterpgdat_init_internals(), no?Thanks!
Re: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
Quoted 313 lines · Yu Zhao
.
Quoted 294 lines · Yu Zhao
Quoted 18 lines · Johannes Weiner
Hi Yu,
Quoted 15 lines · Yu Zhao
Quoted 13 lines · Johannes Weiner
Review
Quoted 9 lines
(quoted diff) +9
-0+static inline bool lru_gen_is_active(struct lruvec *lruvec, int gen) +{ + unsigned long max_seq = lruvec->lrugen.max_seq; + + VM_BUG_ON(gen >= MAX_NR_GENS); + + /* see the comment on MIN_NR_GENS */ + return gen == lru_gen_from_seq(max_seq) || gen == lru_gen_from_seq(max_seq - 1); +}I'm still reading the series, so correct me if I'm wrong: the "active" set is split into two generations for the sole purpose of the second-chance policy for fresh faults, right?
Quoted 5 lines
Quoted 3 lines
To be precise, the active/inactive notion on top of generations is just for ABI compatibility, e.g., the counters in
/proc/vmstat. Otherwise, this function wouldn't be needed.Ah! would you mind adding this as a comment to the function?
Will do.
Quoted 5 lines
But AFAICS there is the
lru_gen_del_folio()callsite that maps it to thePG_activeflag - which in turn gets used byadd_folio()to place the thing back on themax_seqgeneration. So I suppose there is a secondary purpose of the function for remembering the page's rough age for non-reclaim isolation.>Yes, e.g., migration.
Quoted 1 line
It would be good to capture that as well in a comment on the function.
Will do.
Quoted 55 lines
Quoted 25 lines
Quoted 14 lines
Review
Quoted 11 lines
(quoted diff) +11
-0+static inline void lru_gen_update_size(struct lruvec *lruvec, enum lru_list lru, + int zone, long delta) +{ + struct pglist_data *pgdat = lruvec_pgdat(lruvec); + + lockdep_assert_held(&lruvec->lru_lock); + WARN_ON_ONCE(delta != (int)delta); + + __mod_lruvec_state(lruvec, NR_LRU_BASE + lru, delta); + __mod_zone_page_state(&pgdat->node_zones[zone], NR_ZONE_LRU_BASE + lru, delta); +}This is a duplicate of
update_lru_size(), please use that instead.Quoted 2 lines
Yeah technically you don't need the
mem_cgroup_update_lru_size()but that's not worth sweating over, better to keep it simple.I agree we don't need the
mem_cgroup_update_lru_size()-- let me spell out why:
this function is not needed here because it updates the counters used only by the active/inactive lru code, i.e.,get_scan_count().However, we can't reuse
update_lru_size()because MGLRU can trip theWARN_ONCE()inmem_cgroup_update_lru_size().Quoted 7 lines
Unlike lru_zone_size[], lrugen->nr_pages[] is eventually consistent. To move a page to a different generation, the gen counter in
page->flagsis updated first, which doesn't require the LRU lock. The second step, i.e., the update of lrugen->nr_pages[], requires the LRU lock, and it usually isn't done immediately due to batching. Meanwhile, if this page is, for example, isolated, nr_pages[] becomes temporarily unbalanced. And this trips theWARN_ONCE().Good insight.
But in that case, I'd still think it's better to use
update_lru_size()and gate the memcg update on lrugen-enabled, with a short comment saying that lrugen has its own per-cgroup counts already. It's just a bit too error prone to duplicate the stat updates.Even better would be:
static
__always_inline
voidlruvec_add_folio(struct lruvec *lruvec, struct folio *folio){ enum lru_list lru = folio_lru_list(folio); update_lru_size(lruvec, lru, folio_zonenum(folio), folio_nr_pages(folio)); if (lrugen_enabled(lruvec)) lrugen_add_folio(lruvec, folio); else list_add(&folio->lru, &lruvec->lists[lru]); }Quoted 6 lines
But it does mean you'd have to handle unevictable pages. I'm reviewing from the position that mglru is going to supplant the existing reclaim algorithm in the long term, though, so being more comprehensive and eliminating special cases where possible is all-positive, IMO.
Up to you. I'd only insist on reusing
update_lru_size()at least.Will do.
Quoted 19 lines
Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
(quoted diff) +6
-0+static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming) +{ + int gen; + unsigned long old_flags, new_flags; + int type = folio_is_file_lru(folio); + int zone = folio_zonenum(folio);Quoted 8 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+ struct lru_gen_struct *lrugen = &lruvec->lrugen; + + if (folio_test_unevictable(folio) || !lrugen->enabled) + return false;These two checks should be in the callsite and the function should return void. Otherwise you can't understand the callsite without drilling down into lrugen code, even if lrugen is disabled.
Quoted 3 lines
I agree it's a bit of nuisance this way. The alternative is we'd need ifdef or another helper at the call sites because
lrugen->enabledis specific to lrugen.Quoted 8 lines
Coming from memcg, my experience has been that when you have a compile time-optional MM extension like this, you'll sooner or later need a config-independent helper to gate callbacks in generic code. So I think it's a good idea to add one now.
One of these?
lruvec_on_lrugen()SGTM.
Personally I'd reuse
lru_gen_enabled(), by passing NULL/lruvec. But my guess is you wouldn't like it.Quoted 49 lines
lruvec_using_lrugen()lruvec_lrugen_enabled()
lruvec_has_generations():-)Quoted 37 lines
Quoted 32 lines
On that note, I think #1 is reintroducing a problem we have fixed before, which is trashing the workingset with a flood of use-once mmapped pages. It's the classic scenario where LFU beats LRU.
Mapped streaming IO isn't very common, but it does happen. See these commits:
dfc8d636cdb95f7b792d5ba8c9f3b295809c125d31c0569c3b0b6cc8a867ac6665ca081553f7984c645747462435d84c6c6a64269ed49cc3015f753dFrom the changelog:
The used-once mapped file page detection patchset.
It is meant to help workloads with large amounts of shortly used file mappings, like rtorrent hashing a file or git when dealing with loose objects (git gc on a bigger site?).
Right now, the VM activates referenced mapped file pages on first encounter on the inactive list and it takes a full memory cycle to reclaim them again. When those pages dominate memory, the system no longer has a meaningful notion of 'working set' and is required to give up the active list to make reclaim progress. Obviously, this results in rather bad scanning latencies and the wrong pages being reclaimed.
This patch makes the VM be more careful about activating mapped file pages in the first place. The minimum granted lifetime without another memory access becomes an inactive list cycle instead of the full memory cycle, which is more natural given the mentioned loads.
Quoted 4 lines
Translating this to multigen, it seems fresh faults should really start on the second oldest rather than on the youngest generation, to get a second chance but without jeopardizing the workingset if they don't take it.
Quoted 6 lines
This is a good point, and I had worked on a similar idea but failed to measure its benefits. In addition to placing mmapped file pages in older generations, I also tried placing refaulted anon pages in older generations. My conclusion was that the initial LRU positions of NFU pages are not a bottleneck for workloads I've tested. The efficiency of testing/clearing the accessed bit is.
Quoted 2 lines
The concern isn't the scan overhead, but jankiness from the workingset being flooded out by streaming IO.
Yes, MGLRU uses a different approach to solve this problem, and for its approach, the scan overhead is the concern.
MGLRU detects (defines) the working set by scanning the entire memory for each generation, and it counters the flooding by accelerating the creation of generations. IOW, all mapped pages have an equal chance to get scanned, no matter which generation they are in. This is a design difference compared with the active/inactive LRU, which tries to scans the active/inactive lists less/more frequently.
Quoted 7 lines
The concrete usecase at the time was a torrent client hashing a downloaded file and thereby kicking out the desktop environment, which caused jankiness. The hashing didn't benefit from caching - the file wouldn't have fit into RAM anyway - so this was pointless to boot.
Essentially, the tradeoff is this:
Quoted 7 lines
1) If you treat new pages as hot, you accelerate workingset transitions, but on the flipside you risk unnecessary refaults in running applications when those new pages are one-off.
2) If you take new pages with a grain of salt, you protect existing applications better from one-off floods, but risk refaults in NEW application while they're trying to start up.
Agreed.
Quoted 4 lines
There are two arguments for why 2) is preferable:
1) Users are tolerant of cache misses when applications first launch, much less so after they've been running for hours.
Our CUJs (Critical User Journeys) respectfully disagree :)
They are built on the observation that once users have moved onto another tab/app, they are more likely to stay with the new tab/app rather than go back to the old ones. Speaking for myself, this is generally the case.
Quoted 6 lines
2) Workingset transitions (and associated jankiness) are bounded by the amount of RAM you need to repopulate. But streaming IO is bounded by storage, and datasets are routinely several times the amount of RAM. Uncacheable sets in excess of RAM can produce an infinite stream of "new" references; not protecting the workingset from that means longer or even sustained jankiness.
I'd argue the opposite -- we shouldn't risk refaulting fresh hot pages just to accommodate this concrete yet minor use case, especially considering torrent has been given the means (
MADV_SEQUENTIAL) to help itself.I appreciate all your points here. The bottom line is we agree this is a trade off. For what disagree about, we could be both right -- it comes down to what workloads we care about *more*.
To move forward, I propose we look at it from a non-technical POV:
would we want to offer users an alternative trade off so that they can have greater flexibility?Quoted 7 lines
Quoted 3 lines
And some applications are smart enough to leverage
MADV_SEQUENTIAL. In this case, MGLRU does place mmapped file pages in the oldest generation.Yes, it makes sense to optimize when
MADV_SEQUENTIALis requested. But that hint isn't reliably there, so it matters that we don't do poorly when it's missing.Agreed.
Quoted 7 lines
Quoted 2 lines
I have an oversimplified script that uses memcached to mimic a non-streaming workload and fio a (mmapped) streaming workload:
Looking at the paramters and observed behavior, let me say up front that this looks like a useful benchmark, but doesn't capture the scenario I was talking about above.
Quoted 5 lines
For one, the presence of swapping in both kernels suggests that the "streaming IO" component actually has repeat access that could benefit from caching. Second, I would expect memcache is accessing its memory frequently and consistently, and so could withstand workingset challenges from streaming IO better than, say, a desktop environment.
The fio workload is a real streaming workload, but the memcached workload might have been too large to be a typical desktop workload.
More below.
Quoted 6 lines
More on that below.
Quoted 3 lines
1. With
MADV_SEQUENTIAL, the non-streaming workload is about 5 times faster when using MGLRU. Somehow the baseline (rc3) swapped a lot. (It shouldn't, and I haven't figured out why.)Quoted 7 lines
Baseline swaps when there are cache refaults. This is regardless of the hint: you may say you're accessing these pages sequentially, but the refaults say you're reusing them, with a frequency that suggests they might be cacheable. So it tries to cache them.
I'd be curious if that results in fio being faster, or whether it's all just pointless thrashing. Can you share the fio results too?
More below.
Quoted 2 lines
We could patch baseline to prioritize
MADV_SEQUENTIALmore, but...Quoted 16 lines
Quoted 5 lines
Quoted 3 lines
2. Without
MADV_SEQUENTIAL, the non-streaming workload is about 1 time faster when using MGLRU. Both MGLRU and the baseline swapped a lot....in practice I think this scenario will matter to a lot more users.
I strongly feel we should prioritize what's advertised on a man page over an unspecified (performance) behavior.
Quoted 7 lines
I would again be interested in the fio results.
Quoted 5 lines
MADV_SEQUENTIALnon-streaming ops/sec (memcached)rc3 yes 292k rc3 no 203k rc3+v7 yes 1967k rc3+v7 no 436k
Quoted 5 lines
Appending a few notes on the baseline results:
1. ApparentlyFADV_DONTNEEDrejects mmaped pages -- I found no reasons from the man page or the original commit why it should. I propose we remove thepage_mapped()check inlru_deactivate_file_fn(). Adding Minchan to see how he thinks about this.
Hi Yu,
It's quite old code. I already forgot all the details. Maybe, I wanted t minimize behavior changes since invalidate_inode_page already fiter mapped pages out.
I don't have any strong reason not to move mapped pages for deactivation but if we do, it would be better to move them into head of inactive list instead of tail to give a promote chance since other processes are still mappping the page in their address space.
Thanks.
Johannes Weiner
maintainerRe: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
In reply to Yu ZhaoHi Yu,
Quoted 25 lines · Yu Zhao
Quoted 16 lines · Johannes Weiner
Quoted 15 lines · Yu Zhao
Quoted 13 lines · Johannes Weiner
Review
Quoted 9 lines
(quoted diff) +9
-0+static inline bool lru_gen_is_active(struct lruvec *lruvec, int gen) +{ + unsigned long max_seq = lruvec->lrugen.max_seq; + + VM_BUG_ON(gen >= MAX_NR_GENS); + + /* see the comment on MIN_NR_GENS */ + return gen == lru_gen_from_seq(max_seq) || gen == lru_gen_from_seq(max_seq - 1); +}I'm still reading the series, so correct me if I'm wrong: the "active" set is split into two generations for the sole purpose of the second-chance policy for fresh faults, right?
Quoted 5 lines
Quoted 3 lines
To be precise, the active/inactive notion on top of generations is just for ABI compatibility, e.g., the counters in
/proc/vmstat. Otherwise, this function wouldn't be needed.Ah! would you mind adding this as a comment to the function?
Will do.
Quoted 7 lines
Quoted 5 lines
But AFAICS there is the
lru_gen_del_folio()callsite that maps it to thePG_activeflag - which in turn gets used byadd_folio()to place the thing back on themax_seqgeneration. So I suppose there is a secondary purpose of the function for remembering the page's rough age for non-reclaim isolation.>Yes, e.g., migration.
Ok, thanks for clarifying. That should also be in the comment.
On scan resistance:
Quoted 35 lines
Quoted 2 lines
The concern isn't the scan overhead, but jankiness from the workingset being flooded out by streaming IO.
Yes, MGLRU uses a different approach to solve this problem, and for its approach, the scan overhead is the concern.
MGLRU detects (defines) the working set by scanning the entire memory for each generation, and it counters the flooding by accelerating the creation of generations. IOW, all mapped pages have an equal chance to get scanned, no matter which generation they are in. This is a design difference compared with the active/inactive LRU, which tries to scans the active/inactive lists less/more frequently.
Quoted 7 lines
The concrete usecase at the time was a torrent client hashing a downloaded file and thereby kicking out the desktop environment, which caused jankiness. The hashing didn't benefit from caching - the file wouldn't have fit into RAM anyway - so this was pointless to boot.
Essentially, the tradeoff is this:
Quoted 7 lines
1) If you treat new pages as hot, you accelerate workingset transitions, but on the flipside you risk unnecessary refaults in running applications when those new pages are one-off.
2) If you take new pages with a grain of salt, you protect existing applications better from one-off floods, but risk refaults in NEW application while they're trying to start up.
Agreed.
Quoted 4 lines
There are two arguments for why 2) is preferable:
1) Users are tolerant of cache misses when applications first launch, much less so after they've been running for hours.
Quoted 6 lines
Our CUJs (Critical User Journeys) respectfully disagree :)
They are built on the observation that once users have moved onto another tab/app, they are more likely to stay with the new tab/app rather than go back to the old ones. Speaking for myself, this is generally the case.
That's in line with what I said. Where is the disagreement?
Quoted 7 lines
Quoted 6 lines
2) Workingset transitions (and associated jankiness) are bounded by the amount of RAM you need to repopulate. But streaming IO is bounded by storage, and datasets are routinely several times the amount of RAM. Uncacheable sets in excess of RAM can produce an infinite stream of "new" references; not protecting the workingset from that means longer or even sustained jankiness.
Quoted 8 lines
I'd argue the opposite -- we shouldn't risk refaulting fresh hot pages just to accommodate this concrete yet minor use case, especially considering torrent has been given the means (
MADV_SEQUENTIAL) to help itself.I appreciate all your points here. The bottom line is we agree this is a trade off. For what disagree about, we could be both right -- it comes down to what workloads we care about *more*.
It's a straight-forward question: How does MGLRU avoid cache pollution from scans?
Your answer above seems to be "it just does". Your answer here seems to be "it doesn't, but it doesn't matter". Forgive me if I'm misreading what you're saying.
But it's not a minor concern. Read the motivation behind any modern cache algorithm - ARC, LIRS, Clock-Pro, LRU-K, 2Q - and scan resistance is the reason for why they all exist in the first place.
"The LRU-K algorithm surpasses conventional buffering algorithms
in discriminating between frequently and infrequently referenced pages."
- The LRU-K page replacement algorithm for database disk buffering, O'Neil et al, 1993
"Although LRU replacement policy has been commonly used in the
buffer cache management, it is well known for its inability to cope with access patterns with weak locality."
- LIRS: an efficient low inter-reference recency set replacement policy to improve buffer cache performance, Jiang, Zhang, 2002
"The self-tuning, low-overhead, scan-resistant adaptive
replacement cache algorithm outperforms the least-recently-used algorithm by dynamically responding to changing access patterns and continually balancing between workload recency and frequency features."
- Outperforming LRU with an adaptive replacement cache algorithm, Megiddo, Modha, 2004
"Over the last three decades, the inability of LRU as well as
CLOCK to handle weak locality accesses has become increasingly serious, and an effective fix becomes increasingly desirable.
- CLOCK-Pro: An Effective Improvement of the CLOCK Replacement, Jiang et al, 2005
We can't rely on MADV_SEQUENTIAL alone. Not all accesses know in advance that they'll be one-off; it can be a group of uncoordinated tasks causing the pattern etc.
This is a pretty fundamental issue. It would be good to get a more satisfying answer on this.
Quoted 12 lines
Quoted 12 lines
Quoted 4 lines
Quoted 1 line
You can drop the memcg parameter and use
lruvec_memcg().
lruvec_memcg()isn't available yet whenpgdat_init_internals()calls this function becausemem_cgroup_disabled()is initialized afterward.Good catch. That'll
container_of()into garbage. However, we have to assume that somebody's going to try that simplification again, so we should set up the code now to prevent issues.
cgroup_disableparsing is self-contained, so we can pull it ahead in the init sequence. How about this?
Quoted 15 lines
kernel/cgroup/cgroup.c static int __init cgroup_disable(char *str) · line 6469
Quoted 13 lines
kernel/cgroup/cgroup.c +2
-2@@ -6464,9 +6464,9 @@ static int __init cgroup_disable(char *str)break ; } } -return 1 ; +return 0 ; } -__setup ("cgroup_disable=" , cgroup_disable); +early_param ("cgroup_disable" , cgroup_disable);I think
early_param()is still afterpgdat_init_internals(), no?
It's called twice for some reason, but AFAICS the first one is always called before pgdat_init_internals():
start_kernel()
setup_arch()
parse_early_param()
x86_init.paging.pagetable_init();
paging_init()
zone_sizes_init()
free_area_init()
free_area_init_node()
free_area_init_core()
pgdat_init_internals() parse_early_param()
It's the same/similar for arm, sparc and mips.
Re: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
Quoted 35 lines · Johannes Weiner
Hi Yu,
Quoted 31 lines · Yu Zhao
Quoted 16 lines · Johannes Weiner
Quoted 15 lines · Yu Zhao
Quoted 13 lines · Johannes Weiner
Review
Quoted 9 lines
(quoted diff) +9
-0+static inline bool lru_gen_is_active(struct lruvec *lruvec, int gen) +{ + unsigned long max_seq = lruvec->lrugen.max_seq; + + VM_BUG_ON(gen >= MAX_NR_GENS); + + /* see the comment on MIN_NR_GENS */ + return gen == lru_gen_from_seq(max_seq) || gen == lru_gen_from_seq(max_seq - 1); +}I'm still reading the series, so correct me if I'm wrong: the "active" set is split into two generations for the sole purpose of the second-chance policy for fresh faults, right?
Quoted 5 lines
Quoted 3 lines
To be precise, the active/inactive notion on top of generations is just for ABI compatibility, e.g., the counters in
/proc/vmstat. Otherwise, this function wouldn't be needed.Ah! would you mind adding this as a comment to the function?
Will do.
Quoted 5 lines
But AFAICS there is the
lru_gen_del_folio()callsite that maps it to thePG_activeflag - which in turn gets used byadd_folio()to place the thing back on themax_seqgeneration. So I suppose there is a secondary purpose of the function for remembering the page's rough age for non-reclaim isolation.>
Quoted 3 lines
Quoted 1 line
Yes, e.g., migration.
Ok, thanks for clarifying. That should also be in the comment.
Thanks. Will do.
Quoted 37 lines
On scan resistance:
Quoted 35 lines
Quoted 2 lines
The concern isn't the scan overhead, but jankiness from the workingset being flooded out by streaming IO.
Yes, MGLRU uses a different approach to solve this problem, and for its approach, the scan overhead is the concern.
MGLRU detects (defines) the working set by scanning the entire memory for each generation, and it counters the flooding by accelerating the creation of generations. IOW, all mapped pages have an equal chance to get scanned, no matter which generation they are in. This is a design difference compared with the active/inactive LRU, which tries to scans the active/inactive lists less/more frequently.
Quoted 7 lines
The concrete usecase at the time was a torrent client hashing a downloaded file and thereby kicking out the desktop environment, which caused jankiness. The hashing didn't benefit from caching - the file wouldn't have fit into RAM anyway - so this was pointless to boot.
Essentially, the tradeoff is this:
Quoted 7 lines
1) If you treat new pages as hot, you accelerate workingset transitions, but on the flipside you risk unnecessary refaults in running applications when those new pages are one-off.
2) If you take new pages with a grain of salt, you protect existing applications better from one-off floods, but risk refaults in NEW application while they're trying to start up.
Agreed.
Quoted 4 lines
There are two arguments for why 2) is preferable:
1) Users are tolerant of cache misses when applications first launch, much less so after they've been running for hours.
Quoted 8 lines
Quoted 6 lines
Our CUJs (Critical User Journeys) respectfully disagree :)
They are built on the observation that once users have moved onto another tab/app, they are more likely to stay with the new tab/app rather than go back to the old ones. Speaking for myself, this is generally the case.
That's in line with what I said. Where is the disagreement?
Probably I've misinterpreted what you meant. The reasoning behind 1) sounds to me like:
Cache misses of existing apps are more detrimental to user experience, and therefore we choose to sacrifice the performance of newly launched apps to avoid flooding.
My argument is that (phone/laptop/desktop) users usually care more about the performance of newly launched apps -- this sounds a contradiction of what 1) says -- and therefore sacrificing the performance of newly launched apps is not generally a good idea.
Quoted 16 lines
Quoted 7 lines
Quoted 6 lines
2) Workingset transitions (and associated jankiness) are bounded by the amount of RAM you need to repopulate. But streaming IO is bounded by storage, and datasets are routinely several times the amount of RAM. Uncacheable sets in excess of RAM can produce an infinite stream of "new" references; not protecting the workingset from that means longer or even sustained jankiness.
Quoted 8 lines
I'd argue the opposite -- we shouldn't risk refaulting fresh hot pages just to accommodate this concrete yet minor use case, especially considering torrent has been given the means (
MADV_SEQUENTIAL) to help itself.I appreciate all your points here. The bottom line is we agree this is a trade off. For what disagree about, we could be both right -- it comes down to what workloads we care about *more*.
Quoted 6 lines
It's a straight-forward question: How does MGLRU avoid cache pollution from scans?
Your answer above seems to be "it just does". Your answer here seems to be "it doesn't, but it doesn't matter". Forgive me if I'm misreading what you're saying.
This might have gotten lost. At the beginning, I explained:
Quoted 9 lines
Quoted 9 lines
Yes, MGLRU uses a different approach to solve this problem, and for its approach, the scan overhead is the concern.
MGLRU detects (defines) the working set by scanning the entire memory for each generation, and it counters the flooding by accelerating the creation of generations. IOW, all mapped pages have an equal chance to get scanned, no matter which generation they are in. This is a design difference compared with the active/inactive LRU, which tries to scan the active/inactive lists less/more frequently.
To summarize: MGLRU counteracts flooding by shortening the time interval that defines an entire working set. In contrast, the active/inactive LRU uses a longer time interval for the active (established working set) and a shorter one for the inactive (flood or new working set).
Quoted 1 line
But it's not a minor concern.
This depends on the POV :)
For what workloads I care about more, i.e., the majority of phone/laptop/desktop users and top open-source memory hogs running on servers, I've heard no complaints (yet).
Quoted 3 lines
Read the motivation behind any modern
cache algorithm - ARC, LIRS, Clock-Pro, LRU-K, 2Q - and scan resistance is the reason for why they all exist in the first place.
I agree that flooding is a concern (major from your POV; minor from my POV). I assume that you agree that the solution is always a tradeoff. I'm merely suggesting both tradeoffs I summarized above have their merits and we shouldn't put all eggs in one bucket.
Quoted 28 lines
"The LRU-K algorithm surpasses conventional buffering algorithms
in discriminating between frequently and infrequently referenced pages."- The LRU-K page replacement algorithm for database disk buffering, O'Neil et al, 1993
"Although LRU replacement policy has been commonly used in the
buffer cache management, it is well known for its inability to cope with access patterns with weak locality."- LIRS: an efficient low inter-reference recency set replacement policy to improve buffer cache performance, Jiang, Zhang, 2002
"The self-tuning, low-overhead, scan-resistant adaptive
replacement cache algorithm outperforms the least-recently-used algorithm by dynamically responding to changing access patterns and continually balancing between workload recency and frequency features."- Outperforming LRU with an adaptive replacement cache algorithm, Megiddo, Modha, 2004
"Over the last three decades, the inability of LRU as well as
CLOCK to handle weak locality accesses has become increasingly serious, and an effective fix becomes increasingly desirable.
Quoted 5 lines
- CLOCK-Pro: An Effective Improvement of the CLOCK Replacement, Jiang et al, 2005
We can't rely on
MADV_SEQUENTIALalone.
Agreed.
Quoted 3 lines
Not all accesses know in
advance that they'll be one-off; it can be a group of uncoordinated tasks causing the pattern etc.
For what we know, MADV_SEQUENTIAL can be a solution. There are things we don't know, hence the "agreed" above :)
Quoted 1 line
This is a pretty fundamental issue.
Agreed.
My points are:
1. The practice value of this fundamental issue (minor or major).
2. The merits of different tradeoffs (better or worse).
IMO, both depend on the POV.
Quoted 2 lines
It would be good to get a more
satisfying answer on this.
Agreed.
Quoted 41 lines
Quoted 12 lines
Quoted 12 lines
Quoted 4 lines
Quoted 1 line
You can drop the memcg parameter and use
lruvec_memcg().
lruvec_memcg()isn't available yet whenpgdat_init_internals()calls this function becausemem_cgroup_disabled()is initialized afterward.Good catch. That'll
container_of()into garbage. However, we have to assume that somebody's going to try that simplification again, so we should set up the code now to prevent issues.
cgroup_disableparsing is self-contained, so we can pull it ahead in the init sequence. How about this?Quoted 15 lines
kernel/cgroup/cgroup.c static int __init cgroup_disable(char *str) · line 6469
Quoted 13 lines
kernel/cgroup/cgroup.c +2
-2@@ -6464,9 +6464,9 @@ static int __init cgroup_disable(char *str)break ; } } -return 1 ; +return 0 ; } -__setup ("cgroup_disable=" , cgroup_disable); +early_param ("cgroup_disable" , cgroup_disable);I think
early_param()is still afterpgdat_init_internals(), no?It's called twice for some reason, but AFAICS the first one is always called before
pgdat_init_internals():
start_kernel()
setup_arch()parse_early_param() x86_init.paging.pagetable_init(); paging_init() zone_sizes_init() free_area_init() free_area_init_node() free_area_init_core() pgdat_init_internals()
Quoted 3 lines
parse_early_param()It's the same/similar for arm, sparc and mips.
Thanks for checking. But I'd rather live with an additional parameter than risk breaking some archs.
Johannes Weiner
maintainerRe: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
Quoted 48 lines · Yu Zhao
Quoted 47 lines · Johannes Weiner
Quoted 15 lines · Yu Zhao
Quoted 14 lines · Johannes Weiner
Quoted 5 lines · Yu Zhao
Quoted 1 line · Johannes Weiner
You can drop the memcg parameter and use
lruvec_memcg().
lruvec_memcg()isn't available yet whenpgdat_init_internals()calls this function becausemem_cgroup_disabled()is initialized afterward.Good catch. That'll
container_of()into garbage. However, we have to assume that somebody's going to try that simplification again, so we should set up the code now to prevent issues.
cgroup_disableparsing is self-contained, so we can pull it ahead in the init sequence. How about this?Quoted 15 lines
kernel/cgroup/cgroup.c static int __init cgroup_disable(char *str) · line 6469
Quoted 13 lines
kernel/cgroup/cgroup.c +2
-2@@ -6464,9 +6464,9 @@ static int __init cgroup_disable(char *str)break ; } } -return 1 ; +return 0 ; } -__setup ("cgroup_disable=" , cgroup_disable); +early_param ("cgroup_disable" , cgroup_disable);I think
early_param()is still afterpgdat_init_internals(), no?It's called twice for some reason, but AFAICS the first one is always called before
pgdat_init_internals():
start_kernel()
setup_arch()parse_early_param() x86_init.paging.pagetable_init(); paging_init() zone_sizes_init() free_area_init() free_area_init_node() free_area_init_core() pgdat_init_internals()
parse_early_param()
Quoted 4 lines
Quoted 1 line
It's the same/similar for arm, sparc and mips.
Thanks for checking. But I'd rather live with an additional parameter than risk breaking some archs.
As per above, somebody is going to try to make that simplification again in the future. It doesn't make a lot of sense to have a reviewer trip over it, have a discussion about just how subtle this dependency is, and then still leave it in for others. parse_early_param() is documented to be called by arch code early on, there isn't a good reason to mistrust our own codebase like that. And special-casing this situation just complicates maintainability and hackability.
Please just fix the ordering and use lruvec_memcg(), thanks.
Re: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
In reply to Yu ZhaoQuoted 19 lines · Yu Zhao
On Thu, Feb 10, 2022 at 03:41:57PM -0500, Johannes Weiner wrote:
Thanks for reviewing.
Quoted 5 lines
Quoted 5 lines
include/linux/mm_inline.h +5
-0+static inline bool lru_gen_is_active (struct lruvec *lruvec,int gen) +{ +unsigned long max_seq = lruvec->lrugen.max_seq; + +VM_BUG_ON (gen >=MAX_NR_GENS );Quoted 8 lines
include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 120
Quoted 4 lines
include/linux/mm_inline.h +4
-0+ +/* see the comment on MIN_NR_GENS */ +return gen ==lru_gen_from_seq (max_seq) || gen ==lru_gen_from_seq (max_seq -1 ); +}I'm still reading the series, so correct me if I'm wrong: the "active" set is split into two generations for the sole purpose of the second-chance policy for fresh faults, right?
Quoted 3 lines
To be precise, the active/inactive notion on top of generations is just for ABI compatibility, e.g., the counters in
/proc/vmstat. Otherwise, this function wouldn't be needed.
Hi Yu,
I am still quite confused as i am seeing both active/inactive and lru_gen. eg:
root@ubuntu:~# cat /proc/vmstat | grep active nr_zone_inactive_anon 22797 nr_zone_active_anon 578405 nr_zone_inactive_file 0 nr_zone_active_file 4156 nr_inactive_anon 22800 nr_active_anon 578574 nr_inactive_file 0 nr_active_file 4215
and:
root@ubuntu:~# cat /sys//kernel/debug/lru_gen
...
memcg 36 /user.slice/user-0.slice/[email protected]
node 0
20 18820 22 0
21 7452 0 0
22 7448 0 0
memcg 33 /user.slice/user-0.slice/[email protected]/app.slice
node 0
0 2171452 0 0
1 2171452 0 0
2 2171452 0 0
3 2171452 0 0
memcg 37 /user.slice/user-0.slice/session-1.scope
node 0
42 51804 102127 0
43 18840 275622 0
44 16104 216805 1Does it mean one page could be in both one of the generations and one of the active/inactive lists?
Do we have some mapping relationship between active/inactive lists with generations?
Quoted 195 lines
Quoted 6 lines
If so, it'd be better to have the comment here instead of down by
MIN_NR_GENS. This is the place that defines what "active" is, so this is where the reader asks what it means and what it implies. The definition ofMIN_NR_GENScan be briefer: "need at least two for second chance, seelru_gen_is_active()for details".This could be understood this way. It'd be more appropriate to see this function as an auxiliary and
MIN_NR_GENSas something fundamental. Therefore the former should refer to the latter. Specifically, the "see the comment onMIN_NR_GENS" refers to this part:
And to be compatible with the active/inactive LRU, these two generations are mapped to the active; the rest of generations, if they exist, are mapped to the inactive.Quoted 14 lines
include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 132
Quoted 11 lines
include/linux/mm_inline.h +11
-0+static inline void lru_gen_update_size (struct lruvec *lruvec,enum lru_list lru, +int zone,long delta) +{ +struct pglist_data *pgdat =lruvec_pgdat (lruvec); + +lockdep_assert_held (&lruvec->lru_lock); +WARN_ON_ONCE (delta != (int )delta); + +__mod_lruvec_state (lruvec,NR_LRU_BASE + lru, delta); +__mod_zone_page_state (&pgdat->node_zones[zone],NR_ZONE_LRU_BASE + lru, delta); +}This is a duplicate of
update_lru_size(), please use that instead.Quoted 2 lines
Yeah technically you don't need the
mem_cgroup_update_lru_size()but that's not worth sweating over, better to keep it simple.I agree we don't need the
mem_cgroup_update_lru_size()-- let me spell out why:
this function is not needed here because it updates the counters used only by the active/inactive lru code, i.e.,get_scan_count().However, we can't reuse
update_lru_size()because MGLRU can trip theWARN_ONCE()inmem_cgroup_update_lru_size().Unlike lru_zone_size[], lrugen->nr_pages[] is eventually consistent. To move a page to a different generation, the gen counter in
page->flagsis updated first, which doesn't require the LRU lock. The second step, i.e., the update of lrugen->nr_pages[], requires the LRU lock, and it usually isn't done immediately due to batching. Meanwhile, if this page is, for example, isolated, nr_pages[] becomes temporarily unbalanced. And this trips theWARN_ONCE().<snipped>
Quoted 8 lines
/* Promotion */include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 173
Quoted 6 lines
include/linux/mm_inline.h +6
-0+if (!lru_gen_is_active (lruvec, old_gen) &&lru_gen_is_active (lruvec, new_gen)) { +lru_gen_update_size (lruvec, lru, zone, -delta); +lru_gen_update_size (lruvec, lru +LRU_ACTIVE , zone, delta); + } + +/* Promotion is legit while a page is on an LRU list, but demotion isn't. */ Quoted 8 lines
/* Demotion happens during aging when pages are isolated, never on-LRU */Review
Quoted 2 lines
(quoted diff) +2
-0+ VM_BUG_ON(lru_gen_is_active(lruvec, old_gen) && !lru_gen_is_active(lruvec, new_gen)); +}On that note, please move introduction of the promotion and demotion bits to the next patch. They aren't used here yet, and I spent some time jumping around patches to verify the promotion callers and confirm the validy of the
BUG_ON.Will do.
Quoted 6 lines
Quoted 6 lines
include/linux/mm_inline.h +6
-0+static inline bool lru_gen_add_folio (struct lruvec *lruvec,struct folio *folio,bool reclaiming) +{ +int gen; +unsigned long old_flags, new_flags; +int type =folio_is_file_lru (folio); +int zone =folio_zonenum (folio);Quoted 8 lines
include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 186
Quoted 4 lines
include/linux/mm_inline.h +4
-0+struct lru_gen_struct *lrugen = &lruvec->lrugen; + +if (folio_test_unevictable (folio) || !lrugen->enabled) +return false;These two checks should be in the callsite and the function should return void. Otherwise you can't understand the callsite without drilling down into lrugen code, even if lrugen is disabled.
I agree it's a bit of nuisance this way. The alternative is we'd need ifdef or another helper at the call sites because
lrugen->enabledis specific to lrugen.Quoted 8 lines
include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 190
Quoted 4 lines
include/linux/mm_inline.h +4
-0+/* +* There are three common cases for this page: +* 1) If it shouldn't be evicted, e.g., it was just faulted in, add it +* to the youngest generation. "shouldn't be evicted" makes it sound like mlock. But they should just be evicted last, right? Maybe:
Quoted 11 lines
/*
* Pages start in different generations depending on
* advance knowledge we have about their hotness and
* evictability:
*
* 1. Already active pages start out youngest. This can be
* fresh faults, or refaults of previously hot pages.
* 2. Cold pages that require writeback before becoming
* evictable start on the second oldest generation.
* 3. Everything else (clean, cold) starts old. */Will do.
Quoted 32 lines
On that note, I think #1 is reintroducing a problem we have fixed before, which is trashing the workingset with a flood of use-once mmapped pages. It's the classic scenario where LFU beats LRU.
Mapped streaming IO isn't very common, but it does happen. See these commits:
dfc8d636cdb95f7b792d5ba8c9f3b295809c125d31c0569c3b0b6cc8a867ac6665ca081553f7984c645747462435d84c6c6a64269ed49cc3015f753dFrom the changelog:
The used-once mapped file page detection patchset.
It is meant to help workloads with large amounts of shortly used file mappings, like rtorrent hashing a file or git when dealing with loose objects (git gc on a bigger site?).
Right now, the VM activates referenced mapped file pages on first encounter on the inactive list and it takes a full memory cycle to reclaim them again. When those pages dominate memory, the system no longer has a meaningful notion of 'working set' and is required to give up the active list to make reclaim progress. Obviously, this results in rather bad scanning latencies and the wrong pages being reclaimed.
This patch makes the VM be more careful about activating mapped file pages in the first place. The minimum granted lifetime without another memory access becomes an inactive list cycle instead of the full memory cycle, which is more natural given the mentioned loads.
Quoted 4 lines
Translating this to multigen, it seems fresh faults should really start on the second oldest rather than on the youngest generation, to get a second chance but without jeopardizing the workingset if they don't take it.
This is a good point, and I had worked on a similar idea but failed to measure its benefits. In addition to placing mmapped file pages in older generations, I also tried placing refaulted anon pages in older generations. My conclusion was that the initial LRU positions of NFU pages are not a bottleneck for workloads I've tested. The efficiency of testing/clearing the accessed bit is.
And some applications are smart enough to leverage
MADV_SEQUENTIAL. In this case, MGLRU does place mmapped file pages in the oldest generation.I have an oversimplified script that uses memcached to mimic a non-streaming workload and fio a (mmapped) streaming workload:
1. WithMADV_SEQUENTIAL, the non-streaming workload is about 5 times faster when using MGLRU. Somehow the baseline (rc3) swapped a lot. (It shouldn't, and I haven't figured out why.)
2. WithoutMADV_SEQUENTIAL, the non-streaming workload is about 1 time faster when using MGLRU. Both MGLRU and the baseline swapped a lot.
MADV_SEQUENTIALnon-streaming ops/sec (memcached)rc3 yes 292k rc3 no 203k rc3+v7 yes 1967k rc3+v7 no 436kcat
mmap.sh
modprobe brd rd_nr=2 rd_size=56623104mkswap
/dev/ram0
swapon/dev/ram0mkfs.ext4
/dev/ram1
mount -t ext4/dev/ram1/mntmemtier_benchmark -S /var/run/memcached/memcached.sock -P memcache_binary \ -n allkeys --key-minimum=1 --key-maximum=50000000 --key-pattern=P:P -c 1 \ -t 36 --ratio 1:0 --pipeline 8 -d 2000 # streaming workload: --fadvise_hint=0 disables MADV_SEQUENTIAL fio -name=mglru --numjobs=12 --directory=/mnt --size=4224m --buffered=1 \ --ioengine=mmap --iodepth=128 --iodepth_batch_submit=32 \ --iodepth_batch_complete=32 --rw=read --time_based --ramp_time=10m \ --runtime=180m --group_reporting & pid=$!sleep 200
Quoted 7 lines
# non-streaming workload memtier_benchmark -S /var/run/memcached/memcached.sock -P memcache_binary \ -n allkeys --key-minimum=1 --key-maximum=50000000 --key-pattern=R:R \ -c 1 -t 36 --ratio 0:1 --pipeline 8 --randomize --distinct-client-seedkill -INT $pid
wait
We used to put a faulted file page in inactive, if we access it a second time, it can be promoted
to active. then in recent years, we have also applied this to anon pages while kernel adds
workingset protection for anon pages. so basically both anon and file pages go into the inactive
list for the 1st time, if we access it for the second time, they go to the active list. if we don't access
it any more, they are likely to be reclaimed as they are inactive. we do have some special fastpath for code section, executable file pages are kept on active list
as long as they are accessed.
so all of the above concerns are actually not that correct?
Quoted 187 lines
Quoted 16 lines
Quoted 5 lines
include/linux/mm_inline.h +4
-0+* 2) If it can't be evicted immediately, i.e., it's an anon page and +* not in swapcache, or a dirty page pending writeback, add it to the +* second oldest generation. +* 3) If it may be evicted immediately, e.g., it's a clean page, add it include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 204
Quoted 10 lines
include/linux/mm_inline.h +10
-0+* to the oldest generation. +*/ +if (folio_test_active (folio)) + gen =lru_gen_from_seq (lrugen->max_seq); +else if ((!type && !folio_test_swapcache (folio)) || + (folio_test_reclaim (folio) && + (folio_test_dirty (folio) ||folio_test_writeback (folio)))) + gen =lru_gen_from_seq (lrugen->min_seq[type] +1 ); +else + gen =lru_gen_from_seq (lrugen->min_seq[type]);Quoted 4 lines
Condition #2 is not quite clear to me, and the comment is incomplete:
The code does put dirty/writeback pages on the oldest gen as long as they haven't been marked for immediate reclaim by the scanner yet.Right.
Quoted 4 lines
HOWEVER, once the scanner does see those pages and sets
PG_reclaim, it will also activate them to move them out of the way until writeback finishes (seeshrink_page_list()) - at which point we'll trigger #1. So that second part of #2 appears unreachable.Yes, dirty file pages go to #1; dirty pages in swapcache go to #2. (Ideally we want dirty file pages go to #2 too. IMO, the code would
be cleaner that way.)Quoted 3 lines
It could be a good exercise to describe how cache pages move through the generations, similar to the comment on
lru_deactivate_file_fn(). It's a good example of intent vs implementation.Will do.
Quoted 2 lines
On another note, "!type" meaning "anon" is a bit rough. Please follow the "bool file" convention used elsewhere.
Originally I used "file", e.g., in v2:
https://lore.kernel.org/linux-mm/[email protected]/But I was told to renamed it since "file" usually means file. Let me rename it back to "file", unless somebody still objects.
Quoted 11 lines
include/linux/mm_inline.h void lruvec_add_folio_tail(struct lruvec *lruvec, struct folio *folio) · line 303
Quoted 7 lines
include/linux/mm_inline.h +3
-0@@ -113,6 +298,9 @@ void lruvec_add_folio_tail(struct lruvec *lruvec, struct folio *folio) {enum lru_list lru =folio_lru_list (folio); +if (lru_gen_add_folio (lruvec, folio, true)) +return ; +bool parameters are notoriously hard to follow in the callsite. Can you please add
lru_gen_add_folio_tail()instead and have them use a common helper?I'm not sure -- there are several places like this one. My question is whether we want to do it throughout this patchset. We'd end up with many helpers and duplicate code. E.g., in this file alone, we have two functions taking bool parameters:
lru_gen_add_folio(..., bool reclaiming)lru_gen_del_folio(..., bool reclaiming)I can't say they are very readable; at least they are very compact right now. My concern is that we might loose the latter without having enough of the former.
Perhaps this is something that we could revisit after you've finished reviewing the entire patchset?
Quoted 18 lines
Quoted 18 lines
include/linux/mm_inline.h +3
-0@@ -127,6 +315,9 @@ static __always_inline void add_page_to_lru_list_tail(struct page *page,static __always_inline void lruvec_del_folio(struct lruvec *lruvec, struct folio *folio) { +if (lru_gen_del_folio (lruvec, folio, false)) +return ; +list_del (&folio->lru);update_lru_size (lruvec,folio_lru_list (folio),folio_zonenum (folio), -folio_nr_pages (folio));include/linux/mmzone.h +0
-0@@ -303,6 +303,78 @@ enum lruvec_flags {*/ };Quoted 12 lines
include/linux/mmzone.h enum lruvec_flags { · line 314
Quoted 9 lines
include/linux/mmzone.h +9
-0+struct lruvec ; + +#define LRU_GEN_MASK ((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF) +#define LRU_REFS_MASK ((BIT(LRU_REFS_WIDTH) - 1) << LRU_REFS_PGOFF) + +#ifdef CONFIG_LRU_GEN + +#define MIN_LRU_BATCH BITS_PER_LONG +#define MAX_LRU_BATCH (MIN_LRU_BATCH * 128)Those two aren't used in this patch, so it's hard to say whether they are chosen correctly.
Right. They slipped during the v6/v7 refactoring. Will move them to the next patch.
Quoted 25 lines
Quoted 12 lines
include/linux/mmzone.h +12
-0+* Evictable pages are divided into multiple generations. The youngest and the +* oldest generation numbers, max_seq and min_seq, are monotonically increasing. +* They form a sliding window of a variable size [MIN_NR_GENS, MAX_NR_GENS]. An +* offset within MAX_NR_GENS, gen, indexes the LRU list of the corresponding +* generation. The gen counter in folio->flags stores gen+1 while a page is on +* one of lrugen->lists[]. Otherwise it stores 0. +* +* A page is added to the youngest generation on faulting. The aging needs to +* check the accessed bit at least twice before handing this page over to the +* eviction. The first check takes care of the accessed bit set on the initial +* fault; the second check makes sure this page hasn't been used since then. +* This process, AKA second chance, requires a minimum of two generations, include/linux/mmzone.h enum lruvec_flags { · line 338
Quoted 10 lines
include/linux/mmzone.h +10
-0+* hence MIN_NR_GENS. And to be compatible with the active/inactive LRU, these +* two generations are mapped to the active; the rest of generations, if they +* exist, are mapped to the inactive. PG_active is always cleared while a page +* is on one of lrugen->lists[] so that demotion, which happens consequently +* when the aging produces a new generation, needs not to worry about it. +*/ +#define MIN_NR_GENS 2U +#define MAX_NR_GENS ((unsigned int)CONFIG_NR_LRU_GENS) + +struct lru_gen_struct {struct lrugen?
Quoted 7 lines
In fact, "lrugen" for the general function and variable namespace might be better, the _ doesn't seem to pull its weight.
CONFIG_LRUGEN
struct lrugenlrugen_foo()
etc.No strong opinion here. I usually add underscores to functions and types so that grep doesn't end up with tons of local variables.
Quoted 7 lines
include/linux/mmzone.h enum lruvec_flags { · line 342
Quoted 4 lines
include/linux/mmzone.h +4
-0+/* the aging increments the youngest generation number */ +unsigned long max_seq; +/* the eviction increments the oldest generation numbers */ +unsigned long min_seq[ANON_AND_FILE ];The singular
max_seqvs the splitmin_seqraises questions. Please add a comment that explains or points to an explanation.Will do.
Quoted 4 lines
include/linux/mmzone.h enum lruvec_flags { · line 344
Quoted 2 lines
include/linux/mmzone.h +2
-0+/* the birth time of each generation in jiffies */ +unsigned long timestamps[MAX_NR_GENS ];This isn't in use until the thrashing-based OOM killing patch.
Will move it there.
Quoted 10 lines
include/linux/mmzone.h enum lruvec_flags { · line 350
Quoted 6 lines
include/linux/mmzone.h +6
-0+/* the multigenerational LRU lists */ +struct list_head lists[MAX_NR_GENS ][ANON_AND_FILE ][MAX_NR_ZONES ]; +/* the sizes of the above lists */ +unsigned long nr_pages[MAX_NR_GENS ][ANON_AND_FILE ][MAX_NR_ZONES ]; +/* whether the multigenerational LRU is enabled */ +bool enabled;Not (really) in use until the runtime switch. Best to keep everybody checking the global flag for now, and have the runtime switch patch introduce this flag and switch necessary callsites over.
Will do.
Quoted 3 lines
include/linux/mmzone.h enum lruvec_flags { · line 353
Quoted 1 line
include/linux/mmzone.h +1
-0+void lru_gen_init_state (struct mem_cgroup *memcg,struct lruvec *lruvec);"state" is what we usually init :) How about
lrugen_init_lruvec()?Same story as "file", lol -- this used to be
lru_gen_init_lruvec():
https://lore.kernel.org/linux-mm/[email protected]/Naming is hard. Hopefully we can finalize it this time.
Quoted 1 line
You can drop the memcg parameter and use
lruvec_memcg().
lruvec_memcg()isn't available yet whenpgdat_init_internals()calls this function becausemem_cgroup_disabled()is initialized afterward.
Quoted 7 lines
Quoted 5 lines
include/linux/mmzone.h enum lruvec_flags { · line 357
Quoted 3 lines
include/linux/mmzone.h +3
-0+#ifdef CONFIG_MEMCG +void lru_gen_init_memcg (struct mem_cgroup *memcg); +void lru_gen_free_memcg (struct mem_cgroup *memcg);This should be either init+exit, or alloc+free.
Will do.
Thanks
Barry
Re: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
Quoted 29 lines · Barry Song
Quoted 19 lines · Yu Zhao
On Thu, Feb 10, 2022 at 03:41:57PM -0500, Johannes Weiner wrote:
Thanks for reviewing.
Quoted 5 lines
Quoted 5 lines
include/linux/mm_inline.h +5
-0+static inline bool lru_gen_is_active (struct lruvec *lruvec,int gen) +{ +unsigned long max_seq = lruvec->lrugen.max_seq; + +VM_BUG_ON (gen >=MAX_NR_GENS );Quoted 8 lines
include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 120
Quoted 4 lines
include/linux/mm_inline.h +4
-0+ +/* see the comment on MIN_NR_GENS */ +return gen ==lru_gen_from_seq (max_seq) || gen ==lru_gen_from_seq (max_seq -1 ); +}I'm still reading the series, so correct me if I'm wrong: the "active" set is split into two generations for the sole purpose of the second-chance policy for fresh faults, right?
Quoted 3 lines
To be precise, the active/inactive notion on top of generations is just for ABI compatibility, e.g., the counters in
/proc/vmstat. Otherwise, this function wouldn't be needed.Hi Yu,
I am still quite confused as i am seeing both active/inactive andlru_gen. eg:
Quoted 9 lines
root@ubuntu:~# cat /proc/vmstat | grep active nr_zone_inactive_anon 22797 nr_zone_active_anon 578405 nr_zone_inactive_file 0 nr_zone_active_file 4156 nr_inactive_anon 22800 nr_active_anon 578574 nr_inactive_file 0 nr_active_file 4215
Yes, this is expected. We have to maintain the ABI, i.e., the *_active/inactive_* counters.
Quoted 22 lines
and:
root@ubuntu:~# cat /sys//kernel/debug/lru_gen...
memcg 36 /user.slice/user-0.slice/[email protected]
node 020 18820 22 0 21 7452 0 0 22 7448 0 0 memcg 33 /user.slice/user-0.slice/[email protected]/app.slice node 0 0 2171452 0 0 1 2171452 0 0 2 2171452 0 0 3 2171452 0 0 memcg 37 /user.slice/user-0.slice/session-1.scope node 0 42 51804 102127 0 43 18840 275622 0 44 16104 216805 1
Quoted 2 lines
Does it mean one page could be in both one of the generations and one of the active/inactive lists?
In terms of the data structure, evictable pages are either on lruvec->lists or lrugen->lists.
Quoted 2 lines
Do we have some mapping relationship between active/inactive lists with generations?
For the counters, yes -- pages in max_seq and max_seq-1 are counted as active, and the rest are inactive.
Quoted 4 lines
We used to put a faulted file page in inactive, if we access it a second time, it can be promoted
to active. then in recent years, we have also applied this to anon pages while kernel adds
Quoted 8 lines
workingset protection for anon pages. so basically both anon and file pages go into the inactive
list for the 1st time, if we access it for the second time, they go to the active list. if we don't access
it any more, they are likely to be reclaimed as they are inactive. we do have some special fastpath for code section, executable file pages are kept on active list
as long as they are accessed.
Yes.
Quoted 1 line
so all of the above concerns are actually not that correct?
They are valid concerns but I don't know any popular workloads that care about them.
Re: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
Quoted 91 lines · Yu Zhao
Quoted 29 lines · Barry Song
Quoted 19 lines · Yu Zhao
On Thu, Feb 10, 2022 at 03:41:57PM -0500, Johannes Weiner wrote:
Thanks for reviewing.
Quoted 5 lines
Quoted 5 lines
include/linux/mm_inline.h +5
-0+static inline bool lru_gen_is_active (struct lruvec *lruvec,int gen) +{ +unsigned long max_seq = lruvec->lrugen.max_seq; + +VM_BUG_ON (gen >=MAX_NR_GENS );Quoted 8 lines
include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 120
Quoted 4 lines
include/linux/mm_inline.h +4
-0+ +/* see the comment on MIN_NR_GENS */ +return gen ==lru_gen_from_seq (max_seq) || gen ==lru_gen_from_seq (max_seq -1 ); +}I'm still reading the series, so correct me if I'm wrong: the "active" set is split into two generations for the sole purpose of the second-chance policy for fresh faults, right?
Quoted 3 lines
To be precise, the active/inactive notion on top of generations is just for ABI compatibility, e.g., the counters in
/proc/vmstat. Otherwise, this function wouldn't be needed.Hi Yu,
I am still quite confused as i am seeing both active/inactive andlru_gen. eg:Quoted 9 lines
root@ubuntu:~# cat /proc/vmstat | grep active nr_zone_inactive_anon 22797 nr_zone_active_anon 578405 nr_zone_inactive_file 0 nr_zone_active_file 4156 nr_inactive_anon 22800 nr_active_anon 578574 nr_inactive_file 0 nr_active_file 4215Yes, this is expected. We have to maintain the ABI, i.e., the *_active/inactive_* counters.
Quoted 22 lines
and:
root@ubuntu:~# cat /sys//kernel/debug/lru_gen...
memcg 36 /user.slice/user-0.slice/[email protected]
node 020 18820 22 0 21 7452 0 0 22 7448 0 0 memcg 33 /user.slice/user-0.slice/[email protected]/app.slice node 0 0 2171452 0 0 1 2171452 0 0 2 2171452 0 0 3 2171452 0 0 memcg 37 /user.slice/user-0.slice/session-1.scope node 0 42 51804 102127 0 43 18840 275622 0 44 16104 216805 1Quoted 2 lines
Does it mean one page could be in both one of the generations and one of the active/inactive lists?
In terms of the data structure, evictable pages are either on
lruvec->listsorlrugen->lists.Quoted 2 lines
Do we have some mapping relationship between active/inactive lists with generations?
For the counters, yes -- pages in
max_seqand max_seq-1 are counted as active, and the rest are inactive.Quoted 4 lines
We used to put a faulted file page in inactive, if we access it a second time, it can be promoted
to active. then in recent years, we have also applied this to anon pages while kernel addsQuoted 8 lines
workingset protection for anon pages. so basically both anon and file pages go into the inactive
list for the 1st time, if we access it for the second time, they go to the active list. if we don't access
it any more, they are likely to be reclaimed as they are inactive. we do have some special fastpath for code section, executable file pages are kept on active list
as long as they are accessed.
Quoted 6 lines
Yes.
Quoted 1 line
so all of the above concerns are actually not that correct?
They are valid concerns but I don't know any popular workloads that care about them.
Hi Yu,
here we can get a workload in Kim's patchset while he added workingset protection
for anon pages:
https://patchwork.kernel.org/project/linux-mm/cover/[email protected]/
anon pages used to go to active rather than inactive, but kim's patchset moved to use inactive first. then only after the anon page is accessed second time, it can move to active.
"In current implementation, newly created or swap-in anonymous page is
started on the active list. Growing the active list results in rebalancing active/inactive list so old pages on the active list are demoted to the inactive list. Hence, hot page on the active list isn't protected at all.
Following is an example of this situation.
Assume that 50 hot pages on active list and system can contain total 100 pages. Numbers denote the number of pages on active/inactive list (active | inactive). (h) stands for hot pages and (uo) stands for used-once pages.
1. 50 hot pages on active list
50(h) | 0
2. workload: 50 newly created (used-once) pages
50(uo) | 50(h)
3. workload: another 50 newly created (used-once) pages 50(uo) | 50(uo), swap-out 50(h)
As we can see, hot pages are swapped-out and it would cause swap-in later."
Is MGLRU able to avoid the swap-out of the 50 hot pages? since MGLRU is putting faulted pages to the youngest generation directly, do we have the risk mentioned in Kim's patchset?
Thanks
Barry
Re: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
Quoted 97 lines · Barry Song
Quoted 95 lines · Yu Zhao
Quoted 29 lines · Barry Song
Quoted 19 lines · Yu Zhao
On Thu, Feb 10, 2022 at 03:41:57PM -0500, Johannes Weiner wrote:
Thanks for reviewing.
Quoted 5 lines
Quoted 5 lines
include/linux/mm_inline.h +5
-0+static inline bool lru_gen_is_active (struct lruvec *lruvec,int gen) +{ +unsigned long max_seq = lruvec->lrugen.max_seq; + +VM_BUG_ON (gen >=MAX_NR_GENS );Quoted 8 lines
include/linux/mm_inline.h static __always_inline enum lru_list folio_lru_list(struct folio *folio) · line 120
Quoted 4 lines
include/linux/mm_inline.h +4
-0+ +/* see the comment on MIN_NR_GENS */ +return gen ==lru_gen_from_seq (max_seq) || gen ==lru_gen_from_seq (max_seq -1 ); +}I'm still reading the series, so correct me if I'm wrong: the "active" set is split into two generations for the sole purpose of the second-chance policy for fresh faults, right?
Quoted 3 lines
To be precise, the active/inactive notion on top of generations is just for ABI compatibility, e.g., the counters in
/proc/vmstat. Otherwise, this function wouldn't be needed.Hi Yu,
I am still quite confused as i am seeing both active/inactive andlru_gen. eg:Quoted 9 lines
root@ubuntu:~# cat /proc/vmstat | grep active nr_zone_inactive_anon 22797 nr_zone_active_anon 578405 nr_zone_inactive_file 0 nr_zone_active_file 4156 nr_inactive_anon 22800 nr_active_anon 578574 nr_inactive_file 0 nr_active_file 4215Yes, this is expected. We have to maintain the ABI, i.e., the *_active/inactive_* counters.
Quoted 22 lines
and:
root@ubuntu:~# cat /sys//kernel/debug/lru_gen...
memcg 36 /user.slice/user-0.slice/[email protected]
node 020 18820 22 0 21 7452 0 0 22 7448 0 0 memcg 33 /user.slice/user-0.slice/[email protected]/app.slice node 0 0 2171452 0 0 1 2171452 0 0 2 2171452 0 0 3 2171452 0 0 memcg 37 /user.slice/user-0.slice/session-1.scope node 0 42 51804 102127 0 43 18840 275622 0 44 16104 216805 1Quoted 2 lines
Does it mean one page could be in both one of the generations and one of the active/inactive lists?
In terms of the data structure, evictable pages are either on
lruvec->listsorlrugen->lists.Quoted 2 lines
Do we have some mapping relationship between active/inactive lists with generations?
For the counters, yes -- pages in
max_seqand max_seq-1 are counted as active, and the rest are inactive.Quoted 4 lines
We used to put a faulted file page in inactive, if we access it a second time, it can be promoted
to active. then in recent years, we have also applied this to anon pages while kernel addsQuoted 8 lines
workingset protection for anon pages. so basically both anon and file pages go into the inactive
list for the 1st time, if we access it for the second time, they go to the active list. if we don't access
it any more, they are likely to be reclaimed as they are inactive. we do have some special fastpath for code section, executable file pages are kept on active list
as long as they are accessed.Yes.
Quoted 1 line
so all of the above concerns are actually not that correct?
Quoted 8 lines
Quoted 2 lines
They are valid concerns but I don't know any popular workloads that care about them.
Hi Yu,
here we can get a workload in Kim's patchset while he added workingset protection
for anon pages:
https://patchwork.kernel.org/project/linux-mm/cover/[email protected]/
Thanks. I wouldn't call that a workload because it's not a real application. By popular workloads, I mean applications that the majority of people actually run on phones, in cloud, etc.
Quoted 3 lines
anon pages used to go to active rather than inactive, but kim's patchset moved to use inactive first. then only after the anon page is accessed second time, it can move to active.
Yes. To clarify, the A-bit doesn't really mean the first or second access. It can be many accesses each time it's set.
Quoted 19 lines
"In current implementation, newly created or swap-in anonymous page is
started on the active list. Growing the active list results in rebalancing active/inactive list so old pages on the active list are demoted to the inactive list. Hence, hot page on the active list isn't protected at all.
Following is an example of this situation.
Assume that 50 hot pages on active list and system can contain total 100 pages. Numbers denote the number of pages on active/inactive list (active | inactive). (h) stands for hot pages and (uo) stands for used-once pages.
1. 50 hot pages on active list
50(h) | 02. workload: 50 newly created (used-once) pages
50(uo) | 50(h)
Quoted 6 lines
3. workload: another 50 newly created (used-once) pages 50(uo) | 50(uo), swap-out 50(h)
As we can see, hot pages are swapped-out and it would cause swap-in later."
Is MGLRU able to avoid the swap-out of the 50 hot pages?
I think the real question is why the 50 hot pages can be moved to the inactive list. If they are really hot, the A-bit should protect them.
Quoted 3 lines
since MGLRU
is putting faulted pages to the youngest generation directly, do we have the risk mentioned in Kim's patchset?
There are always risks :) I could imagine a thousand ways to make VM suffer, but all of them could be irrelevant to how it actually does in production. So a concrete use case of yours would be much appreciated for this discussion.
Re: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
Quoted 141 lines · Yu Zhao
Quoted 97 lines · Barry Song
Quoted 95 lines · Yu Zhao
Quoted 29 lines · Barry Song
Quoted 19 lines · Yu Zhao
On Thu, Feb 10, 2022 at 03:41:57PM -0500, Johannes Weiner wrote:
Thanks for reviewing.
Quoted 5 lines
Quoted 5 lines
(quoted diff) +5
-0+static inline bool lru_gen_is_active(struct lruvec *lruvec, int gen) +{ + unsigned long max_seq = lruvec->lrugen.max_seq; + + VM_BUG_ON(gen >= MAX_NR_GENS);Quoted 8 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+ + /* see the comment on MIN_NR_GENS */ + return gen == lru_gen_from_seq(max_seq) || gen == lru_gen_from_seq(max_seq - 1); +}I'm still reading the series, so correct me if I'm wrong: the "active" set is split into two generations for the sole purpose of the second-chance policy for fresh faults, right?
Quoted 3 lines
To be precise, the active/inactive notion on top of generations is just for ABI compatibility, e.g., the counters in
/proc/vmstat. Otherwise, this function wouldn't be needed.Hi Yu,
I am still quite confused as i am seeing both active/inactive andlru_gen. eg:Quoted 9 lines
root@ubuntu:~# cat /proc/vmstat | grep active nr_zone_inactive_anon 22797 nr_zone_active_anon 578405 nr_zone_inactive_file 0 nr_zone_active_file 4156 nr_inactive_anon 22800 nr_active_anon 578574 nr_inactive_file 0 nr_active_file 4215Yes, this is expected. We have to maintain the ABI, i.e., the *_active/inactive_* counters.
Quoted 22 lines
and:
root@ubuntu:~# cat /sys//kernel/debug/lru_gen...
memcg 36 /user.slice/user-0.slice/[email protected]
node 020 18820 22 0 21 7452 0 0 22 7448 0 0 memcg 33 /user.slice/user-0.slice/[email protected]/app.slice node 0 0 2171452 0 0 1 2171452 0 0 2 2171452 0 0 3 2171452 0 0 memcg 37 /user.slice/user-0.slice/session-1.scope node 0 42 51804 102127 0 43 18840 275622 0 44 16104 216805 1Quoted 2 lines
Does it mean one page could be in both one of the generations and one of the active/inactive lists?
In terms of the data structure, evictable pages are either on
lruvec->listsorlrugen->lists.Quoted 2 lines
Do we have some mapping relationship between active/inactive lists with generations?
For the counters, yes -- pages in
max_seqand max_seq-1 are counted as active, and the rest are inactive.Quoted 4 lines
We used to put a faulted file page in inactive, if we access it a second time, it can be promoted
to active. then in recent years, we have also applied this to anon pages while kernel addsQuoted 8 lines
workingset protection for anon pages. so basically both anon and file pages go into the inactive
list for the 1st time, if we access it for the second time, they go to the active list. if we don't access
it any more, they are likely to be reclaimed as they are inactive. we do have some special fastpath for code section, executable file pages are kept on active list
as long as they are accessed.Yes.
Quoted 1 line
so all of the above concerns are actually not that correct?
Quoted 8 lines
Quoted 2 lines
They are valid concerns but I don't know any popular workloads that care about them.
Hi Yu,
here we can get a workload in Kim's patchset while he added workingset protection
for anon pages:
https://patchwork.kernel.org/project/linux-mm/cover/[email protected]/Thanks. I wouldn't call that a workload because it's not a real application. By popular workloads, I mean applications that the majority of people actually run on phones, in cloud, etc.
Quoted 3 lines
anon pages used to go to active rather than inactive, but kim's patchset moved to use inactive first. then only after the anon page is accessed second time, it can move to active.
Yes. To clarify, the A-bit doesn't really mean the first or second access. It can be many accesses each time it's set.
Quoted 22 lines
"In current implementation, newly created or swap-in anonymous page is
started on the active list. Growing the active list results in rebalancing active/inactive list so old pages on the active list are demoted to the inactive list. Hence, hot page on the active list isn't protected at all.
Following is an example of this situation.
Assume that 50 hot pages on active list and system can contain total 100 pages. Numbers denote the number of pages on active/inactive list (active | inactive). (h) stands for hot pages and (uo) stands for used-once pages.
1. 50 hot pages on active list
50(h) | 02. workload: 50 newly created (used-once) pages
50(uo) | 50(h)3. workload: another 50 newly created (used-once) pages 50(uo) | 50(uo), swap-out 50(h)
Quoted 6 lines
Quoted 3 lines
As we can see, hot pages are swapped-out and it would cause swap-in later."
Is MGLRU able to avoid the swap-out of the 50 hot pages?
I think the real question is why the 50 hot pages can be moved to the inactive list. If they are really hot, the A-bit should protect them.
This is a good question.
I guess it is probably because the current lru is trying to maintain a balance between the sizes of active and inactive lists. Thus, it can shrink active list even though pages might be still "hot" but not the recently accessed ones.
1. 50 hot pages on active list
50(h) | 0
2. workload: 50 newly created (used-once) pages
50(uo) | 50(h)
3. workload: another 50 newly created (used-once) pages 50(uo) | 50(uo), swap-out 50(h)
the old kernel without anon workingset protection put workload 2 on active, so pushed 50 hot pages from active to inactive. workload 3 would further contribute to evict the 50 hot pages.
it seems mglru doesn't demote pages from the youngest generation to older generation only in order to balance the list size? so mglru is probably safe in these cases.
I will run some tests mentioned in Kim's patchset and report the result to you afterwards.
Quoted 9 lines
Quoted 4 lines
since MGLRU
is putting faulted pages to the youngest generation directly, do we have the risk mentioned in Kim's patchset?There are always risks :) I could imagine a thousand ways to make VM suffer, but all of them could be irrelevant to how it actually does in production. So a concrete use case of yours would be much appreciated for this discussion.
Thanks
Barry
Re: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
Quoted 86 lines
Quoted 60 lines
Quoted 18 lines
Quoted 18 lines
Quoted 13 lines
We used to put a faulted file page in inactive, if we access it a second time, it can be promoted
to active. then in recent years, we have also applied this to anon pages while kernel adds
workingset protection for anon pages. so basically both anon and file pages go into the inactive
list for the 1st time, if we access it for the second time, they go to the active list. if we don't access
it any more, they are likely to be reclaimed as they are inactive. we do have some special fastpath for code section, executable file pages are kept on active list
as long as they are accessed.Yes.
Quoted 1 line
so all of the above concerns are actually not that correct?
Quoted 8 lines
Quoted 2 lines
They are valid concerns but I don't know any popular workloads that care about them.
Hi Yu,
here we can get a workload in Kim's patchset while he added workingset protection
for anon pages:
https://patchwork.kernel.org/project/linux-mm/cover/[email protected]/Thanks. I wouldn't call that a workload because it's not a real application. By popular workloads, I mean applications that the majority of people actually run on phones, in cloud, etc.
Quoted 3 lines
anon pages used to go to active rather than inactive, but kim's patchset moved to use inactive first. then only after the anon page is accessed second time, it can move to active.
Yes. To clarify, the A-bit doesn't really mean the first or second access. It can be many accesses each time it's set.
Quoted 22 lines
"In current implementation, newly created or swap-in anonymous page is
started on the active list. Growing the active list results in rebalancing active/inactive list so old pages on the active list are demoted to the inactive list. Hence, hot page on the active list isn't protected at all.
Following is an example of this situation.
Assume that 50 hot pages on active list and system can contain total 100 pages. Numbers denote the number of pages on active/inactive list (active | inactive). (h) stands for hot pages and (uo) stands for used-once pages.
1. 50 hot pages on active list
50(h) | 02. workload: 50 newly created (used-once) pages
50(uo) | 50(h)3. workload: another 50 newly created (used-once) pages 50(uo) | 50(uo), swap-out 50(h)
Quoted 6 lines
Quoted 3 lines
As we can see, hot pages are swapped-out and it would cause swap-in later."
Is MGLRU able to avoid the swap-out of the 50 hot pages?
I think the real question is why the 50 hot pages can be moved to the inactive list. If they are really hot, the A-bit should protect them.
This is a good question.
I guess it is probably because the current lru is trying to maintain a balance between the sizes of active and inactive lists. Thus, it can shrink active list even though pages might be still "hot" but not the recently accessed ones.
1. 50 hot pages on active list
50(h) | 02. workload: 50 newly created (used-once) pages
50(uo) | 50(h)3. workload: another 50 newly created (used-once) pages 50(uo) | 50(uo), swap-out 50(h)
the old kernel without anon workingset protection put workload 2 on active, so pushed 50 hot pages from active to inactive. workload 3 would further contribute to evict the 50 hot pages.
Quoted 7 lines
it seems mglru doesn't demote pages from the youngest generation to older generation only in order to balance the list size? so mglru is probably safe in these cases.
I will run some tests mentioned in Kim's patchset and report the result to you afterwards.
Hi Yu,
I did find putting faulted pages to the youngest generation lead to some regression in the case ebizzy Kim's patchset mentioned while he tried to support workingset protection for anon pages.
i did a little bit modification for rand_chunk() which is probably similar with the modifcation() Kim mentioned in his patchset. The modification can be found here:
https://github.com/21cnbao/ltp/commit/7134413d747bfa9ef
The test env is a x86 machine in which I have set memory size to 2.5GB and set zRAM to 2GB and disabled external disk swap.
with the vanilla kernel:
\time -v ./a.out -vv -t 4 -s 209715200 -S 200000
so we have 10 chunks and 4 threads, each trunk is 209715200(200MB)
typical result:
Command being timed: "./a.out -vv -t 4 -s 209715200 -S 200000" User time (seconds): 36.19
System time (seconds): 229.72
Percent of CPU this job got: 371%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:11.59 Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2166196
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 9990128
Minor (reclaiming a frame) page faults: 33315945 Voluntary context switches: 59144
Involuntary context switches: 167754
Swaps: 0
File system inputs: 2760
File system outputs: 8
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
with gen_lru and lru_gen/enabled=0x3:
typical result:
Command being timed: "./a.out -vv -t 4 -s 209715200 -S 200000" User time (seconds): 36.34
System time (seconds): 276.07
Percent of CPU this job got: 378%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:22.46
**** 15% time +
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2168120
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 13362810
***** 30% page fault +
Minor (reclaiming a frame) page faults: 33394617
Voluntary context switches: 55216
Involuntary context switches: 137220
Swaps: 0
File system inputs: 4088
File system outputs: 8
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
with gen_lru and lru_gen/enabled=0x7:
typical result:
Command being timed: "./a.out -vv -t 4 -s 209715200 -S 200000" User time (seconds): 36.13
System time (seconds): 251.71
Percent of CPU this job got: 378%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:16.00
*****better than enabled=0x3, worse than vanilla
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2120988
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 12706512
Minor (reclaiming a frame) page faults: 33422243
Voluntary context switches: 49485
Involuntary context switches: 126765
Swaps: 0
File system inputs: 2976
File system outputs: 8
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
I can also reproduce the problem on arm64.
I am not saying this is going to block mglru from being mainlined. But I am still curious if this is an issue worth being addressed somehow in mglru.
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines
since MGLRU
is putting faulted pages to the youngest generation directly, do we have the risk mentioned in Kim's patchset?There are always risks :) I could imagine a thousand ways to make VM suffer, but all of them could be irrelevant to how it actually does in production. So a concrete use case of yours would be much appreciated for this discussion.
Thanks
Barry
Re: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
Quoted 192 lines · Barry Song
Quoted 87 lines
Quoted 61 lines
Quoted 19 lines
Quoted 19 lines
Quoted 14 lines
We used to put a faulted file page in inactive, if we access it a second time, it can be promoted
to active. then in recent years, we have also applied this to anon pages while kernel adds
workingset protection for anon pages. so basically both anon and file pages go into the inactive
list for the 1st time, if we access it for the second time, they go to the active list. if we don't access
it any more, they are likely to be reclaimed as they are inactive. we do have some special fastpath for code section, executable file pages are kept on active list
as long as they are accessed.Yes.
Quoted 1 line
so all of the above concerns are actually not that correct?
Quoted 8 lines
Quoted 2 lines
They are valid concerns but I don't know any popular workloads that care about them.
Hi Yu,
here we can get a workload in Kim's patchset while he added workingset protection
for anon pages:
https://patchwork.kernel.org/project/linux-mm/cover/[email protected]/Thanks. I wouldn't call that a workload because it's not a real application. By popular workloads, I mean applications that the majority of people actually run on phones, in cloud, etc.
Quoted 3 lines
anon pages used to go to active rather than inactive, but kim's patchset moved to use inactive first. then only after the anon page is accessed second time, it can move to active.
Yes. To clarify, the A-bit doesn't really mean the first or second access. It can be many accesses each time it's set.
Quoted 22 lines
"In current implementation, newly created or swap-in anonymous page is
started on the active list. Growing the active list results in rebalancing active/inactive list so old pages on the active list are demoted to the inactive list. Hence, hot page on the active list isn't protected at all.
Following is an example of this situation.
Assume that 50 hot pages on active list and system can contain total 100 pages. Numbers denote the number of pages on active/inactive list (active | inactive). (h) stands for hot pages and (uo) stands for used-once pages.
1. 50 hot pages on active list
50(h) | 02. workload: 50 newly created (used-once) pages
50(uo) | 50(h)3. workload: another 50 newly created (used-once) pages 50(uo) | 50(uo), swap-out 50(h)
Quoted 6 lines
Quoted 3 lines
As we can see, hot pages are swapped-out and it would cause swap-in later."
Is MGLRU able to avoid the swap-out of the 50 hot pages?
I think the real question is why the 50 hot pages can be moved to the inactive list. If they are really hot, the A-bit should protect them.
This is a good question.
I guess it is probably because the current lru is trying to maintain a balance between the sizes of active and inactive lists. Thus, it can shrink active list even though pages might be still "hot" but not the recently accessed ones.
1. 50 hot pages on active list
50(h) | 02. workload: 50 newly created (used-once) pages
50(uo) | 50(h)3. workload: another 50 newly created (used-once) pages 50(uo) | 50(uo), swap-out 50(h)
the old kernel without anon workingset protection put workload 2 on active, so pushed 50 hot pages from active to inactive. workload 3 would further contribute to evict the 50 hot pages.
Quoted 7 lines
it seems mglru doesn't demote pages from the youngest generation to older generation only in order to balance the list size? so mglru is probably safe in these cases.
I will run some tests mentioned in Kim's patchset and report the result to you afterwards.
Hi Yu,
I did find putting faulted pages to the youngest generation lead to some regression in the case ebizzy Kim's patchset mentioned while he tried to support workingset protection for anon pages.
i did a little bit modification forrand_chunk()which is probably similar with themodifcation()Kim mentioned in his patchset. The modification can be found here:
https://github.com/21cnbao/ltp/commit/7134413d747bfa9efThe test env is a x86 machine in which I have set memory size to 2.5GB and set zRAM to 2GB and disabled external disk swap.
with the vanilla kernel:
\time -v ./a.out -vv -t 4 -s 209715200 -S 200000so we have 10 chunks and 4 threads, each trunk is 209715200(200MB)
typical result:
Command being timed: "./a.out -vv -t 4 -s 209715200 -S 200000" User time (seconds): 36.19
System time (seconds): 229.72
Percent of CPU this job got: 371%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:11.59 Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2166196
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 9990128
Minor (reclaiming a frame) page faults: 33315945 Voluntary context switches: 59144
Involuntary context switches: 167754
Swaps: 0
File system inputs: 2760
File system outputs: 8
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0with
gen_lruand lru_gen/enabled=0x3:
typical result:
Command being timed: "./a.out -vv -t 4 -s 209715200 -S 200000" User time (seconds): 36.34
System time (seconds): 276.07
Percent of CPU this job got: 378%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:22.46**** 15% time +Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2168120
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 13362810***** 30% page fault +Minor (reclaiming a frame) page faults: 33394617
Voluntary context switches: 55216
Involuntary context switches: 137220
Swaps: 0
File system inputs: 4088
File system outputs: 8
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0with
gen_lruand lru_gen/enabled=0x7:
typical result:
Command being timed: "./a.out -vv -t 4 -s 209715200 -S 200000" User time (seconds): 36.13
System time (seconds): 251.71
Percent of CPU this job got: 378%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:16.00*****better than enabled=0x3, worse than vanillaAverage shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2120988
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 12706512
Minor (reclaiming a frame) page faults: 33422243
Voluntary context switches: 49485
Involuntary context switches: 126765
Swaps: 0
File system inputs: 2976
File system outputs: 8
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
Quoted 4 lines
I can also reproduce the problem on arm64.
I am not saying this is going to block mglru from being mainlined. But I am still curious if this is an issue worth being addressed somehow in mglru.
You've missed something very important: *thoughput* :)
Dollars to doughnuts there was a large increase in throughput -- I haven't tried this benchmark but I've seen many reports similar to this one.
Re: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
Quoted 194 lines · Yu Zhao
Quoted 192 lines · Barry Song
Quoted 87 lines
Quoted 61 lines
Quoted 19 lines
Quoted 19 lines
Quoted 14 lines
We used to put a faulted file page in inactive, if we access it a second time, it can be promoted
to active. then in recent years, we have also applied this to anon pages while kernel adds
workingset protection for anon pages. so basically both anon and file pages go into the inactive
list for the 1st time, if we access it for the second time, they go to the active list. if we don't access
it any more, they are likely to be reclaimed as they are inactive. we do have some special fastpath for code section, executable file pages are kept on active list
as long as they are accessed.Yes.
Quoted 1 line
so all of the above concerns are actually not that correct?
Quoted 8 lines
Quoted 2 lines
They are valid concerns but I don't know any popular workloads that care about them.
Hi Yu,
here we can get a workload in Kim's patchset while he added workingset protection
for anon pages:
https://patchwork.kernel.org/project/linux-mm/cover/[email protected]/Thanks. I wouldn't call that a workload because it's not a real application. By popular workloads, I mean applications that the majority of people actually run on phones, in cloud, etc.
Quoted 3 lines
anon pages used to go to active rather than inactive, but kim's patchset moved to use inactive first. then only after the anon page is accessed second time, it can move to active.
Yes. To clarify, the A-bit doesn't really mean the first or second access. It can be many accesses each time it's set.
Quoted 22 lines
"In current implementation, newly created or swap-in anonymous page is
started on the active list. Growing the active list results in rebalancing active/inactive list so old pages on the active list are demoted to the inactive list. Hence, hot page on the active list isn't protected at all.
Following is an example of this situation.
Assume that 50 hot pages on active list and system can contain total 100 pages. Numbers denote the number of pages on active/inactive list (active | inactive). (h) stands for hot pages and (uo) stands for used-once pages.
1. 50 hot pages on active list
50(h) | 02. workload: 50 newly created (used-once) pages
50(uo) | 50(h)3. workload: another 50 newly created (used-once) pages 50(uo) | 50(uo), swap-out 50(h)
Quoted 6 lines
Quoted 3 lines
As we can see, hot pages are swapped-out and it would cause swap-in later."
Is MGLRU able to avoid the swap-out of the 50 hot pages?
I think the real question is why the 50 hot pages can be moved to the inactive list. If they are really hot, the A-bit should protect them.
This is a good question.
I guess it is probably because the current lru is trying to maintain a balance between the sizes of active and inactive lists. Thus, it can shrink active list even though pages might be still "hot" but not the recently accessed ones.
1. 50 hot pages on active list
50(h) | 02. workload: 50 newly created (used-once) pages
50(uo) | 50(h)3. workload: another 50 newly created (used-once) pages 50(uo) | 50(uo), swap-out 50(h)
the old kernel without anon workingset protection put workload 2 on active, so pushed 50 hot pages from active to inactive. workload 3 would further contribute to evict the 50 hot pages.
Quoted 7 lines
it seems mglru doesn't demote pages from the youngest generation to older generation only in order to balance the list size? so mglru is probably safe in these cases.
I will run some tests mentioned in Kim's patchset and report the result to you afterwards.
Hi Yu,
I did find putting faulted pages to the youngest generation lead to some regression in the case ebizzy Kim's patchset mentioned while he tried to support workingset protection for anon pages.
i did a little bit modification forrand_chunk()which is probably similar with themodifcation()Kim mentioned in his patchset. The modification can be found here:
https://github.com/21cnbao/ltp/commit/7134413d747bfa9efThe test env is a x86 machine in which I have set memory size to 2.5GB and set zRAM to 2GB and disabled external disk swap.
with the vanilla kernel:
\time -v ./a.out -vv -t 4 -s 209715200 -S 200000so we have 10 chunks and 4 threads, each trunk is 209715200(200MB)
typical result:
Command being timed: "./a.out -vv -t 4 -s 209715200 -S 200000" User time (seconds): 36.19
System time (seconds): 229.72
Percent of CPU this job got: 371%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:11.59 Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2166196
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 9990128
Minor (reclaiming a frame) page faults: 33315945 Voluntary context switches: 59144
Involuntary context switches: 167754
Swaps: 0
File system inputs: 2760
File system outputs: 8
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0with
gen_lruand lru_gen/enabled=0x3:
typical result:
Command being timed: "./a.out -vv -t 4 -s 209715200 -S 200000" User time (seconds): 36.34
System time (seconds): 276.07
Percent of CPU this job got: 378%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:22.46 **** 15% time +
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2168120
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 13362810
***** 30% page fault +
Minor (reclaiming a frame) page faults: 33394617 Voluntary context switches: 55216
Involuntary context switches: 137220
Swaps: 0
File system inputs: 4088
File system outputs: 8
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0with
gen_lruand lru_gen/enabled=0x7:
typical result:
Command being timed: "./a.out -vv -t 4 -s 209715200 -S 200000" User time (seconds): 36.13
System time (seconds): 251.71
Percent of CPU this job got: 378%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:16.00 *****better than enabled=0x3, worse than vanilla Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2120988
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 12706512
Minor (reclaiming a frame) page faults: 33422243 Voluntary context switches: 49485
Involuntary context switches: 126765
Swaps: 0
File system inputs: 2976
File system outputs: 8
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
Quoted 7 lines
Quoted 4 lines
I can also reproduce the problem on arm64.
I am not saying this is going to block mglru from being mainlined. But I am still curious if this is an issue worth being addressed somehow in mglru.
You've missed something very important: *thoughput* :)
noop :-)
in the test case, there are 4 threads. they are searching a key in 10 chunks of memory. for each chunk, the size is 200MB.
a "random" chunk index is returned for those threads to search. but chunk2 is the hottest, and chunk3, 7, 4 are relatively hotter than others. static inline unsigned int rand_chunk(void)
{
/* simulate hot and cold chunk */
unsigned int rand[16] = {2, 2, 3, 4, 5, 2, 6, 7, 9, 2, 8, 3, 7, 2, 2, 4};
static int nr = 0;
return rand[nr++%16];
}each thread does search_mem():
static unsigned int search_mem(void)
{
record_t key, *found;
record_t *src, *copy;
unsigned int chunk;
size_t copy_size = chunk_size;
unsigned int i;
unsigned int state = 0;
/* run 160 loops or till timeout */
for (i = 0; threads_go == 1 && i < 160; i++) {
chunk = rand_chunk();
src = mem[chunk];
...
copy = alloc_mem(copy_size);
...
memcpy(copy, src, copy_size);
key = rand_num(copy_size / record_size, &state);
bsearch(&key, copy, copy_size / record_size,
record_size, compare);
/* Below check is mainly for memory corruption or other bug */
if (found == NULL) {
fprintf(stderr, "Couldn't find key %zd\n", key);
exit(1);
}
} /* end if ! touch_pages */
free_mem(copy, copy_size);
}
return (i);
}each thread picks up a chunk, then allocates a new memory and copies the chunk to the new allocated memory, and searches a key in the allocated memory.
as i have set time to rather big by -S, so each thread actually exits while it completes 160 loops.
$ \time -v ./ebizzy -t 4 -s $((200*1024*1024)) -S 6000000
so the one who finishes the whole jobs earlier wins in throughput as well.
Quoted 3 lines
Dollars to doughnuts there was a large increase in throughput -- I haven't tried this benchmark but I've seen many reports similar to this one.
I have no doubt about this. I am just trying to figure out some potential we can further achieve in mglru.
Thanks,
Barry
Re: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
Quoted 210 lines · Barry Song
Quoted 194 lines · Yu Zhao
Quoted 192 lines · Barry Song
Quoted 87 lines
Quoted 61 lines
Quoted 19 lines
Quoted 19 lines
Quoted 14 lines
We used to put a faulted file page in inactive, if we access it a second time, it can be promoted
to active. then in recent years, we have also applied this to anon pages while kernel adds
workingset protection for anon pages. so basically both anon and file pages go into the inactive
list for the 1st time, if we access it for the second time, they go to the active list. if we don't access
it any more, they are likely to be reclaimed as they are inactive. we do have some special fastpath for code section, executable file pages are kept on active list
as long as they are accessed.Yes.
Quoted 1 line
so all of the above concerns are actually not that correct?
Quoted 8 lines
Quoted 2 lines
They are valid concerns but I don't know any popular workloads that care about them.
Hi Yu,
here we can get a workload in Kim's patchset while he added workingset protection
for anon pages:
https://patchwork.kernel.org/project/linux-mm/cover/[email protected]/Thanks. I wouldn't call that a workload because it's not a real application. By popular workloads, I mean applications that the majority of people actually run on phones, in cloud, etc.
Quoted 3 lines
anon pages used to go to active rather than inactive, but kim's patchset moved to use inactive first. then only after the anon page is accessed second time, it can move to active.
Yes. To clarify, the A-bit doesn't really mean the first or second access. It can be many accesses each time it's set.
Quoted 22 lines
"In current implementation, newly created or swap-in anonymous page is
started on the active list. Growing the active list results in rebalancing active/inactive list so old pages on the active list are demoted to the inactive list. Hence, hot page on the active list isn't protected at all.
Following is an example of this situation.
Assume that 50 hot pages on active list and system can contain total 100 pages. Numbers denote the number of pages on active/inactive list (active | inactive). (h) stands for hot pages and (uo) stands for used-once pages.
1. 50 hot pages on active list
50(h) | 02. workload: 50 newly created (used-once) pages
50(uo) | 50(h)3. workload: another 50 newly created (used-once) pages 50(uo) | 50(uo), swap-out 50(h)
Quoted 6 lines
Quoted 3 lines
As we can see, hot pages are swapped-out and it would cause swap-in later."
Is MGLRU able to avoid the swap-out of the 50 hot pages?
I think the real question is why the 50 hot pages can be moved to the inactive list. If they are really hot, the A-bit should protect them.
This is a good question.
I guess it is probably because the current lru is trying to maintain a balance between the sizes of active and inactive lists. Thus, it can shrink active list even though pages might be still "hot" but not the recently accessed ones.
1. 50 hot pages on active list
50(h) | 02. workload: 50 newly created (used-once) pages
50(uo) | 50(h)3. workload: another 50 newly created (used-once) pages 50(uo) | 50(uo), swap-out 50(h)
the old kernel without anon workingset protection put workload 2 on active, so pushed 50 hot pages from active to inactive. workload 3 would further contribute to evict the 50 hot pages.
Quoted 7 lines
it seems mglru doesn't demote pages from the youngest generation to older generation only in order to balance the list size? so mglru is probably safe in these cases.
I will run some tests mentioned in Kim's patchset and report the result to you afterwards.
Hi Yu,
I did find putting faulted pages to the youngest generation lead to some regression in the case ebizzy Kim's patchset mentioned while he tried to support workingset protection for anon pages.
i did a little bit modification forrand_chunk()which is probably similar with themodifcation()Kim mentioned in his patchset. The modification can be found here:
https://github.com/21cnbao/ltp/commit/7134413d747bfa9efThe test env is a x86 machine in which I have set memory size to 2.5GB and set zRAM to 2GB and disabled external disk swap.
with the vanilla kernel:
\time -v ./a.out -vv -t 4 -s 209715200 -S 200000so we have 10 chunks and 4 threads, each trunk is 209715200(200MB)
typical result:
Command being timed: "./a.out -vv -t 4 -s 209715200 -S 200000" User time (seconds): 36.19
System time (seconds): 229.72
Percent of CPU this job got: 371%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:11.59 Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2166196
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 9990128
Minor (reclaiming a frame) page faults: 33315945 Voluntary context switches: 59144
Involuntary context switches: 167754
Swaps: 0
File system inputs: 2760
File system outputs: 8
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0with
gen_lruand lru_gen/enabled=0x3:
typical result:
Command being timed: "./a.out -vv -t 4 -s 209715200 -S 200000" User time (seconds): 36.34
System time (seconds): 276.07
Percent of CPU this job got: 378%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:22.46**** 15% time +Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2168120
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 13362810***** 30% page fault +Minor (reclaiming a frame) page faults: 33394617
Voluntary context switches: 55216
Involuntary context switches: 137220
Swaps: 0
File system inputs: 4088
File system outputs: 8
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0with
gen_lruand lru_gen/enabled=0x7:
typical result:
Command being timed: "./a.out -vv -t 4 -s 209715200 -S 200000" User time (seconds): 36.13
System time (seconds): 251.71
Percent of CPU this job got: 378%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:16.00*****better than enabled=0x3, worse than vanillaAverage shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2120988
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 12706512
Minor (reclaiming a frame) page faults: 33422243
Voluntary context switches: 49485
Involuntary context switches: 126765
Swaps: 0
File system inputs: 2976
File system outputs: 8
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0Quoted 7 lines
Quoted 4 lines
I can also reproduce the problem on arm64.
I am not saying this is going to block mglru from being mainlined. But I am still curious if this is an issue worth being addressed somehow in mglru.
You've missed something very important: *thoughput* :)
noop :-)
in the test case, there are 4 threads. they are searching a key in 10 chunks of memory. for each chunk, the size is 200MB.
a "random" chunk index is returned for those threads to search. but chunk2 is the hottest, and chunk3, 7, 4 are relatively hotter than others. static inline unsigned intrand_chunk(void)
Quoted 3 lines
{
/* simulate hot and cold chunk */
unsigned int rand[16] = {2, 2, 3, 4, 5, 2, 6, 7, 9, 2, 8, 3, 7, 2, 2, 4};This is sequential access, not what you claim above, because you have a repeating sequence.
In this case MGLRU is expected to be slower because it doesn't try to optimize it, as discussed before [1]. The reason is, with a manageable complexity, we can only optimize so many things. And MGLRU chose to optimize (arguably) popular workloads, since, AFAIK, no real-world applications streams anon memory.
To verify this is indeed sequential access, you could make rand[] larger, e.g., 160, with the same portions of 2s, 3s, 4s, etc, but their positions are random. The following change shows MGLRU is ~20% faster on my Snapdragon 7c + 2.5G DRAM + 2GB zram.
static inline unsigned int rand_chunk(void)
{
/* simulate hot and cold chunk */
- unsigned int rand[16] = {2, 2, 3, 4, 5, 2, 6, 7, 9, 2, 8, 3,
7, 2, 2, 4};
+ unsigned int rand[160] = {
+ 2, 4, 7, 3, 4, 2, 7, 2, 7, 8, 6, 9, 7, 6, 5, 4,
+ 6, 2, 6, 4, 2, 9, 2, 5, 5, 4, 7, 2, 7, 7, 5, 2,
+ 4, 4, 3, 3, 2, 4, 2, 2, 5, 2, 4, 2, 8, 2, 2, 3,
+ 2, 2, 2, 2, 2, 8, 4, 2, 2, 4, 2, 2, 2, 2, 3, 2,
+ 8, 5, 2, 2, 3, 2, 8, 2, 6, 2, 4, 8, 5, 2, 9, 2,
+ 8, 7, 9, 2, 4, 4, 3, 3, 2, 8, 2, 2, 3, 3, 2, 7,
+ 7, 5, 2, 2, 8, 2, 2, 2, 5, 2, 4, 3, 2, 3, 6, 3,
+ 3, 3, 9, 4, 2, 3, 9, 7, 7, 6, 2, 2, 4, 2, 6, 2,
+ 9, 7, 7, 7, 9, 3, 4, 2, 3, 2, 7, 3, 2, 2, 2, 6,
+ 8, 3, 7, 6, 2, 2, 2, 4, 7, 2, 5, 7, 4, 7, 9, 9,
+ };
static int nr = 0;- return rand[nr++%16];
+ return rand[nr++%160];
}
Yet better, you could use some standard benchmark suites, written by reputable organizations, e.g., memtier, YCSB, to generate more realistic distributions, as I've suggested before [2].
Quoted 6 lines
static int nr = 0; return rand[nr++%16]; }each thread does
search_mem():
static unsigned intsearch_mem(void)
Quoted 10 lines
{
record_t key, *found;
record_t *src, *copy;
unsigned int chunk;
size_t copy_size = chunk_size;
unsigned int i;
unsigned int state = 0;
/* run 160 loops or till timeout */
for (i = 0; threads_go == 1 && i < 160; i++) {I see you've modified the original benchmark. But with "-S 200000", should this test finish within an hour instead of the following?
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:11.59
Quoted 25 lines
chunk = rand_chunk();
src = mem[chunk];
...
copy = alloc_mem(copy_size);
...
memcpy(copy, src, copy_size);
key = rand_num(copy_size / record_size, &state);
bsearch(&key, copy, copy_size / record_size,
record_size, compare);
/* Below check is mainly for memory corruption or other bug */
if (found == NULL) {
fprintf(stderr, "Couldn't find key %zd\n", key);
exit(1);
}
} /* end if ! touch_pages */
free_mem(copy, copy_size);
}
return (i);
}Quoted 6 lines
each thread picks up a chunk, then allocates a new memory and copies the chunk to the new allocated memory, and searches a key in the allocated memory.
as i have set time to rather big by -S, so each thread actually exits while it completes 160 loops.
$ \time -v ./ebizzy -t 4 -s $((200*1024*1024)) -S 6000000
Ok, you actually used "-S 6000000".
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
Re: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
Quoted 302 lines · Yu Zhao
Quoted 210 lines · Barry Song
Quoted 194 lines · Yu Zhao
Quoted 192 lines · Barry Song
Quoted 87 lines
Quoted 61 lines
Quoted 19 lines
Quoted 19 lines
Quoted 14 lines
We used to put a faulted file page in inactive, if we access it a second time, it can be promoted
to active. then in recent years, we have also applied this to anon pages while kernel adds
workingset protection for anon pages. so basically both anon and file pages go into the inactive
list for the 1st time, if we access it for the second time, they go to the active list. if we don't access
it any more, they are likely to be reclaimed as they are inactive. we do have some special fastpath for code section, executable file pages are kept on active list
as long as they are accessed.Yes.
Quoted 1 line
so all of the above concerns are actually not that correct?
Quoted 8 lines
Quoted 2 lines
They are valid concerns but I don't know any popular workloads that care about them.
Hi Yu,
here we can get a workload in Kim's patchset while he added workingset protection
for anon pages:
https://patchwork.kernel.org/project/linux-mm/cover/[email protected]/Thanks. I wouldn't call that a workload because it's not a real application. By popular workloads, I mean applications that the majority of people actually run on phones, in cloud, etc.
Quoted 3 lines
anon pages used to go to active rather than inactive, but kim's patchset moved to use inactive first. then only after the anon page is accessed second time, it can move to active.
Yes. To clarify, the A-bit doesn't really mean the first or second access. It can be many accesses each time it's set.
Quoted 22 lines
"In current implementation, newly created or swap-in anonymous page is
started on the active list. Growing the active list results in rebalancing active/inactive list so old pages on the active list are demoted to the inactive list. Hence, hot page on the active list isn't protected at all.
Following is an example of this situation.
Assume that 50 hot pages on active list and system can contain total 100 pages. Numbers denote the number of pages on active/inactive list (active | inactive). (h) stands for hot pages and (uo) stands for used-once pages.
1. 50 hot pages on active list
50(h) | 02. workload: 50 newly created (used-once) pages
50(uo) | 50(h)3. workload: another 50 newly created (used-once) pages 50(uo) | 50(uo), swap-out 50(h)
Quoted 6 lines
Quoted 3 lines
As we can see, hot pages are swapped-out and it would cause swap-in later."
Is MGLRU able to avoid the swap-out of the 50 hot pages?
I think the real question is why the 50 hot pages can be moved to the inactive list. If they are really hot, the A-bit should protect them.
This is a good question.
I guess it is probably because the current lru is trying to maintain a balance between the sizes of active and inactive lists. Thus, it can shrink active list even though pages might be still "hot" but not the recently accessed ones.
1. 50 hot pages on active list
50(h) | 02. workload: 50 newly created (used-once) pages
50(uo) | 50(h)3. workload: another 50 newly created (used-once) pages 50(uo) | 50(uo), swap-out 50(h)
the old kernel without anon workingset protection put workload 2 on active, so pushed 50 hot pages from active to inactive. workload 3 would further contribute to evict the 50 hot pages.
Quoted 7 lines
it seems mglru doesn't demote pages from the youngest generation to older generation only in order to balance the list size? so mglru is probably safe in these cases.
I will run some tests mentioned in Kim's patchset and report the result to you afterwards.
Hi Yu,
I did find putting faulted pages to the youngest generation lead to some regression in the case ebizzy Kim's patchset mentioned while he tried to support workingset protection for anon pages.
i did a little bit modification forrand_chunk()which is probably similar with themodifcation()Kim mentioned in his patchset. The modification can be found here:
https://github.com/21cnbao/ltp/commit/7134413d747bfa9efThe test env is a x86 machine in which I have set memory size to 2.5GB and set zRAM to 2GB and disabled external disk swap.
with the vanilla kernel:
\time -v ./a.out -vv -t 4 -s 209715200 -S 200000so we have 10 chunks and 4 threads, each trunk is 209715200(200MB)
typical result:
Command being timed: "./a.out -vv -t 4 -s 209715200 -S 200000" User time (seconds): 36.19
System time (seconds): 229.72
Percent of CPU this job got: 371%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:11.59 Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2166196
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 9990128
Minor (reclaiming a frame) page faults: 33315945 Voluntary context switches: 59144
Involuntary context switches: 167754
Swaps: 0
File system inputs: 2760
File system outputs: 8
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0with
gen_lruand lru_gen/enabled=0x3:
typical result:
Command being timed: "./a.out -vv -t 4 -s 209715200 -S 200000" User time (seconds): 36.34
System time (seconds): 276.07
Percent of CPU this job got: 378%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:22.46**** 15% time +Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2168120
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 13362810***** 30% page fault +Minor (reclaiming a frame) page faults: 33394617
Voluntary context switches: 55216
Involuntary context switches: 137220
Swaps: 0
File system inputs: 4088
File system outputs: 8
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0with
gen_lruand lru_gen/enabled=0x7:
typical result:
Command being timed: "./a.out -vv -t 4 -s 209715200 -S 200000" User time (seconds): 36.13
System time (seconds): 251.71
Percent of CPU this job got: 378%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:16.00*****better than enabled=0x3, worse than vanillaAverage shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2120988
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 12706512
Minor (reclaiming a frame) page faults: 33422243
Voluntary context switches: 49485
Involuntary context switches: 126765
Swaps: 0
File system inputs: 2976
File system outputs: 8
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0Quoted 7 lines
Quoted 4 lines
I can also reproduce the problem on arm64.
I am not saying this is going to block mglru from being mainlined. But I am still curious if this is an issue worth being addressed somehow in mglru.
You've missed something very important: *thoughput* :)
noop :-)
in the test case, there are 4 threads. they are searching a key in 10 chunks of memory. for each chunk, the size is 200MB.
a "random" chunk index is returned for those threads to search. but chunk2 is the hottest, and chunk3, 7, 4 are relatively hotter than others. static inline unsigned intrand_chunk(void)Quoted 3 lines
{ /* simulate hot and cold chunk */ unsigned int rand[16] = {2, 2, 3, 4, 5, 2, 6, 7, 9, 2, 8, 3, 7, 2, 2, 4};This is sequential access, not what you claim above, because you have a repeating sequence.
In this case MGLRU is expected to be slower because it doesn't try to optimize it, as discussed before [1]. The reason is, with a manageable complexity, we can only optimize so many things. And MGLRU chose to optimize (arguably) popular workloads, since, AFAIK, no real-world applications streams anon memory.
To verify this is indeed sequential access, you could make rand[] larger, e.g., 160, with the same portions of 2s, 3s, 4s, etc, but their positions are random. The following change shows MGLRU is ~20% faster on my Snapdragon 7c + 2.5G DRAM + 2GB zram.
static inline unsigned int
rand_chunk(void){ /* simulate hot and cold chunk */- unsigned int rand[16] = {2, 2, 3, 4, 5, 2, 6, 7, 9, 2, 8, 3,
7, 2, 2, 4};(quoted diff) +13
-1+ unsigned int rand[160] = { + 2, 4, 7, 3, 4, 2, 7, 2, 7, 8, 6, 9, 7, 6, 5, 4, + 6, 2, 6, 4, 2, 9, 2, 5, 5, 4, 7, 2, 7, 7, 5, 2, + 4, 4, 3, 3, 2, 4, 2, 2, 5, 2, 4, 2, 8, 2, 2, 3, + 2, 2, 2, 2, 2, 8, 4, 2, 2, 4, 2, 2, 2, 2, 3, 2, + 8, 5, 2, 2, 3, 2, 8, 2, 6, 2, 4, 8, 5, 2, 9, 2, + 8, 7, 9, 2, 4, 4, 3, 3, 2, 8, 2, 2, 3, 3, 2, 7, + 7, 5, 2, 2, 8, 2, 2, 2, 5, 2, 4, 3, 2, 3, 6, 3, + 3, 3, 9, 4, 2, 3, 9, 7, 7, 6, 2, 2, 4, 2, 6, 2, + 9, 7, 7, 7, 9, 3, 4, 2, 3, 2, 7, 3, 2, 2, 2, 6, + 8, 3, 7, 6, 2, 2, 2, 4, 7, 2, 5, 7, 4, 7, 9, 9, + }; static int nr = 0; - return rand[nr++%16]; + return rand[nr++%160]; }Yet better, you could use some standard benchmark suites, written by reputable organizations, e.g., memtier, YCSB, to generate more realistic distributions, as I've suggested before [2].
Quoted 6 lines
static int nr = 0; return rand[nr++%16]; }each thread does
search_mem():
static unsigned intsearch_mem(void)Quoted 10 lines
{ record_t key, *found; record_t *src, *copy; unsigned int chunk; size_t copy_size = chunk_size; unsigned int i; unsigned int state = 0; /* run 160 loops or till timeout */ for (i = 0; threads_go == 1 && i < 160; i++) {I see you've modified the original benchmark. But with "-S 200000", should this test finish within an hour instead of the following?
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:11.59Quoted 25 lines
chunk = rand_chunk(); src = mem[chunk]; ... copy = alloc_mem(copy_size); ... memcpy(copy, src, copy_size); key = rand_num(copy_size / record_size, &state); bsearch(&key, copy, copy_size / record_size, record_size, compare); /* Below check is mainly for memory corruption or other bug */ if (found == NULL) { fprintf(stderr, "Couldn't find key %zd\n", key); exit(1); } } /* end if ! touch_pages */ free_mem(copy, copy_size); } return (i); }
Quoted 8 lines
Quoted 6 lines
each thread picks up a chunk, then allocates a new memory and copies the chunk to the new allocated memory, and searches a key in the allocated memory.
as i have set time to rather big by -S, so each thread actually exits while it completes 160 loops.
$ \time -v ./ebizzy -t 4 -s $((200*1024*1024)) -S 6000000Ok, you actually used "-S 6000000".
I have two exits, either 160 loops have been done or -S gets timeout. Since -S is very big, the process exits from the completion of 160 loops.
I am seeing mglru is getting very similar speed with vanilla lru by using your rand_chunk() with 160 entries. the command is like:
\time -v ./a.out -t 4 -s $((200*1024*1024)) -S 600000 -m
The time to complete jobs begins to be more random, but on average, mglru seems to be 5% faster. actually, i am seeing mglru can be faster than vanilla even with more page faults. for example,
MGLRU:
Command being timed: "./mt.out -t 4 -s 209715200 -S 600000 -m" User time (seconds): 32.68
System time (seconds): 227.19
Percent of CPU this job got: 370%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:10.23 Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2175292
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 10977244
Minor (reclaiming a frame) page faults: 33447638 Voluntary context switches: 44466
Involuntary context switches: 108413
Swaps: 0
File system inputs: 7704
File system outputs: 8
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
VANILLA:
Command being timed: "./mt.out -t 4 -s 209715200 -S 600000 -m" User time (seconds): 32.20
System time (seconds): 248.18
Percent of CPU this job got: 371%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:15.55 Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2174384
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 10002206
Minor (reclaiming a frame) page faults: 33392151 Voluntary context switches: 76966
Involuntary context switches: 184841
Swaps: 0
File system inputs: 2032
File system outputs: 8
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
I guess the main cause of the regression for the previous sequence with 16 entries is that the ebizzy has a new allocated copy in search_mem(), which is mapped and used only once in each loop. and the temp copy can push out those hot chunks.
Anyway, I understand it is a trade-off between warmly embracing new pages and holding old pages tightly. Real user cases from phone, server, desktop will be judging this better.
Quoted 3 lines
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
Thanks
Barry
Re: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
Quoted 367 lines · Barry Song
Quoted 302 lines · Yu Zhao
Quoted 210 lines · Barry Song
Quoted 194 lines · Yu Zhao
Quoted 192 lines · Barry Song
Quoted 87 lines
Quoted 61 lines
Quoted 19 lines
Quoted 19 lines
Quoted 14 lines
We used to put a faulted file page in inactive, if we access it a second time, it can be promoted
to active. then in recent years, we have also applied this to anon pages while kernel adds
workingset protection for anon pages. so basically both anon and file pages go into the inactive
list for the 1st time, if we access it for the second time, they go to the active list. if we don't access
it any more, they are likely to be reclaimed as they are inactive. we do have some special fastpath for code section, executable file pages are kept on active list
as long as they are accessed.Yes.
Quoted 1 line
so all of the above concerns are actually not that correct?
Quoted 8 lines
Quoted 2 lines
They are valid concerns but I don't know any popular workloads that care about them.
Hi Yu,
here we can get a workload in Kim's patchset while he added workingset protection
for anon pages:
https://patchwork.kernel.org/project/linux-mm/cover/[email protected]/Thanks. I wouldn't call that a workload because it's not a real application. By popular workloads, I mean applications that the majority of people actually run on phones, in cloud, etc.
Quoted 3 lines
anon pages used to go to active rather than inactive, but kim's patchset moved to use inactive first. then only after the anon page is accessed second time, it can move to active.
Yes. To clarify, the A-bit doesn't really mean the first or second access. It can be many accesses each time it's set.
Quoted 22 lines
"In current implementation, newly created or swap-in anonymous page is
started on the active list. Growing the active list results in rebalancing active/inactive list so old pages on the active list are demoted to the inactive list. Hence, hot page on the active list isn't protected at all.
Following is an example of this situation.
Assume that 50 hot pages on active list and system can contain total 100 pages. Numbers denote the number of pages on active/inactive list (active | inactive). (h) stands for hot pages and (uo) stands for used-once pages.
1. 50 hot pages on active list
50(h) | 02. workload: 50 newly created (used-once) pages
50(uo) | 50(h)3. workload: another 50 newly created (used-once) pages 50(uo) | 50(uo), swap-out 50(h)
Quoted 6 lines
Quoted 3 lines
As we can see, hot pages are swapped-out and it would cause swap-in later."
Is MGLRU able to avoid the swap-out of the 50 hot pages?
I think the real question is why the 50 hot pages can be moved to the inactive list. If they are really hot, the A-bit should protect them.
This is a good question.
I guess it is probably because the current lru is trying to maintain a balance between the sizes of active and inactive lists. Thus, it can shrink active list even though pages might be still "hot" but not the recently accessed ones.
1. 50 hot pages on active list
50(h) | 02. workload: 50 newly created (used-once) pages
50(uo) | 50(h)3. workload: another 50 newly created (used-once) pages 50(uo) | 50(uo), swap-out 50(h)
the old kernel without anon workingset protection put workload 2 on active, so pushed 50 hot pages from active to inactive. workload 3 would further contribute to evict the 50 hot pages.
Quoted 7 lines
it seems mglru doesn't demote pages from the youngest generation to older generation only in order to balance the list size? so mglru is probably safe in these cases.
I will run some tests mentioned in Kim's patchset and report the result to you afterwards.
Hi Yu,
I did find putting faulted pages to the youngest generation lead to some regression in the case ebizzy Kim's patchset mentioned while he tried to support workingset protection for anon pages.
i did a little bit modification forrand_chunk()which is probably similar with themodifcation()Kim mentioned in his patchset. The modification can be found here:
https://github.com/21cnbao/ltp/commit/7134413d747bfa9efThe test env is a x86 machine in which I have set memory size to 2.5GB and set zRAM to 2GB and disabled external disk swap.
with the vanilla kernel:
\time -v ./a.out -vv -t 4 -s 209715200 -S 200000so we have 10 chunks and 4 threads, each trunk is 209715200(200MB)
typical result:
Command being timed: "./a.out -vv -t 4 -s 209715200 -S 200000" User time (seconds): 36.19
System time (seconds): 229.72
Percent of CPU this job got: 371%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:11.59 Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2166196
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 9990128
Minor (reclaiming a frame) page faults: 33315945 Voluntary context switches: 59144
Involuntary context switches: 167754
Swaps: 0
File system inputs: 2760
File system outputs: 8
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0with
gen_lruand lru_gen/enabled=0x3:
typical result:
Command being timed: "./a.out -vv -t 4 -s 209715200 -S 200000" User time (seconds): 36.34
System time (seconds): 276.07
Percent of CPU this job got: 378%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:22.46**** 15% time +Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2168120
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 13362810***** 30% page fault +Minor (reclaiming a frame) page faults: 33394617
Voluntary context switches: 55216
Involuntary context switches: 137220
Swaps: 0
File system inputs: 4088
File system outputs: 8
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0with
gen_lruand lru_gen/enabled=0x7:
typical result:
Command being timed: "./a.out -vv -t 4 -s 209715200 -S 200000" User time (seconds): 36.13
System time (seconds): 251.71
Percent of CPU this job got: 378%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:16.00*****better than enabled=0x3, worse than vanillaAverage shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2120988
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 12706512
Minor (reclaiming a frame) page faults: 33422243
Voluntary context switches: 49485
Involuntary context switches: 126765
Swaps: 0
File system inputs: 2976
File system outputs: 8
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0Quoted 7 lines
Quoted 4 lines
I can also reproduce the problem on arm64.
I am not saying this is going to block mglru from being mainlined. But I am still curious if this is an issue worth being addressed somehow in mglru.
You've missed something very important: *thoughput* :)
noop :-)
in the test case, there are 4 threads. they are searching a key in 10 chunks of memory. for each chunk, the size is 200MB.
a "random" chunk index is returned for those threads to search. but chunk2 is the hottest, and chunk3, 7, 4 are relatively hotter than others. static inline unsigned intrand_chunk(void)Quoted 3 lines
{ /* simulate hot and cold chunk */ unsigned int rand[16] = {2, 2, 3, 4, 5, 2, 6, 7, 9, 2, 8, 3, 7, 2, 2, 4};This is sequential access, not what you claim above, because you have a repeating sequence.
In this case MGLRU is expected to be slower because it doesn't try to optimize it, as discussed before [1]. The reason is, with a manageable complexity, we can only optimize so many things. And MGLRU chose to optimize (arguably) popular workloads, since, AFAIK, no real-world applications streams anon memory.
To verify this is indeed sequential access, you could make rand[] larger, e.g., 160, with the same portions of 2s, 3s, 4s, etc, but their positions are random. The following change shows MGLRU is ~20% faster on my Snapdragon 7c + 2.5G DRAM + 2GB zram.
static inline unsigned int
rand_chunk(void){ /* simulate hot and cold chunk */- unsigned int rand[16] = {2, 2, 3, 4, 5, 2, 6, 7, 9, 2, 8, 3,
7, 2, 2, 4};(quoted diff) +13
-1+ unsigned int rand[160] = { + 2, 4, 7, 3, 4, 2, 7, 2, 7, 8, 6, 9, 7, 6, 5, 4, + 6, 2, 6, 4, 2, 9, 2, 5, 5, 4, 7, 2, 7, 7, 5, 2, + 4, 4, 3, 3, 2, 4, 2, 2, 5, 2, 4, 2, 8, 2, 2, 3, + 2, 2, 2, 2, 2, 8, 4, 2, 2, 4, 2, 2, 2, 2, 3, 2, + 8, 5, 2, 2, 3, 2, 8, 2, 6, 2, 4, 8, 5, 2, 9, 2, + 8, 7, 9, 2, 4, 4, 3, 3, 2, 8, 2, 2, 3, 3, 2, 7, + 7, 5, 2, 2, 8, 2, 2, 2, 5, 2, 4, 3, 2, 3, 6, 3, + 3, 3, 9, 4, 2, 3, 9, 7, 7, 6, 2, 2, 4, 2, 6, 2, + 9, 7, 7, 7, 9, 3, 4, 2, 3, 2, 7, 3, 2, 2, 2, 6, + 8, 3, 7, 6, 2, 2, 2, 4, 7, 2, 5, 7, 4, 7, 9, 9, + }; static int nr = 0; - return rand[nr++%16]; + return rand[nr++%160]; }Yet better, you could use some standard benchmark suites, written by reputable organizations, e.g., memtier, YCSB, to generate more realistic distributions, as I've suggested before [2].
Quoted 6 lines
static int nr = 0; return rand[nr++%16]; }each thread does
search_mem():
static unsigned intsearch_mem(void)Quoted 10 lines
{ record_t key, *found; record_t *src, *copy; unsigned int chunk; size_t copy_size = chunk_size; unsigned int i; unsigned int state = 0; /* run 160 loops or till timeout */ for (i = 0; threads_go == 1 && i < 160; i++) {I see you've modified the original benchmark. But with "-S 200000", should this test finish within an hour instead of the following?
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:11.59Quoted 25 lines
chunk = rand_chunk(); src = mem[chunk]; ... copy = alloc_mem(copy_size); ... memcpy(copy, src, copy_size); key = rand_num(copy_size / record_size, &state); bsearch(&key, copy, copy_size / record_size, record_size, compare); /* Below check is mainly for memory corruption or other bug */ if (found == NULL) { fprintf(stderr, "Couldn't find key %zd\n", key); exit(1); } } /* end if ! touch_pages */ free_mem(copy, copy_size); } return (i); }Quoted 8 lines
Quoted 6 lines
each thread picks up a chunk, then allocates a new memory and copies the chunk to the new allocated memory, and searches a key in the allocated memory.
as i have set time to rather big by -S, so each thread actually exits while it completes 160 loops.
$ \time -v ./ebizzy -t 4 -s $((200*1024*1024)) -S 6000000Ok, you actually used "-S 6000000".
I have two exits, either 160 loops have been done or -S gets timeout. Since -S is very big, the process exits from the completion of 160 loops.
I am seeing mglru is getting very similar speed with vanilla lru by using your
rand_chunk()with 160 entries. the command is like:
\time -v ./a.out -t 4 -s $((200*1024*1024)) -S 600000 -mThe time to complete jobs begins to be more random, but on average, mglru seems to be 5% faster. actually, i am seeing mglru can be faster than vanilla even with more page faults. for example,
MGLRU:
Command being timed: "./mt.out -t 4 -s 209715200 -S 600000 -m" User time (seconds): 32.68
System time (seconds): 227.19
Percent of CPU this job got: 370%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:10.23 Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2175292
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 10977244
Minor (reclaiming a frame) page faults: 33447638 Voluntary context switches: 44466
Involuntary context switches: 108413
Swaps: 0
File system inputs: 7704
File system outputs: 8
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0VANILLA:
Command being timed: "./mt.out -t 4 -s 209715200 -S 600000 -m" User time (seconds): 32.20
System time (seconds): 248.18
Percent of CPU this job got: 371%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:15.55 Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2174384
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 10002206
Minor (reclaiming a frame) page faults: 33392151 Voluntary context switches: 76966
Involuntary context switches: 184841
Quoted 9 lines
Swaps: 0
File system inputs: 2032
File system outputs: 8
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
basically a perf comparison:
vanilla:
23.81% [lz4_compress] [k] LZ4_compress_fast_extState
14.15% [kernel] [k] LZ4_decompress_safe
10.48% libc-2.33.so [.] __memmove_avx_unaligned_erms
2.49% [kernel] [k] native_queued_spin_lock_slowpath
2.05% [kernel] [k] clear_page_erms
1.69% [kernel] [k] native_irq_return_iret
1.49% [kernel] [k] mem_cgroup_css_rstat_flush
1.05% [kernel] [k] _raw_spin_lock
1.05% [kernel] [k] sync_regs
1.00% [kernel] [k] smp_call_function_many_cond
0.97% [kernel] [k] memset_erms
0.95% [zram] [k] zram_bvec_rw.constprop.0
0.91% [kernel] [k] down_read_trylock
0.90% [kernel] [k] memcpy_erms
0.89% [zram] [k] __zram_bvec_read.constprop.0
0.88% [kernel] [k] psi_group_change
0.84% [kernel] [k] isolate_lru_pages
0.78% [kernel] [k] zs_map_object
0.76% [kernel] [k] __handle_mm_fault
0.72% [kernel] [k] page_vma_mapped_walkmglru:
23.43% [lz4_compress] [k] LZ4_compress_fast_extState
16.90% [kernel] [k] LZ4_decompress_safe
12.60% libc-2.33.so [.] __memmove_avx_unaligned_erms
2.26% [kernel] [k] clear_page_erms
2.06% [kernel] [k] native_queued_spin_lock_slowpath
1.77% [kernel] [k] native_irq_return_iret
1.18% [kernel] [k] sync_regs
1.12% [zram] [k] __zram_bvec_read.constprop.0
0.98% [kernel] [k] psi_group_change
0.97% [zram] [k] zram_bvec_rw.constprop.0
0.96% [kernel] [k] memset_erms
0.95% [kernel] [k] isolate_folios
0.92% [kernel] [k] zs_map_object
0.92% [kernel] [k] _raw_spin_lock
0.87% [kernel] [k] memcpy_erms
0.83% [kernel] [k] smp_call_function_many_cond
0.83% [kernel] [k] __handle_mm_fault
0.78% [kernel] [k] unmap_page_range
0.71% [kernel] [k] rmqueue_bulk
0.70% [kernel] [k] page_counter_unchargeit seems vanilla kernel puts more time on native_queued_spin_lock_slowpath(), down_read_trylock(), mem_cgroup_css_rstat_flush(), isolate_lru_pages() and page_vma_mapped_walk(), but mglru puts more time on decompress, memmove and isolate_folios().
That is probably why mglru can be a bit faster even with more major page faults.
Quoted 14 lines
I guess the main cause of the regression for the previous sequence with 16 entries is that the ebizzy has a new allocated copy in
search_mem(), which is mapped and used only once in each loop. and the temp copy can push out those hot chunks.Anyway, I understand it is a trade-off between warmly embracing new pages and holding old pages tightly. Real user cases from phone, server, desktop will be judging this better.
Thanks
Barry
Re: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
On Tue, Mar 15, 2022 at 4:29 AM Barry Song <[email protected]> wrote:
<snipped>
Quoted 8 lines
Quoted 8 lines
I guess the main cause of the regression for the previous sequence with 16 entries is that the ebizzy has a new allocated copy in
search_mem(), which is mapped and used only once in each loop. and the temp copy can push out those hot chunks.Anyway, I understand it is a trade-off between warmly embracing new pages and holding old pages tightly. Real user cases from phone, server, desktop will be judging this better.
Thanks for all the details. I looked into them today and found no regressions when running with your original program.
After I explain why, I hope you'd be convinced that using programs like this one is not a good way to measure things :)
Problems:
1) Given the 2.5GB configuration and a sequence of cold/hot chunks, I assume your program tries to simulate a handful of apps running on a phone. A short repeating sequence is closer to sequential access than to real user behaviors, as I suggested last time. You could check out how something similar is done here [1].
2) Under the same assumption (phone), C programs are very different from Android apps in terms of runtime memory behaviors, e.g., JVM GC [2].
3) Assuming you are interested in the runtime memory behavior of C/C++ programs, your program is still not very representative. All C/C++ programs I'm familiar with choose to link against TCmalloc, jemalloc or implement their own allocators. GNU libc, IMO, has a small market share nowadays.
4) TCmalloc/jemalloc are not only optimized for multithreading, they are also THP aware. THP is very important when benchmarking page reclaim, e.g., two similarly warm THPs can comprise 511+1 or 1+511 of warm+cold 4K pages. The LRU algorithm that chooses more of the former is at the disadvantage. Unless it's recommended by the applications you are trying to benchmark, THP should be disabled. (Android generally doesn't use THP.)
5) Swap devices are also important. Zram should NOT be used unless you know your benchmark doesn't generate incompressible data. The LRU algorithm that chooses more incompressible pages is at disadvantage.
Here is my result: on the same Snapdragon 7c + 2.5GB RAM + 1.5GB ramdisk swap, with your original program compiled against libc malloc and TCMalloc, to 32-bit and 64-bit binaries:
# cat /sys/kernel/mm/lru_gen/enabled 0x0003 # cat /sys/kernel/mm/transparent_hugepage/enabled always madvise [never]
# modprobe brd rd_nr=1 rd_size=1572864 # if=/dev/zero of=/dev/ram0 bs=1M # mkswap /dev/ram0 # swapoff -a # swapon /dev/ram0
# ldd test_absl_32
linux-vdso.so.1 (0xf6e7f000)
libabsl_malloc.so.2103.0.1 =>
/usr/lib/libabsl_malloc.so.2103.0.1 (0xf6e23000)
libpthread.so.0 => /lib/libpthread.so.0 (0xf6dff000)
libc.so.6 => /lib/libc.so.6 (0xf6d07000)
/lib/ld-linux-armhf.so.3 (0x09df0000)
libabsl_base.so.2103.0.1 => /usr/lib/libabsl_base.so.2103.0.1
(0xf6ce5000)
libabsl_raw_logging.so.2103.0.1 =>
/usr/lib/libabsl_raw_logging.so.2103.0.1 (0xf6cc4000)
libabsl_spinlock_wait.so.2103.0.1 =>
/usr/lib/libabsl_spinlock_wait.so.2103.0.1 (0xf6ca3000)
libc++.so.1 => /usr/lib/libc++.so.1 (0xf6c04000)
libc++abi.so.1 => /usr/lib/libc++abi.so.1 (0xf6bcd000)
# file test_absl_64
test_absl_64: ELF 64-bit LSB executable, ARM aarch64, version 1
(SYSV), statically linked
# ldd test_gnu_32
linux-vdso.so.1 (0xeabef000)
libpthread.so.0 => /lib/libpthread.so.0 (0xeab92000)
libc.so.6 => /lib/libc.so.6 (0xeaa9a000)
/lib/ld-linux-armhf.so.3 (0x05690000)
# file test_gnu_64
test_gnu_64: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV),
statically linked### baseline 5.17-rc8
# perf record ./test_gnu_64 -t 4 -s $((200*1024*1024)) -S 6000000 10 records/s real 59.00 s user 39.83 s sys 174.18 s
18.51% [.] memcpy
15.98% [k] __pi_clear_page
5.59% [k] rmqueue_pcplist
5.19% [k] do_raw_spin_lock
5.09% [k] memmove
4.60% [k] _raw_spin_unlock_irq
3.62% [k] _raw_spin_unlock_irqrestore
3.61% [k] free_unref_page_list
3.29% [k] zap_pte_range
2.53% [k] local_daif_restore
2.50% [k] down_read_trylock
1.41% [k] handle_mm_fault
1.32% [k] do_anonymous_page
1.31% [k] up_read
1.03% [k] free_swap_cache### MGLRU v9
# perf record ./test_gnu_64 -t 4 -s $((200*1024*1024)) -S 6000000 11 records/s real 57.00 s user 39.39 s
19.36% [.] memcpy
16.50% [k] __pi_clear_page
6.21% [k] memmove
5.57% [k] rmqueue_pcplist
5.07% [k] do_raw_spin_lock
4.96% [k] _raw_spin_unlock_irqrestore
4.25% [k] free_unref_page_list
3.80% [k] zap_pte_range
3.69% [k] _raw_spin_unlock_irq
2.71% [k] local_daif_restore
2.10% [k] down_read_trylock
1.50% [k] handle_mm_fault
1.29% [k] do_anonymous_page
1.17% [k] free_swap_cache
1.08% [k] up_read[1] https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/refs/heads/main/src/chromiumos/tast/local/memory/mempressure/mempressure.go
[2] https://developer.android.com/topic/performance/memory-overview
Re: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
Quoted 14 lines · Yu Zhao
On Tue, Mar 15, 2022 at 4:29 AM Barry Song <[email protected]> wrote:
<snipped>
Quoted 8 lines
Quoted 8 lines
I guess the main cause of the regression for the previous sequence with 16 entries is that the ebizzy has a new allocated copy in
search_mem(), which is mapped and used only once in each loop. and the temp copy can push out those hot chunks.Anyway, I understand it is a trade-off between warmly embracing new pages and holding old pages tightly. Real user cases from phone, server, desktop will be judging this better.
Quoted 6 lines
Thanks for all the details. I looked into them today and found no regressions when running with your original program.
After I explain why, I hope you'd be convinced that using programs like this one is not a good way to measure things :)
Yep. I agree ebizzy might not be a good one to measure things. I chose it only because Kim's patchset which moved anon pages to inactive at the first detected access was using it. Before kim's patchset, anon pages were placed in the active list from the first beginning:
https://patchwork.kernel.org/project/linux-mm/cover/[email protected]/
in ebizzy, there is a used-once allocated memory in each search_mem(). I guess that is why Kim's patchset chose it.
Quoted 16 lines
Problems:
1) Given the 2.5GB configuration and a sequence of cold/hot chunks, I assume your program tries to simulate a handful of apps running on a phone. A short repeating sequence is closer to sequential access than to real user behaviors, as I suggested last time. You could check out how something similar is done here [1].
2) Under the same assumption (phone), C programs are very different from Android apps in terms of runtime memory behaviors, e.g., JVM GC [2].
3) Assuming you are interested in the runtime memory behavior of C/C++ programs, your program is still not very representative. All C/C++ programs I'm familiar with choose to link against TCmalloc, jemalloc or implement their own allocators. GNU libc, IMO, has a small market share nowadays.
4) TCmalloc/jemalloc are not only optimized for multithreading, they are also THP aware. THP is very important when benchmarking page
Quoted 9 lines
reclaim, e.g., two similarly warm THPs can comprise 511+1 or 1+511 of warm+cold 4K pages. The LRU algorithm that chooses more of the former is at the disadvantage. Unless it's recommended by the applications you are trying to benchmark, THP should be disabled. (Android generally doesn't use THP.)
5) Swap devices are also important. Zram should NOT be used unless you know your benchmark doesn't generate incompressible data. The LRU algorithm that chooses more incompressible pages is at disadvantage.
Thanks for all the information above. very useful.
Quoted 3 lines
Here is my result: on the same Snapdragon 7c + 2.5GB RAM + 1.5GB ramdisk swap, with your original program compiled against libc malloc and TCMalloc, to 32-bit and 64-bit binaries:
I noticed an important difference is that you are using ramdisk, so there is no cost on "i/o". I assume compression/decompression is the i/o cost to zRAM.
Quoted 70 lines
# cat /sys/kernel/mm/lru_gen/enabled 0x0003 # cat /sys/kernel/mm/transparent_hugepage/enabled always madvise [never]# modprobe brd rd_nr=1 rd_size=1572864 # if=/dev/zero of=/dev/ram0 bs=1M # mkswap /dev/ram0 # swapoff -a # swapon /dev/ram0# ldd test_absl_32 linux-vdso.so.1 (0xf6e7f000) libabsl_malloc.so.2103.0.1 => /usr/lib/libabsl_malloc.so.2103.0.1 (0xf6e23000) libpthread.so.0 => /lib/libpthread.so.0 (0xf6dff000) libc.so.6 => /lib/libc.so.6 (0xf6d07000) /lib/ld-linux-armhf.so.3 (0x09df0000) libabsl_base.so.2103.0.1 => /usr/lib/libabsl_base.so.2103.0.1 (0xf6ce5000) libabsl_raw_logging.so.2103.0.1 => /usr/lib/libabsl_raw_logging.so.2103.0.1 (0xf6cc4000) libabsl_spinlock_wait.so.2103.0.1 => /usr/lib/libabsl_spinlock_wait.so.2103.0.1 (0xf6ca3000) libc++.so.1 => /usr/lib/libc++.so.1 (0xf6c04000) libc++abi.so.1 => /usr/lib/libc++abi.so.1 (0xf6bcd000) # file test_absl_64 test_absl_64: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked # ldd test_gnu_32 linux-vdso.so.1 (0xeabef000) libpthread.so.0 => /lib/libpthread.so.0 (0xeab92000) libc.so.6 => /lib/libc.so.6 (0xeaa9a000) /lib/ld-linux-armhf.so.3 (0x05690000) # file test_gnu_64 test_gnu_64: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked### baseline 5.17-rc8
# perf record ./test_gnu_64 -t 4 -s $((200*1024*1024)) -S 6000000 10 records/s real 59.00 s user 39.83 s sys 174.18 s18.51% [.] memcpy 15.98% [k] __pi_clear_page 5.59% [k] rmqueue_pcplist 5.19% [k] do_raw_spin_lock 5.09% [k] memmove 4.60% [k] _raw_spin_unlock_irq 3.62% [k] _raw_spin_unlock_irqrestore 3.61% [k] free_unref_page_list 3.29% [k] zap_pte_range 2.53% [k] local_daif_restore 2.50% [k] down_read_trylock 1.41% [k] handle_mm_fault 1.32% [k] do_anonymous_page 1.31% [k] up_read 1.03% [k] free_swap_cache### MGLRU v9
# perf record ./test_gnu_64 -t 4 -s $((200*1024*1024)) -S 6000000 11 records/s real 57.00 s user 39.39 s
Quoted 16 lines
19.36% [.] memcpy
16.50% [k] __pi_clear_page
6.21% [k] memmove
5.57% [k] rmqueue_pcplist
5.07% [k] do_raw_spin_lock
4.96% [k] _raw_spin_unlock_irqrestore
4.25% [k] free_unref_page_list
3.80% [k] zap_pte_range
3.69% [k] _raw_spin_unlock_irq
2.71% [k] local_daif_restore
2.10% [k] down_read_trylock
1.50% [k] handle_mm_fault
1.29% [k] do_anonymous_page
1.17% [k] free_swap_cache
1.08% [k] up_readI think your result is right. but if you take a look at the number of major faults, will you find mglru have more page faults?
i ask this question because i can see mglru even wins with lower hit ratio in the previous report I sent.
Quoted 2 lines
[1] https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/refs/heads/main/src/chromiumos/tast/local/memory/mempressure/mempressure.go
[2] https://developer.android.com/topic/performance/memory-overview
Thanks
Barry
Re: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
Quoted 62 lines · Barry Song
Quoted 14 lines · Yu Zhao
On Tue, Mar 15, 2022 at 4:29 AM Barry Song <[email protected]> wrote:
<snipped>
Quoted 8 lines
Quoted 8 lines
I guess the main cause of the regression for the previous sequence with 16 entries is that the ebizzy has a new allocated copy in
search_mem(), which is mapped and used only once in each loop. and the temp copy can push out those hot chunks.Anyway, I understand it is a trade-off between warmly embracing new pages and holding old pages tightly. Real user cases from phone, server, desktop will be judging this better.
Quoted 6 lines
Thanks for all the details. I looked into them today and found no regressions when running with your original program.
After I explain why, I hope you'd be convinced that using programs like this one is not a good way to measure things :)
Yep. I agree ebizzy might not be a good one to measure things. I chose it only because Kim's patchset which moved anon pages to inactive at the first detected access was using it. Before kim's patchset, anon pages were placed in the active list from the first beginning:
https://patchwork.kernel.org/project/linux-mm/cover/[email protected]/in ebizzy, there is a used-once allocated memory in each
search_mem(). I guess that is why Kim's patchset chose it.Quoted 16 lines
Problems:
1) Given the 2.5GB configuration and a sequence of cold/hot chunks, I assume your program tries to simulate a handful of apps running on a phone. A short repeating sequence is closer to sequential access than to real user behaviors, as I suggested last time. You could check out how something similar is done here [1].
2) Under the same assumption (phone), C programs are very different from Android apps in terms of runtime memory behaviors, e.g., JVM GC [2].
3) Assuming you are interested in the runtime memory behavior of C/C++ programs, your program is still not very representative. All C/C++ programs I'm familiar with choose to link against TCmalloc, jemalloc or implement their own allocators. GNU libc, IMO, has a small market share nowadays.
4) TCmalloc/jemalloc are not only optimized for multithreading, they are also THP aware. THP is very important when benchmarking pageQuoted 9 lines
reclaim, e.g., two similarly warm THPs can comprise 511+1 or 1+511 of warm+cold 4K pages. The LRU algorithm that chooses more of the former is at the disadvantage. Unless it's recommended by the applications you are trying to benchmark, THP should be disabled. (Android generally doesn't use THP.)
5) Swap devices are also important. Zram should NOT be used unless you know your benchmark doesn't generate incompressible data. The LRU algorithm that chooses more incompressible pages is at disadvantage.Thanks for all the information above. very useful.
Quoted 7 lines
Quoted 3 lines
Here is my result: on the same Snapdragon 7c + 2.5GB RAM + 1.5GB ramdisk swap, with your original program compiled against libc malloc and TCMalloc, to 32-bit and 64-bit binaries:
I noticed an important difference is that you are using ramdisk, so there is no cost on "i/o". I assume compression/decompression is the i/o cost to zRAM.
The cost is not the point; the fairness is:
1) Ramdisk is fair to both LRU algorithms.
2) Zram punishes the LRU algorithm that chooses incompressible pages. IOW, this algorithm needs to compress more pages in order to save the same amount of memory.
Quoted 86 lines
Quoted 69 lines
# cat /sys/kernel/mm/lru_gen/enabled 0x0003 # cat /sys/kernel/mm/transparent_hugepage/enabled always madvise [never]# modprobe brd rd_nr=1 rd_size=1572864 # if=/dev/zero of=/dev/ram0 bs=1M # mkswap /dev/ram0 # swapoff -a # swapon /dev/ram0# ldd test_absl_32 linux-vdso.so.1 (0xf6e7f000) libabsl_malloc.so.2103.0.1 => /usr/lib/libabsl_malloc.so.2103.0.1 (0xf6e23000) libpthread.so.0 => /lib/libpthread.so.0 (0xf6dff000) libc.so.6 => /lib/libc.so.6 (0xf6d07000) /lib/ld-linux-armhf.so.3 (0x09df0000) libabsl_base.so.2103.0.1 => /usr/lib/libabsl_base.so.2103.0.1 (0xf6ce5000) libabsl_raw_logging.so.2103.0.1 => /usr/lib/libabsl_raw_logging.so.2103.0.1 (0xf6cc4000) libabsl_spinlock_wait.so.2103.0.1 => /usr/lib/libabsl_spinlock_wait.so.2103.0.1 (0xf6ca3000) libc++.so.1 => /usr/lib/libc++.so.1 (0xf6c04000) libc++abi.so.1 => /usr/lib/libc++abi.so.1 (0xf6bcd000) # file test_absl_64 test_absl_64: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked # ldd test_gnu_32 linux-vdso.so.1 (0xeabef000) libpthread.so.0 => /lib/libpthread.so.0 (0xeab92000) libc.so.6 => /lib/libc.so.6 (0xeaa9a000) /lib/ld-linux-armhf.so.3 (0x05690000) # file test_gnu_64 test_gnu_64: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked### baseline 5.17-rc8
# perf record ./test_gnu_64 -t 4 -s $((200*1024*1024)) -S 6000000 10 records/s real 59.00 s user 39.83 s sys 174.18 s18.51% [.] memcpy 15.98% [k] __pi_clear_page 5.59% [k] rmqueue_pcplist 5.19% [k] do_raw_spin_lock 5.09% [k] memmove 4.60% [k] _raw_spin_unlock_irq 3.62% [k] _raw_spin_unlock_irqrestore 3.61% [k] free_unref_page_list 3.29% [k] zap_pte_range 2.53% [k] local_daif_restore 2.50% [k] down_read_trylock 1.41% [k] handle_mm_fault 1.32% [k] do_anonymous_page 1.31% [k] up_read 1.03% [k] free_swap_cache### MGLRU v9
# perf record ./test_gnu_64 -t 4 -s $((200*1024*1024)) -S 6000000 11 records/s real 57.00 s user 39.39 sQuoted 16 lines
19.36% [.] memcpy 16.50% [k] __pi_clear_page 6.21% [k] memmove 5.57% [k] rmqueue_pcplist 5.07% [k] do_raw_spin_lock 4.96% [k] _raw_spin_unlock_irqrestore 4.25% [k] free_unref_page_list 3.80% [k] zap_pte_range 3.69% [k] _raw_spin_unlock_irq 2.71% [k] local_daif_restore 2.10% [k] down_read_trylock 1.50% [k] handle_mm_fault 1.29% [k] do_anonymous_page 1.17% [k] free_swap_cache 1.08% [k] up_read
Quoted 4 lines
I think your result is right. but if you take a look at the number of major faults, will you find mglru have more page faults?
i ask this question because i can see mglru even wins with lower hit ratio in the previous report I sent.
Yes, I did see the elevated major faults:
# baseline total 11503878 majfault 4745116 pgsteal_kswapd 3056793 pgsteal_direct 3701969
# MGLRU total 11928659
pgmajfault 5762213pgsteal_kswapd 2098253pgsteal_direct 4068193
Re: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
Quoted 72 lines · Yu Zhao
Quoted 62 lines · Barry Song
Quoted 14 lines · Yu Zhao
On Tue, Mar 15, 2022 at 4:29 AM Barry Song <[email protected]> wrote:
<snipped>
Quoted 8 lines
Quoted 8 lines
I guess the main cause of the regression for the previous sequence with 16 entries is that the ebizzy has a new allocated copy in
search_mem(), which is mapped and used only once in each loop. and the temp copy can push out those hot chunks.Anyway, I understand it is a trade-off between warmly embracing new pages and holding old pages tightly. Real user cases from phone, server, desktop will be judging this better.
Quoted 6 lines
Thanks for all the details. I looked into them today and found no regressions when running with your original program.
After I explain why, I hope you'd be convinced that using programs like this one is not a good way to measure things :)
Yep. I agree ebizzy might not be a good one to measure things. I chose it only because Kim's patchset which moved anon pages to inactive at the first detected access was using it. Before kim's patchset, anon pages were placed in the active list from the first beginning:
https://patchwork.kernel.org/project/linux-mm/cover/[email protected]/in ebizzy, there is a used-once allocated memory in each
search_mem(). I guess that is why Kim's patchset chose it.Quoted 16 lines
Problems:
1) Given the 2.5GB configuration and a sequence of cold/hot chunks, I assume your program tries to simulate a handful of apps running on a phone. A short repeating sequence is closer to sequential access than to real user behaviors, as I suggested last time. You could check out how something similar is done here [1].
2) Under the same assumption (phone), C programs are very different from Android apps in terms of runtime memory behaviors, e.g., JVM GC [2].
3) Assuming you are interested in the runtime memory behavior of C/C++ programs, your program is still not very representative. All C/C++ programs I'm familiar with choose to link against TCmalloc, jemalloc or implement their own allocators. GNU libc, IMO, has a small market share nowadays.
4) TCmalloc/jemalloc are not only optimized for multithreading, they are also THP aware. THP is very important when benchmarking pageQuoted 9 lines
reclaim, e.g., two similarly warm THPs can comprise 511+1 or 1+511 of warm+cold 4K pages. The LRU algorithm that chooses more of the former is at the disadvantage. Unless it's recommended by the applications you are trying to benchmark, THP should be disabled. (Android generally doesn't use THP.)
5) Swap devices are also important. Zram should NOT be used unless you know your benchmark doesn't generate incompressible data. The LRU algorithm that chooses more incompressible pages is at disadvantage.Thanks for all the information above. very useful.
Quoted 7 lines
Quoted 3 lines
Here is my result: on the same Snapdragon 7c + 2.5GB RAM + 1.5GB ramdisk swap, with your original program compiled against libc malloc and TCMalloc, to 32-bit and 64-bit binaries:
I noticed an important difference is that you are using ramdisk, so there is no cost on "i/o". I assume compression/decompression is the i/o cost to zRAM.
Quoted 6 lines
The cost is not the point; the fairness is:
1) Ramdisk is fair to both LRU algorithms.
2) Zram punishes the LRU algorithm that chooses incompressible pages. IOW, this algorithm needs to compress more pages in order to save the same amount of memory.
I see your point. but my point is that with higher I/O cost to swap in and swap out pages, more major faults(lower hit ratio) will contribute to the loss of final performance.
So for the particular case, if we move to a real disk as a swap device, we might see the same result as zRAM I was using since you also reported more page faults.
Quoted 70 lines
Quoted 70 lines
Quoted 70 lines
# cat /sys/kernel/mm/lru_gen/enabled 0x0003 # cat /sys/kernel/mm/transparent_hugepage/enabled always madvise [never]# modprobe brd rd_nr=1 rd_size=1572864 # if=/dev/zero of=/dev/ram0 bs=1M # mkswap /dev/ram0 # swapoff -a # swapon /dev/ram0# ldd test_absl_32 linux-vdso.so.1 (0xf6e7f000) libabsl_malloc.so.2103.0.1 => /usr/lib/libabsl_malloc.so.2103.0.1 (0xf6e23000) libpthread.so.0 => /lib/libpthread.so.0 (0xf6dff000) libc.so.6 => /lib/libc.so.6 (0xf6d07000) /lib/ld-linux-armhf.so.3 (0x09df0000) libabsl_base.so.2103.0.1 => /usr/lib/libabsl_base.so.2103.0.1 (0xf6ce5000) libabsl_raw_logging.so.2103.0.1 => /usr/lib/libabsl_raw_logging.so.2103.0.1 (0xf6cc4000) libabsl_spinlock_wait.so.2103.0.1 => /usr/lib/libabsl_spinlock_wait.so.2103.0.1 (0xf6ca3000) libc++.so.1 => /usr/lib/libc++.so.1 (0xf6c04000) libc++abi.so.1 => /usr/lib/libc++abi.so.1 (0xf6bcd000) # file test_absl_64 test_absl_64: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked # ldd test_gnu_32 linux-vdso.so.1 (0xeabef000) libpthread.so.0 => /lib/libpthread.so.0 (0xeab92000) libc.so.6 => /lib/libc.so.6 (0xeaa9a000) /lib/ld-linux-armhf.so.3 (0x05690000) # file test_gnu_64 test_gnu_64: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked### baseline 5.17-rc8
# perf record ./test_gnu_64 -t 4 -s $((200*1024*1024)) -S 6000000 10 records/s real 59.00 s user 39.83 s sys 174.18 s18.51% [.] memcpy 15.98% [k] __pi_clear_page 5.59% [k] rmqueue_pcplist 5.19% [k] do_raw_spin_lock 5.09% [k] memmove 4.60% [k] _raw_spin_unlock_irq 3.62% [k] _raw_spin_unlock_irqrestore 3.61% [k] free_unref_page_list 3.29% [k] zap_pte_range 2.53% [k] local_daif_restore 2.50% [k] down_read_trylock 1.41% [k] handle_mm_fault 1.32% [k] do_anonymous_page 1.31% [k] up_read 1.03% [k] free_swap_cache### MGLRU v9
# perf record ./test_gnu_64 -t 4 -s $((200*1024*1024)) -S 6000000 11 records/s real 57.00 s user 39.39 s
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
19.36% [.] memcpy 16.50% [k] __pi_clear_page 6.21% [k] memmove 5.57% [k] rmqueue_pcplist 5.07% [k] do_raw_spin_lock 4.96% [k] _raw_spin_unlock_irqrestore
Enabling ARM64_PSEUDO_NMI and irqchip.gicv3_pseudo_nmi= might help figure out the real code which is taking CPU time in a spin_lock_irqsave area.
Quoted 23 lines
Quoted 11 lines
Quoted 10 lines
4.25% [k] free_unref_page_list 3.80% [k] zap_pte_range 3.69% [k] _raw_spin_unlock_irq 2.71% [k] local_daif_restore 2.10% [k] down_read_trylock 1.50% [k] handle_mm_fault 1.29% [k] do_anonymous_page 1.17% [k] free_swap_cache 1.08% [k] up_readQuoted 4 lines
I think your result is right. but if you take a look at the number of major faults, will you find mglru have more page faults?
i ask this question because i can see mglru even wins with lower hit ratio in the previous report I sent.Yes, I did see the elevated major faults:
# baseline total 11503878 majfault 4745116 pgsteal_kswapd 3056793 pgsteal_direct 3701969
Quoted 4 lines
# MGLRU total 11928659
pgmajfault 5762213pgsteal_kswapd2098253pgsteal_direct4068193
This is a really good sign. Thanks to MGLRU's good implementation, it seems the kernel is spending more time on useful jobs, regardless of the hit ratio.
Thanks
Barry
Re: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
On Wed, Mar 16, 2022 at 12:06 AM Barry Song <[email protected]> wrote:
< snipped>
Quoted 7 lines
Quoted 6 lines
The cost is not the point; the fairness is:
1) Ramdisk is fair to both LRU algorithms.
2) Zram punishes the LRU algorithm that chooses incompressible pages. IOW, this algorithm needs to compress more pages in order to save the same amount of memory.
Quoted 7 lines
I see your point. but my point is that with higher I/O cost to swap in and swap out pages, more major faults(lower hit ratio) will contribute to the loss of final performance.
So for the particular case, if we move to a real disk as a swap device, we might see the same result as zRAM I was using since you also reported more page faults.
If we wanted to talk about I/O cost, we would need to consider the number of writes and writeback patterns as well. The LRU algorithm that *unconsciously* picks more clean pages has an advantage because writes are usually slower than reads. Similarly, the LRU algorithm that *unconsciously* picks a cluster of cold pages that later would be faulted in together also has the advantage because sequential reads are faster than random reads. Do we want to go into this rabbit hole? I think not. That's exactly why I suggested we focus on the fairness. But, just outta curiosity, MGLRU was faster when swapping to a slow MMC disk.
# mmc cid read /sys/class/mmc_host/mmc1/mmc1:0001 type: 'MMC' manufacturer: 'SanDisk-Toshiba Corporation' '' product: 'DA4064' 1.24400152 serial: 0x00000000 manfacturing date: 2006 aug
# baseline + THP=never 0 records/s real 872.00 s user 51.69 s sys 483.09 s
13.07% __memcpy_neon
11.37% __pi_clear_page
9.35% _raw_spin_unlock_irq
5.52% mod_delayed_work_on
5.17% _raw_spin_unlock_irqrestore
3.95% do_raw_spin_lock
3.87% rmqueue_pcplist
3.60% local_daif_restore
3.17% free_unref_page_list
2.74% zap_pte_range
2.00% handle_mm_fault
1.19% do_anonymous_page# MGLRU + THP=never
0 records/s
real 821.00 s
user 44.45 s
sys 428.21 s
13.28% __memcpy_neon
12.78% __pi_clear_page
9.14% _raw_spin_unlock_irq
5.95% _raw_spin_unlock_irqrestore
5.08% mod_delayed_work_on
4.45% do_raw_spin_lock
3.86% local_daif_restore
3.81% rmqueue_pcplist
3.32% free_unref_page_list
2.89% zap_pte_range
1.89% handle_mm_fault
1.10% do_anonymous_page# baseline + THP=madvise 0 records/s real 1341.00 s user 68.15 s sys 681.42 s
12.33% __memcpy_neon
11.78% _raw_spin_unlock_irq
8.79% __pi_clear_page
7.63% mod_delayed_work_on
5.49% _raw_spin_unlock_irqrestore
3.23% local_daif_restore
3.00% do_raw_spin_lock
2.83% rmqueue_pcplist
2.21% handle_mm_fault
2.00% zap_pte_range
1.51% free_unref_page_list
1.33% do_swap_page
1.17% do_anonymous_page# MGLRU + THP=madvise
0 records/s
real 1315.00 s
user 60.59 s
sys 620.56 s
12.34% __memcpy_neon
12.17% _raw_spin_unlock_irq
9.33% __pi_clear_page
7.33% mod_delayed_work_on
6.01% _raw_spin_unlock_irqrestore
3.27% local_daif_restore
3.23% do_raw_spin_lock
2.98% rmqueue_pcplist
2.12% handle_mm_fault
2.04% zap_pte_range
1.65% free_unref_page_list
1.27% do_swap_page
1.11% do_anonymous_pageRe: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
In reply to Yu ZhaoQuoted 26 lines · Yu Zhao
Evictable pages are divided into multiple generations for each lruvec. The youngest generation number is stored in
lrugen->max_seqfor both anon and file types as they're aged on an equal footing. The oldest generation numbers are stored in lrugen->min_seq[] separately for anon and file types as clean file pages can be evicted regardless of swap constraints. These three variables are monotonically increasing.Generation numbers are truncated into
order_base_2(MAX_NR_GENS+1)bits in order to fit into the gen counter infolio->flags. Each truncated generation number is an index to lrugen->lists[]. The sliding window technique is used to track at leastMIN_NR_GENSand at mostMAX_NR_GENSgenerations. The gen counter stores (seq%MAX_NR_GENS)+1 while a page is on one of lrugen->lists[]. Otherwise it stores 0.There are two conceptually independent processes (as in the manufacturing process): "the aging", which produces young generations, and "the eviction", which consumes old generations. They form a closed-loop system, i.e., "the page reclaim". Both processes can be invoked from userspace for the purposes of working set estimation and proactive reclaim. These features are required to optimize job scheduling (bin packing) in data centers. The variable size of the sliding window is designed for such use cases [1][2].
To avoid confusions, the terms "hot" and "cold" will be applied to the multigenerational LRU, as a new convention; the terms "active" and "inactive" will be applied to the active/inactive LRU, as usual.
[...]
include/linux/page-flags-layout.h line 35
Quoted 12 lines
+++
b/include/linux/page-flags-layout.hinclude/linux/page-flags-layout.h +7
-0@@ -26,6 +26,14 @@#define ZONES_WIDTH ZONES_SHIFT +#ifdef CONFIG_LRU_GEN +/* LRU_GEN_WIDTH is generated from order_base_2(CONFIG_NR_LRU_GENS + 1). */ +#define LRU_REFS_WIDTH (CONFIG_TIERS_PER_GEN - 2) +#else +#define LRU_GEN_WIDTH 0 +#define LRU_REFS_WIDTH 0 +#endif /* CONFIG_LRU_GEN */
I'm concerned about the number of bits being used in page->flags. It seems to me that we already have six bits in use to aid us in choosing which pages to reclaim: referenced, lru, active, workingset, reclaim, unevictable.
What I was hoping to see from this patch set was reuse of those bits. That would give us 32 queues in total. Some would be special (eg pages cannot migrate out of the unevictable queue), but it seems to me that you effectively have 4 queues for active and 4 queues for inactive at this point (unless I misunderstood that). I think we need special numbers
for: Not on the LRU and Unevictable, but that still leaves us with 30 generations to split between active & inactive.
But maybe we still need some of those bits? Perhaps it's not OK to say that queue id 0 is !LRU, queue 1 is unevictable, queue #2 is workingset, queues 3-7 are active, queues 8-15 are various degrees of inactive. I'm assuming that it's not sensible to have a page that's marked as both "reclaim" and "workingset", but perhaps it is.
Anyway, I don't understand this area well enough. I was just hoping for some simplification.
Re: [PATCH v7 04/12] mm: multigenerational LRU: groundwork
Quoted 1 line · Matthew Wilcox
On Tue, Feb 08, 2022 at 01:18:54AM -0700, Yu Zhao wrote:
[...]
Quoted 18 lines
include/linux/page-flags-layout.h line 35
Quoted 12 lines
+++
b/include/linux/page-flags-layout.hinclude/linux/page-flags-layout.h +7
-0@@ -26,6 +26,14 @@#define ZONES_WIDTH ZONES_SHIFT +#ifdef CONFIG_LRU_GEN +/* LRU_GEN_WIDTH is generated from order_base_2(CONFIG_NR_LRU_GENS + 1). */ +#define LRU_REFS_WIDTH (CONFIG_TIERS_PER_GEN - 2) +#else +#define LRU_GEN_WIDTH 0 +#define LRU_REFS_WIDTH 0 +#endif /* CONFIG_LRU_GEN */ I'm concerned about the number of bits being used in
page->flags. It seems to me that we already have six bits in use to aid us in choosing which pages to reclaim: referenced, lru, active, workingset, reclaim, unevictable.
Quoted 1 line
What I was hoping to see from this patch set was reuse of those bits.
Agreed. I have a plan to *reduce* some of those bits but it's a relatively low priority item on my to-do list.
Quoted 7 lines
That would give us 32 queues in total. Some would be special (eg pages cannot migrate out of the unevictable queue), but it seems to me that you effectively have 4 queues for active and 4 queues for inactive at this point (unless I misunderstood that). I think we need special numbers
for: Not on the LRU and Unevictable, but that still leaves us with 30 generations to split between active & inactive.
Quoted 8 lines
But maybe we still need some of those bits? Perhaps it's not OK to say that queue id 0 is !LRU, queue 1 is unevictable, queue #2 is workingset, queues 3-7 are active, queues 8-15 are various degrees of inactive. I'm assuming that it's not sensible to have a page that's marked as both "reclaim" and "workingset", but perhaps it is.
Anyway, I don't understand this area well enough. I was just hoping for some simplification.
I plan to use the spare bits in folio->lru to indicate which lru list a folio is on, i.e., active/inactive or generations or unevictable.
In addition, swapbacked could go to folio->mapping -- we wouldn't need it if there were no MADV_FREE, i.e., it would be equivalent to PageAnon() || shmem_mapping().
These two work items can be done separately and in parallel with everything else that's been going on lately. It'd awesome if somebody volunteers, and I can find some resource from our side to test/review the code so that we can have them done sooner.
The rest, in theory, can also be moved to somewhere but, IMO, it's not really worth the effort given the situation isn't dire at the moment. referenced and workingset are already reused between the active/inactive lru and the multigenerational lru; reclaim is reused for readahead, but readahead could be split out as an xa tag; lru is reused for isolation synchronization.
PATCH v7 5/12 mm: multigenerational LRU: minimal implementation
In reply to Yu ZhaoTo avoid confusions, the terms "promotion" and "demotion" will be applied to the multigenerational LRU, as a new convention; the terms "activation" and "deactivation" will be applied to the active/inactive LRU, as usual.
The aging produces young generations. Given an lruvec, it increments max_seq when max_seq-min_seq+1 approaches MIN_NR_GENS. The aging promotes hot pages to the youngest generation when it finds them accessed through page tables; the demotion of cold pages happens consequently when it increments max_seq. Since the aging is only interested in hot pages, its complexity is O(nr_hot_pages). Promotion in the aging path doesn't require any LRU list operations, only the updates of the gen counter and lrugen->nr_pages[]; demotion, unless as the result of the increment of max_seq, requires LRU list operations, e.g., lru_deactivate_fn().
The eviction consumes old generations. Given an lruvec, it increments min_seq when the list indexed by min_seq%MAX_NR_GENS becomes empty. A feedback loop modeled after the PID controller monitors refaults over anon and file types and decides which type to evict when both are available from the same generation.
Each generation is divided into multiple tiers. Tiers represent different ranges of numbers of accesses through file descriptors. A page accessed N times through file descriptors is in tier order_base_2(N). Tiers don't have dedicated lrugen->lists[], only bits in folio->flags. In contrast to moving across generations which requires the LRU lock, moving between tiers only involves operations on folio->flags. The feedback loop also monitors refaults over all tiers and decides when to promote pages in which tiers (N>1), using the first tier (N=0,1) as a baseline. The first tier contains single-use unmapped clean pages, which are most likely the best choices. The eviction promotes a page to the next generation, i.e., min_seq+1 rather than max_seq, if the feedback loop decides so. This approach has the following advantages:
1) It removes the cost of activation in the buffered access path by inferring whether pages accessed multiple times through file descriptors are statistically hot and thus worth promoting in the eviction path.
2) It takes pages accessed through page tables into account and avoids overprotecting pages accessed multiple times through file descriptors. (Pages accessed through page tables are in the first tier since N=0.)
3) More tiers provide better protection for pages accessed more than twice through file descriptors, when under heavy buffered I/O workloads.
Server benchmark results:
Single workload:
fio (buffered I/O): +[47, 49]%
IOPS BW
5.17-rc2: 2242k 8759MiB/s
patch1-5: 3321k 12.7GiB/sSingle workload:
memcached (anon): +[101, 105]%
Ops/sec KB/sec
5.17-rc2: 476771.79 18544.31
patch1-5: 972526.07 37826.95Configurations:
CPU: two Xeon 6154
Mem: total 256G
Node 1 was used as a ram disk only to reduce the variance in the
results.
patch drivers/block/brd.c <<EOF
99,100c99,100
< gfp_flags = GFP_NOIO | __GFP_ZERO | __GFP_HIGHMEM;
< page = alloc_page(gfp_flags);
---
> gfp_flags = GFP_NOIO | __GFP_ZERO | __GFP_HIGHMEM | __GFP_THISNODE;
> page = alloc_pages_node(1, gfp_flags, 0);
EOF
cat >>/etc/systemd/system.conf <<OEF
CPUAffinity=numa
NUMAPolicy=bind
NUMAMask=0
OEF
cat >>/etc/memcached.conf <<OEF
-m 184320
-s /var/run/memcached/memcached.sock
-a 0766
-t 36
-B binary
OEF
cat fio.sh
modprobe brd rd_nr=1 rd_size=113246208
mkfs.ext4 /dev/ram0
mount -t ext4 /dev/ram0 /mnt
mkdir /sys/fs/cgroup/user.slice/test
echo 38654705664 >/sys/fs/cgroup/user.slice/test/memory.max
echo $$ >/sys/fs/cgroup/user.slice/test/cgroup.procs
fio -name=mglru --numjobs=72 --directory=/mnt --size=1408m \
--buffered=1 --ioengine=io_uring --iodepth=128 \
--iodepth_batch_submit=32 --iodepth_batch_complete=32 \
--rw=randread --random_distribution=random --norandommap \
--time_based --ramp_time=10m --runtime=5m --group_reporting
cat memcached.sh
modprobe brd rd_nr=1 rd_size=113246208
swapoff -a
mkswap /dev/ram0
swapon /dev/ram0
memtier_benchmark -S /var/run/memcached/memcached.sock \
-P memcache_binary -n allkeys --key-minimum=1 \
--key-maximum=65000000 --key-pattern=P:P -c 1 -t 36 \
--ratio 1:0 --pipeline 8 -d 2000
memtier_benchmark -S /var/run/memcached/memcached.sock \
-P memcache_binary -n allkeys --key-minimum=1 \
--key-maximum=65000000 --key-pattern=R:R -c 1 -t 36 \
--ratio 0:1 --pipeline 8 --randomize --distinct-client-seedClient benchmark results:
kswapd profiles:
5.17-rc2
38.05% page_vma_mapped_walk
20.86% lzo1x_1_do_compress (real work)
6.16% do_raw_spin_lock
4.61% _raw_spin_unlock_irq
2.20% vma_interval_tree_iter_next
2.19% vma_interval_tree_subtree_search
2.15% page_referenced_one
1.93% anon_vma_interval_tree_iter_first
1.65% ptep_clear_flush
1.00% __zram_bvec_write
patch1-5
39.73% lzo1x_1_do_compress (real work)
14.96% page_vma_mapped_walk
6.97% _raw_spin_unlock_irq
3.07% do_raw_spin_lock
2.53% anon_vma_interval_tree_iter_first
2.04% ptep_clear_flush
1.82% __zram_bvec_write
1.76% __anon_vma_interval_tree_subtree_search
1.57% memmove
1.45% free_unref_page_list Configurations:
CPU: single Snapdragon 7c
Mem: total 4G
Chrome OS MemoryPressure [1]
[1] https://chromium.googlesource.com/chromiumos/platform/tast-tests/
--- include/linux/mm.h | 1 + include/linux/mm_inline.h | 15 + include/linux/mmzone.h | 35 ++ mm/Kconfig | 44 +++ mm/swap.c | 46 ++- mm/vmscan.c | 784 +++++++++++++++++++++++++++++++++++++- mm/workingset.c | 119 +++++- 7 files changed, 1039 insertions(+), 5 deletions(-)
include/linux/mm.h +1 -0
@@ -227,6 +227,7 @@ int overcommit_policy_handler(struct ctl_table *, int, void *, size_t *,#define PAGE_ALIGNED (addr) IS_ALIGNED((unsigned long)(addr), PAGE_SIZE)#define lru_to_page (head) (list_entry((head)->prev, struct page, lru)) +#define lru_to_folio (head) (list_entry((head)->prev, struct folio, lru))void setup_initial_init_mm (void *start_code,void *end_code,void *end_data,void *brk);
include/linux/mm_inline.h +15 -0
@@ -109,6 +109,19 @@ static inline int lru_gen_from_seq(unsigned long seq)return seq %MAX_NR_GENS ; } +static inline int lru_hist_from_seq (unsigned long seq) +{ +return seq %NR_HIST_GENS ; +} + +static inline int lru_tier_from_refs (int refs) +{ +VM_BUG_ON (refs >BIT (LRU_REFS_WIDTH )); + +/* see the comment on MAX_NR_TIERS */ +return order_base_2 (refs +1 ); +} +static inline bool lru_gen_is_active (struct lruvec *lruvec,int gen) {unsigned long max_seq = lruvec->lrugen.max_seq;
@@ -237,6 +250,8 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio, gen = ((new_flags &LRU_GEN_MASK ) >>LRU_GEN_PGOFF ) -1 ; new_flags &= ~LRU_GEN_MASK ; +if ((new_flags &LRU_REFS_FLAGS ) !=LRU_REFS_FLAGS ) + new_flags &= ~(LRU_REFS_MASK |LRU_REFS_FLAGS );/* for shrink_page_list() */ if (reclaiming) new_flags &= ~(BIT (PG_referenced) |BIT (PG_reclaim));
include/linux/mmzone.h +35 -0
@@ -335,6 +335,32 @@ struct lruvec;#define MIN_NR_GENS 2U#define MAX_NR_GENS ((unsigned int)CONFIG_NR_LRU_GENS) +/* +* Each generation is divided into multiple tiers. Tiers represent different +* ranges of numbers of accesses through file descriptors. A page accessed N +* times through file descriptors is in tier order_base_2(N). A page in the +* first tier (N=0,1) is marked by PG_referenced unless it was faulted in +* though page tables or read ahead. A page in any other tier (N>1) is marked +* by PG_referenced and PG_workingset. Additional bits in folio->flags are +* required to support more than two tiers. +* +* In contrast to moving across generations which requires the LRU lock, moving +* across tiers only requires operations on folio->flags and therefore has a +* negligible cost in the buffered access path. In the eviction path, +* comparisons of refaulted/(evicted+promoted) from the first tier and the rest +* infer whether pages accessed multiple times through file descriptors are +* statistically hot and thus worth promoting. +*/ +#define MAX_NR_TIERS ((unsigned int)CONFIG_TIERS_PER_GEN) +#define LRU_REFS_FLAGS (BIT(PG_referenced) | BIT(PG_workingset)) + +/* whether to keep historical stats from evicted generations */ +#ifdef CONFIG_LRU_GEN_STATS +#define NR_HIST_GENS ((unsigned int)CONFIG_NR_LRU_GENS) +#else +#define NR_HIST_GENS 1U +#endif +struct lru_gen_struct {/* the aging increments the youngest generation number */ unsigned long max_seq;
@@ -346,6 +372,15 @@ struct lru_gen_struct {struct list_head lists[MAX_NR_GENS ][ANON_AND_FILE ][MAX_NR_ZONES ];/* the sizes of the above lists */ unsigned long nr_pages[MAX_NR_GENS ][ANON_AND_FILE ][MAX_NR_ZONES ]; +/* the exponential moving average of refaulted */ +unsigned long avg_refaulted[ANON_AND_FILE ][MAX_NR_TIERS ]; +/* the exponential moving average of evicted+promoted */ +unsigned long avg_total[ANON_AND_FILE ][MAX_NR_TIERS ]; +/* the first tier doesn't need promotion, hence the minus one */ +unsigned long promoted[NR_HIST_GENS ][ANON_AND_FILE ][MAX_NR_TIERS -1 ]; +/* can be modified without holding the LRU lock */ +atomic_long_t evicted[NR_HIST_GENS ][ANON_AND_FILE ][MAX_NR_TIERS ]; +atomic_long_t refaulted[NR_HIST_GENS ][ANON_AND_FILE ][MAX_NR_TIERS ];/* whether the multigenerational LRU is enabled */ bool enabled; };
mm/Kconfig +44 -0
50 lines not on this page · raw message
mm/swap.c +44 -2
70 lines not on this page · raw message
mm/vmscan.c +783 -1
843 lines not on this page · raw message
mm/workingset.c +117 -2
151 lines not on this page · raw message
Signature · 1 line
2.35.0.263.gb82422642f-goog
Re: [PATCH v7 05/12] mm: multigenerational LRU: minimal implementation
On Tue, Feb 08, 2022 at 01:18:55AM -0700, Yu Zhao wrote:
<snipped>
Quoted 31 lines
mm/Kconfig +23
-0@@ -892,6 +892,50 @@ config ANON_VMA_NAME area from being merged with adjacent virtual memory areas due to the difference in their name. +# multigenerational LRU { +config LRU_GEN + bool "Multigenerational LRU" + depends on MMU + # the following options can use up the spare bits in page flags + depends on !MAXSMP && (64BIT || !SPARSEMEM || SPARSEMEM_VMEMMAP) + help + A high performance LRU implementation for memory overcommit. See + Documentation/admin-guide/mm/multigen_lru.rst and + Documentation/vm/multigen_lru.rst for details. + +config NR_LRU_GENS + int "Max number of generations" + depends on LRU_GEN + range 4 31 + default 4 + help + Do not increase this value unless you plan to use working set + estimation and proactive reclaim to optimize job scheduling in data + centers. + + This option uses order_base_2(N+1) bits in page flags. +
mm/Kconfig config ANON_VMA_NAME · line 927
Quoted 10 lines
mm/Kconfig +10
-0+config TIERS_PER_GEN + int "Number of tiers per generation" + depends on LRU_GEN + range 2 4 + default 4 + help + Do not decrease this value unless you run out of spare bits in page + flags, i.e., you see the "Not enough bits in page flags" build error. + + This option uses N-2 bits in page flags.
Moved Kconfig to this patch as suggested by:
https://lore.kernel.org/linux-mm/[email protected]/
Added two new macros as requested here:
https://lore.kernel.org/linux-mm/[email protected]/
<snipped>
Quoted 28 lines
mm/vmscan.c +6
-1@@ -1285,9 +1285,11 @@ static int __remove_mapping(struct address_space *mapping, struct page *page,if (PageSwapCache (page)) {swp_entry_t swap = { .val =page_private (page) }; -mem_cgroup_swapout (page, swap); + +/* get a shadow entry before mem_cgroup_swapout() clears folio_memcg() */ if (reclaimed && !mapping_exiting (mapping)) shadow =workingset_eviction (page, target_memcg); +mem_cgroup_swapout (page, swap);__delete_from_swap_cache (page, swap, shadow);xa_unlock_irq (&mapping->i_pages);put_swap_page (page, swap);@@ -2721,6 +2723,9 @@ static void prepare_scan_count(pg_data_t *pgdat, struct scan_control *sc)unsigned long file;struct lruvec *target_lruvec; +if (lru_gen_enabled ()) +return ; + target_lruvec =mem_cgroup_lruvec (sc->target_mem_cgroup, pgdat);/*
mm/vmscan.c static bool can_age_anon_pages(struct pglist_data *pgdat, · line 3053
Quoted 7 lines
mm/vmscan.c +4
-0@@ -3042,15 +3047,47 @@ static bool can_age_anon_pages(struct pglist_data *pgdat,#ifdef CONFIG_LRU_GEN +enum { +TYPE_ANON , +TYPE_FILE , +};
Added two new macros as requested here:
https://lore.kernel.org/linux-mm/[email protected]/
<snipped>
mm/vmscan.c static struct lruvec *get_lruvec(struct mem_cgroup *memcg, int nid) · line 3406
Quoted 13 lines
mm/vmscan.c +13
-0+static void age_lruvec (struct lruvec *lruvec,struct scan_control *sc) +{ +bool need_aging; +long nr_to_scan; +struct mem_cgroup *memcg =lruvec_memcg (lruvec); +int swappiness =get_swappiness (memcg); +DEFINE_MAX_SEQ (lruvec); +DEFINE_MIN_SEQ (lruvec); + +mem_cgroup_calculate_protection (NULL , memcg); + +if (mem_cgroup_below_min (memcg)) +return ;
Added mem_cgroup_calculate_protection() for readability as requested here:
https://lore.kernel.org/linux-mm/Ydf9RXPch5ddg%[email protected]/
<snipped>
Johannes Weiner
maintainerRe: [PATCH v7 05/12] mm: multigenerational LRU: minimal implementation
In reply to Yu ZhaoHi Yu,
Thanks for restructuring this from the last version. It's easier to learn the new model when you start out with the bare bones, then let optimizations and self-contained features follow later.
Quoted 5 lines · Yu Zhao
To avoid confusions, the terms "promotion" and "demotion" will be applied to the multigenerational LRU, as a new convention; the terms "activation" and "deactivation" will be applied to the active/inactive LRU, as usual.
Quoted 10 lines
The aging produces young generations. Given an lruvec, it increments
max_seqwhen max_seq-min_seq+1 approachesMIN_NR_GENS. The aging promotes hot pages to the youngest generation when it finds them accessed through page tables; the demotion of cold pages happens consequently when it incrementsmax_seq. Since the aging is only interested in hot pages, its complexity isO(nr_hot_pages). Promotion in the aging path doesn't require any LRU list operations, only the updates of the gen counter and lrugen->nr_pages[]; demotion, unless as the result of the increment ofmax_seq, requires LRU list operations, e.g.,lru_deactivate_fn().
I'm having trouble with this changelog. It opens with a footnote and summarizes certain aspects of the implementation whose importance to the reader aren't entirely clear at this time.
It would be better to start with a high-level overview of the problem and how this algorithm solves it. How the reclaim algorithm needs to find the page that is most suitable for eviction and to signal when it's time to give up and OOM. Then explain how grouping pages into multiple generations accomplishes that - in particular compared to the current two use-once/use-many lists.
Explain the problem of MMU vs syscall references, and how tiering addresses this.
Explain the significance of refaults and how the algorithm responds to them. Not in terms of which running averages are updated, but in terms of user-visible behavior ("will start swapping (more)" etc.)
Express *intent*, how it's supposed to behave wrt workloads and memory pressure. The code itself will explain the how, its complexity etc.
Most reviewers will understand the fundamental challenges of page reclaim. The difficulty is matching individual aspects of the problem space to your individual components and design choices you have made.
Let us in on that thinking, please ;)
Quoted 4 lines
mm/Kconfig +0
-0@@ -892,6 +892,50 @@ config ANON_VMA_NAME area from being merged with adjacent virtual memory areas due to the difference in their name.
mm/Kconfig config ANON_VMA_NAME · line 904
Quoted 10 lines
mm/Kconfig +10
-0+# multigenerational LRU { +config LRU_GEN + bool "Multigenerational LRU" + depends on MMU + # the following options can use up the spare bits in page flags + depends on !MAXSMP && (64BIT || !SPARSEMEM || SPARSEMEM_VMEMMAP) + help + A high performance LRU implementation for memory overcommit. See + Documentation/admin-guide/mm/multigen_lru.rst and + Documentation/vm/multigen_lru.rst for details.
These files don't exist at this time, please introduce them before or when referencing them. If they document things introduced later in the patchset, please start with a minimal version of the file and update it as you extend the algorithm and add optimizations etc.
It's really important to only reference previous patches, not later ones. This allows reviewers to read the patches linearly. Having to search for missing pieces in patches you haven't looked at yet is bad.
Quoted 12 lines
mm/Kconfig +12
-0+config NR_LRU_GENS + int "Max number of generations" + depends on LRU_GEN + range 4 31 + default 4 + help + Do not increase this value unless you plan to use working set + estimation and proactive reclaim to optimize job scheduling in data + centers. + + This option uses order_base_2(N+1) bits in page flags. +
mm/Kconfig config ANON_VMA_NAME · line 927
Quoted 10 lines
mm/Kconfig +10
-0+config TIERS_PER_GEN + int "Number of tiers per generation" + depends on LRU_GEN + range 2 4 + default 4 + help + Do not decrease this value unless you run out of spare bits in page + flags, i.e., you see the "Not enough bits in page flags" build error. + + This option uses N-2 bits in page flags.
Linus had pointed out that we shouldn't ask these questions of the user. How do you pick numbers here? I'm familiar with workingset estimation and proactive reclaim usecases but I wouldn't know.
Even if we removed the config option and hardcoded the number, this is a question for kernel developers: What does "4" mean? How would behavior differ if it were 3 or 5 instead? Presumably there is some sort of behavior gradient. "As you increase the number of generations/tiers, the user-visible behavior of the kernel will..." This should really be documented.
I'd also reiterate Mel's point: Distribution kernels need to support the full spectrum of applications and production environments. Unless using non-defaults it's an extremely niche usecase (like compiling out BUG() calls) compile-time options are not the right choice. If we do need a tunable, it could make more sense to have a compile time upper limit (to determine page flag space) combined with a runtime knob?
Thanks!
Re: [PATCH v7 05/12] mm: multigenerational LRU: minimal implementation
On Tue, Feb 08, 2022 at 11:50:09AM -0500, Johannes Weiner wrote:
<snipped>
Quoted 21 lines · Yu Zhao
Quoted 5 lines
To avoid confusions, the terms "promotion" and "demotion" will be applied to the multigenerational LRU, as a new convention; the terms "activation" and "deactivation" will be applied to the active/inactive LRU, as usual.
Quoted 10 lines
The aging produces young generations. Given an lruvec, it increments
max_seqwhen max_seq-min_seq+1 approachesMIN_NR_GENS. The aging promotes hot pages to the youngest generation when it finds them accessed through page tables; the demotion of cold pages happens consequently when it incrementsmax_seq. Since the aging is only interested in hot pages, its complexity isO(nr_hot_pages). Promotion in the aging path doesn't require any LRU list operations, only the updates of the gen counter and lrugen->nr_pages[]; demotion, unless as the result of the increment ofmax_seq, requires LRU list operations, e.g.,lru_deactivate_fn().I'm having trouble with this changelog. It opens with a footnote and summarizes certain aspects of the implementation whose importance to the reader aren't entirely clear at this time.
Quoted 6 lines
It would be better to start with a high-level overview of the problem and how this algorithm solves it. How the reclaim algorithm needs to find the page that is most suitable for eviction and to signal when it's time to give up and OOM. Then explain how grouping pages into multiple generations accomplishes that - in particular compared to the current two use-once/use-many lists.
Hi Johannes,
Thanks for reviewing!
I suspect the information you are looking for might have been in the patchset but is scattered in a few places. Could you please glance at the following pieces and let me know
1. whether they cover some of the points you asked for
2. and if so, whether there is a better order/place to present them?
The previous patch has a quick view on the architecture:
https://lore.kernel.org/linux-mm/[email protected]/
Evictable pages are divided into multiple generations for each lruvec. The youngest generation number is stored in lrugen->max_seq for both anon and file types as they're aged on an equal footing. The oldest generation numbers are stored in lrugen->min_seq[] separately for anon and file types as clean file pages can be evicted regardless of swap constraints. These three variables are monotonically increasing.
Generation numbers are truncated into order_base_2(MAX_NR_GENS+1) bits in order to fit into the gen counter in folio->flags. Each truncated generation number is an index to lrugen->lists[]. The sliding window technique is used to track at least MIN_NR_GENS and at most MAX_NR_GENS generations. The gen counter stores (seq%MAX_NR_GENS)+1 while a page is on one of lrugen->lists[]. Otherwise it stores 0.
There are two conceptually independent processes (as in the manufacturing process): "the aging", which produces young generations, and "the eviction", which consumes old generations. They form a closed-loop system, i.e., "the page reclaim". Both processes can be invoked from userspace for the purposes of working set estimation and proactive reclaim. These features are required to optimize job scheduling (bin packing) in data centers. The variable size of the sliding window is designed for such use cases...
And the design doc contains a bit more details, and I'd be happy to present it earlier, if you think doing so would help.
https://lore.kernel.org/linux-mm/[email protected]/
Quoted 2 lines
Explain the problem of MMU vs syscall references, and how tiering addresses this.
The previous patch also touched on this point:
https://lore.kernel.org/linux-mm/[email protected]/
The protection of hot pages and the selection of cold pages are based on page access channels and patterns. There are two access channels:
one through page tables and the other through file descriptors. The protection of the former channel is by design stronger because:
1) The uncertainty in determining the access patterns of the former channel is higher due to the approximation of the accessed bit.
2) The cost of evicting the former channel is higher due to the TLB flushes required and the likelihood of encountering the dirty bit.
3) The penalty of underprotecting the former channel is higher because applications usually don't prepare themselves for major page faults like they do for blocked I/O. E.g., GUI applications commonly use dedicated I/O threads to avoid blocking the rendering threads.
There are also two access patterns: one with temporal locality and the other without. For the reasons listed above, the former channel is assumed to follow the former pattern unless VM_SEQ_READ or VM_RAND_READ is present, and the latter channel is assumed to follow the latter pattern unless outlying refaults have been observed.
Quoted 3 lines
Explain the significance of refaults and how the algorithm responds to them. Not in terms of which running averages are updated, but in terms of user-visible behavior ("will start swapping (more)" etc.)
And this patch touched on how tiers would help:
1) It removes the cost of activation in the buffered access path by inferring whether pages accessed multiple times through file descriptors are statistically hot and thus worth promoting in the eviction path.
2) It takes pages accessed through page tables into account and avoids overprotecting pages accessed multiple times through file descriptors. (Pages accessed through page tables are in the first tier since N=0.)
3) More tiers provide better protection for pages accessed more than twice through file descriptors, when under heavy buffered I/O workloads.
And the design doc:
https://lore.kernel.org/linux-mm/[email protected]/
To select a type and a tier to evict from, it first compares min_seq[] to select the older type. If they are equal, it selects the type whose first tier has a lower refault percentage. The first tier contains single-use unmapped clean pages, which are the best bet.
Quoted 2 lines
Express *intent*, how it's supposed to behave wrt workloads and memory pressure. The code itself will explain the how, its complexity etc.
Hmm... This part I'm not so sure. It seems to me this is equivalent to describing how it works.
Quoted 5 lines
Most reviewers will understand the fundamental challenges of page reclaim. The difficulty is matching individual aspects of the problem space to your individual components and design choices you have made.
Let us in on that thinking, please ;)
Agreed. I'm sure I haven't covered everything. So I'm trying to figure out what's important but missing/insufficient.
Quoted 20 lines
Quoted 4 lines
mm/Kconfig +0
-0@@ -892,6 +892,50 @@ config ANON_VMA_NAME area from being merged with adjacent virtual memory areas due to the difference in their name.mm/Kconfig config ANON_VMA_NAME · line 904
Quoted 10 lines
mm/Kconfig +10
-0+# multigenerational LRU { +config LRU_GEN + bool "Multigenerational LRU" + depends on MMU + # the following options can use up the spare bits in page flags + depends on !MAXSMP && (64BIT || !SPARSEMEM || SPARSEMEM_VMEMMAP) + help + A high performance LRU implementation for memory overcommit. See + Documentation/admin-guide/mm/multigen_lru.rst and + Documentation/vm/multigen_lru.rst for details.These files don't exist at this time, please introduce them before or when referencing them. If they document things introduced later in the patchset, please start with a minimal version of the file and update it as you extend the algorithm and add optimizations etc.
Quoted 3 lines
It's really important to only reference previous patches, not later ones. This allows reviewers to read the patches linearly. Having to search for missing pieces in patches you haven't looked at yet is bad.
Okay, will remove this bit from this patch.
Quoted 34 lines
Quoted 12 lines
mm/Kconfig +12
-0+config NR_LRU_GENS + int "Max number of generations" + depends on LRU_GEN + range 4 31 + default 4 + help + Do not increase this value unless you plan to use working set + estimation and proactive reclaim to optimize job scheduling in data + centers. + + This option uses order_base_2(N+1) bits in page flags. +mm/Kconfig config ANON_VMA_NAME · line 927
Quoted 10 lines
mm/Kconfig +10
-0+config TIERS_PER_GEN + int "Number of tiers per generation" + depends on LRU_GEN + range 2 4 + default 4 + help + Do not decrease this value unless you run out of spare bits in page + flags, i.e., you see the "Not enough bits in page flags" build error. + + This option uses N-2 bits in page flags.Linus had pointed out that we shouldn't ask these questions of the user. How do you pick numbers here? I'm familiar with workingset estimation and proactive reclaim usecases but I wouldn't know.
Even if we removed the config option and hardcoded the number, this is a question for kernel developers: What does "4" mean? How would behavior differ if it were 3 or 5 instead? Presumably there is some sort of behavior gradient. "As you increase the number of generations/tiers, the user-visible behavior of the kernel will..." This should really be documented.
Quoted 6 lines
I'd also reiterate Mel's point: Distribution kernels need to support the full spectrum of applications and production environments. Unless using non-defaults it's an extremely niche usecase (like compiling out
BUG()calls) compile-time options are not the right choice. If we do need a tunable, it could make more sense to have a compile time upper limit (to determine page flag space) combined with a runtime knob?
I agree, and I think only time can answer all theses questions :)
This effort is not in the final stage but at very its beginning. More experiments and wilder adoption are required to see how it's going to evolve or where it leads. For now, there is just no way to tell whether those values make sense for the majority or we need the runtime knobs.
These are valid concerns, but TBH, I think they are minor ones because most users need not to worry about them -- this patchset has been used in several downstream kernels and I haven't heard any complaints about those options/values:
https://lore.kernel.org/linux-mm/[email protected]/
1. Android ARCVM
2. Arch Linux Zen
3. Chrome OS
4. Liquorix
5. post-factum
6. XanMod
Then why do we need these options? Because there are always exceptions, as stated in the descriptions of those options. Sometimes we just can't decide everything for users -- the answers lie in their use cases. The bottom line is, if this starts bothering people or gets in somebody's way, I'd be glad to revisit. Fair enough?
Thanks!
Re: [PATCH v7 05/12] mm: multigenerational LRU: minimal implementation
In reply to Yu ZhaoHello Yu
Quoted 7 lines · On Tue, 8 Feb 2022 01:18:55 -0700 Yu Zhao wrote:
mm/vmscan.c +7
-0+ +/****************************************************************************** +* the aging +******************************************************************************/ + +static int folio_inc_gen (struct lruvec *lruvec,struct folio *folio,bool reclaiming) +{
mm/vmscan.c static struct lruvec *get_lruvec(struct mem_cgroup *memcg, int nid) · line 3241
Quoted 10 lines
mm/vmscan.c +10
-0+unsigned long old_flags, new_flags; +int type =folio_is_file_lru (folio); +struct lru_gen_struct *lrugen = &lruvec->lrugen; +int new_gen, old_gen =lru_gen_from_seq (lrugen->min_seq[type]); + + do { + new_flags = old_flags = READ_ONCE(folio->flags); + VM_BUG_ON_FOLIO(!(new_flags & LRU_GEN_MASK), folio); + + new_gen = ((new_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
Is the chance zero for deadloop if new_gen != old_gen?
mm/vmscan.c static struct lruvec *get_lruvec(struct mem_cgroup *memcg, int nid) · line 3247
Quoted 6 lines
mm/vmscan.c +6
-0+ new_gen = (old_gen +1 ) %MAX_NR_GENS ; + + new_flags &= ~LRU_GEN_MASK ; + new_flags |= (new_gen +1UL ) <<LRU_GEN_PGOFF ; + new_flags &= ~(LRU_REFS_MASK |LRU_REFS_FLAGS ); +/* for folio_end_writeback() */
/* for folio_end_writeback() and sort_folio() */ in terms of
reclaiming?
Quoted 8 lines
mm/vmscan.c +8
-0+if (reclaiming) + new_flags |=BIT (PG_reclaim); + }while (cmpxchg (&folio->flags, old_flags, new_flags) != old_flags); + +lru_gen_balance_size (lruvec, folio, old_gen, new_gen); + +return new_gen; +}
...
mm/vmscan.c static struct lruvec *get_lruvec(struct mem_cgroup *memcg, int nid) · line 3442
Quoted 6 lines
mm/vmscan.c +6
-0+/****************************************************************************** +* the eviction +******************************************************************************/ + +static bool sort_folio (struct lruvec *lruvec,struct folio *folio,int tier_idx) +{
Nit, the 80-column-char format is prefered.
Quoted 34 lines
mm/vmscan.c +34
-0+bool success; +int gen =folio_lru_gen (folio); +int type =folio_is_file_lru (folio); +int zone =folio_zonenum (folio); +int tier =folio_lru_tier (folio); +int delta =folio_nr_pages (folio); +struct lru_gen_struct *lrugen = &lruvec->lrugen; + +VM_BUG_ON_FOLIO (gen >=MAX_NR_GENS , folio); + +if (!folio_evictable (folio)) { + success =lru_gen_del_folio (lruvec, folio, true); +VM_BUG_ON_FOLIO (!success, folio); +folio_set_unevictable (folio); +lruvec_add_folio (lruvec, folio); +__count_vm_events (UNEVICTABLE_PGCULLED , delta); +return true; + } + +if (type &&folio_test_anon (folio) &&folio_test_dirty (folio)) { + success =lru_gen_del_folio (lruvec, folio, true); +VM_BUG_ON_FOLIO (!success, folio); +folio_set_swapbacked (folio); +lruvec_add_folio_tail (lruvec, folio); +return true; + } + +if (tier > tier_idx) { +int hist =lru_hist_from_seq (lrugen->min_seq[type]); + + gen =folio_inc_gen (lruvec, folio, false); +list_move_tail (&folio->lru, &lrugen->lists[gen][type][zone]); + + WRITE_ONCE(lrugen->promoted[hist][type][tier - 1],
mm/vmscan.c static struct lruvec *get_lruvec(struct mem_cgroup *memcg, int nid) · line 3486
Quoted 10 lines
mm/vmscan.c +10
-0+ lrugen->promoted[hist][type][tier - 1] + delta); +__mod_lruvec_state (lruvec,WORKINGSET_ACTIVATE_BASE + type, delta); +return true; + } + +if (folio_test_locked (folio) ||folio_test_writeback (folio) || + (type &&folio_test_dirty (folio))) { + gen =folio_inc_gen (lruvec, folio, true); +list_move (&folio->lru, &lrugen->lists[gen][type][zone]); +return true;
Make the cold dirty page cache younger instead of writeout in the backgroungd reclaimer context, and the question rising is if laundry is defered until the flusher threads are waken up in the following patches.
mm/vmscan.c static struct lruvec *get_lruvec(struct mem_cgroup *memcg, int nid) · line 3490
Quoted 4 lines
mm/vmscan.c +4
-0+ } + +return false; +}
Hillf
Re: [PATCH v7 05/12] mm: multigenerational LRU: minimal implementation
On Sun, Feb 13, 2022 at 06:04:17PM +0800, Hillf Danton wrote:
Hi Hillf,
Quoted 12 lines · On Tue, 8 Feb 2022 01:18:55 -0700 Yu Zhao wrote:
Quoted 11 lines
mm/vmscan.c +11
-0+ +/****************************************************************************** +* the aging +******************************************************************************/ + +static int folio_inc_gen (struct lruvec *lruvec,struct folio *folio,bool reclaiming) +{ +unsigned long old_flags, new_flags; +int type =folio_is_file_lru (folio); +struct lru_gen_struct *lrugen = &lruvec->lrugen; +int new_gen, old_gen =lru_gen_from_seq (lrugen->min_seq[type]);
Quoted 8 lines
mm/vmscan.c static struct lruvec *get_lruvec(struct mem_cgroup *memcg, int nid) · line 3241
Quoted 6 lines
mm/vmscan.c +6
-0+ + do { + new_flags = old_flags = READ_ONCE(folio->flags); + VM_BUG_ON_FOLIO(!(new_flags & LRU_GEN_MASK), folio); + + new_gen = ((new_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;Is the chance zero for deadloop if
new_gen!=old_gen?
No, because the counter is only cleared during isolation, and here it's protected again isolation (under the LRU lock, which is asserted in the lru_gen_balance_size() -> lru_gen_update_size() path).
Quoted 9 lines
mm/vmscan.c static struct lruvec *get_lruvec(struct mem_cgroup *memcg, int nid) · line 3247
Quoted 6 lines
mm/vmscan.c +6
-0+ new_gen = (old_gen +1 ) %MAX_NR_GENS ; + + new_flags &= ~LRU_GEN_MASK ; + new_flags |= (new_gen +1UL ) <<LRU_GEN_PGOFF ; + new_flags &= ~(LRU_REFS_MASK |LRU_REFS_FLAGS ); +/* for folio_end_writeback() */ /* for folio_end_writeback() and sort_folio() */ in terms ofreclaiming?
Right.
Quoted 11 lines
Quoted 8 lines
mm/vmscan.c +8
-0+if (reclaiming) + new_flags |=BIT (PG_reclaim); + }while (cmpxchg (&folio->flags, old_flags, new_flags) != old_flags); + +lru_gen_balance_size (lruvec, folio, old_gen, new_gen); + +return new_gen; +}...
Quoted 8 lines
mm/vmscan.c static struct lruvec *get_lruvec(struct mem_cgroup *memcg, int nid) · line 3442
Quoted 6 lines
mm/vmscan.c +6
-0+/****************************************************************************** +* the eviction +******************************************************************************/ + +static bool sort_folio (struct lruvec *lruvec,struct folio *folio,int tier_idx) +{Nit, the 80-column-char format is prefered.
Will do.
Quoted 40 lines
Quoted 40 lines
mm/vmscan.c +40
-0+bool success; +int gen =folio_lru_gen (folio); +int type =folio_is_file_lru (folio); +int zone =folio_zonenum (folio); +int tier =folio_lru_tier (folio); +int delta =folio_nr_pages (folio); +struct lru_gen_struct *lrugen = &lruvec->lrugen; + +VM_BUG_ON_FOLIO (gen >=MAX_NR_GENS , folio); + +if (!folio_evictable (folio)) { + success =lru_gen_del_folio (lruvec, folio, true); +VM_BUG_ON_FOLIO (!success, folio); +folio_set_unevictable (folio); +lruvec_add_folio (lruvec, folio); +__count_vm_events (UNEVICTABLE_PGCULLED , delta); +return true; + } + +if (type &&folio_test_anon (folio) &&folio_test_dirty (folio)) { + success =lru_gen_del_folio (lruvec, folio, true); +VM_BUG_ON_FOLIO (!success, folio); +folio_set_swapbacked (folio); +lruvec_add_folio_tail (lruvec, folio); +return true; + } + +if (tier > tier_idx) { +int hist =lru_hist_from_seq (lrugen->min_seq[type]); + + gen =folio_inc_gen (lruvec, folio, false); +list_move_tail (&folio->lru, &lrugen->lists[gen][type][zone]); + +WRITE_ONCE (lrugen->promoted[hist][type][tier -1 ], + lrugen->promoted[hist][type][tier -1 ] + delta); +__mod_lruvec_state (lruvec,WORKINGSET_ACTIVATE_BASE + type, delta); +return true; + } + + if (folio_test_locked(folio) || folio_test_writeback(folio) ||
Quoted 8 lines
mm/vmscan.c static struct lruvec *get_lruvec(struct mem_cgroup *memcg, int nid) · line 3486
Quoted 4 lines
mm/vmscan.c +4
-0+ (type && folio_test_dirty(folio))) { + gen =folio_inc_gen (lruvec, folio, true); +list_move (&folio->lru, &lrugen->lists[gen][type][zone]); +return true;Make the cold dirty page cache younger instead of writeout in the backgroungd reclaimer context, and the question rising is if laundry is defered until the flusher threads are waken up in the following patches.
This is a good point. In contrast to the active/inactive LRU, MGLRU doesn't write out dirty file pages (kswapd or direct reclaimers) -- this is writeback's job and it should be better at doing this. In fact, commit 21b4ee7029 ("xfs: drop ->writepage completely") has disabled dirty file page writeouts in the reclaim path completely.
Reclaim indirectly wakes up writeback after clean file pages drop below a threshold (dirty ratio). However, dirty pages might be under counted on a system that uses a large number of mmapped file pages. MGLRU optimizes this by calling folio_mark_dirty() on pages mapped by dirty PTEs when scanning page tables. (Why not since it's already looking at the accessed bit.)
The commit above explained this design choice from the performance aspect. From the implementation aspect, it also creates a boundary between reclaim and writeback. This simplifies things, e.g., the PageWriteback() check in shrink_page_list is no longer relevant for MGLRU, neither is the top half of the PageDirty() check.
Re: [PATCH v7 05/12] mm: multigenerational LRU: minimal implementation
In reply to Yu ZhaoHi, Yu,
Quoted 4 lines · Yu Zhao
To avoid confusions, the terms "promotion" and "demotion" will be applied to the multigenerational LRU, as a new convention; the terms "activation" and "deactivation" will be applied to the active/inactive LRU, as usual.
In the memory tiering related commits and patchset, for example as follows,
commit 668e4147d8850df32ca41e28f52c146025ca45c6
Author: Yang Shi <[email protected]> Date: Thu Sep 2 14:59:19 2021 -0700
mm/vmscan: add page demotion counter
https://lore.kernel.org/linux-mm/[email protected]/
"demote" and "promote" is used for migrating pages between different types of memory. Is it better for us to avoid overloading these words too much to avoid the possible confusion?
mm/vmscan.c static struct lruvec *get_lruvec(struct mem_cgroup *memcg, int nid) · line 3113
Quoted 5 lines
mm/vmscan.c +5
-0+static int get_swappiness (struct mem_cgroup *memcg) +{ +return mem_cgroup_get_nr_swap_pages (memcg) >=MIN_LRU_BATCH ? +mem_cgroup_swappiness (memcg) :0 ; +}
After we introduced demotion support in Linux kernel. The anonymous pages in the fast memory node could be demoted to the slow memory node via the page reclaiming mechanism as in the following commit. Can you consider that too?
commit a2a36488a61cefe3129295c6e75b3987b9d7fd13
Author: Keith Busch <[email protected]> Date: Thu Sep 2 14:59:26 2021 -0700
mm/vmscan: Consider anonymous pages without swap
Reclaim anonymous pages if a migration path is available now that demotion provides a non-swap recourse for reclaiming anon pages.
Note that this check is subtly different from the can_age_anon_pages() checks. This mechanism checks whether a specific page in a specific context can actually be reclaimed, given current swap space and cgroup limits.
can_age_anon_pages() is a much simpler and more preliminary check which just says whether there is a possibility of future reclaim.
Best Regards,
Huang, Ying
Re: [PATCH v7 05/12] mm: multigenerational LRU: minimal implementation
Quoted 16 lines · Ying
Hi, Yu,
Quoted 4 lines · Yu Zhao
To avoid confusions, the terms "promotion" and "demotion" will be applied to the multigenerational LRU, as a new convention; the terms "activation" and "deactivation" will be applied to the active/inactive LRU, as usual.
In the memory tiering related commits and patchset, for example as follows,
commit
668e4147d8850df32ca41e28f52c146025ca45c6Author: Yang Shi <[email protected]> Date: Thu Sep 2 14:59:19 2021 -0700
Quoted 7 lines
mm/vmscan: add page demotion counter
https://lore.kernel.org/linux-mm/[email protected]/
"demote" and "promote" is used for migrating pages between different types of memory. Is it better for us to avoid overloading these words too much to avoid the possible confusion?
Given that LRU and migration are usually different contexts, I think we'd be fine, unless we want a third pair of terms.
Quoted 10 lines
mm/vmscan.c static struct lruvec *get_lruvec(struct mem_cgroup *memcg, int nid) · line 3113
Quoted 5 lines
mm/vmscan.c +5
-0+static int get_swappiness (struct mem_cgroup *memcg) +{ +return mem_cgroup_get_nr_swap_pages (memcg) >=MIN_LRU_BATCH ? +mem_cgroup_swappiness (memcg) :0 ; +}After we introduced demotion support in Linux kernel. The anonymous pages in the fast memory node could be demoted to the slow memory node via the page reclaiming mechanism as in the following commit. Can you consider that too?
Sure. How do I check whether there is still space on the slow node?
Re: [PATCH v7 05/12] mm: multigenerational LRU: minimal implementation
Quoted 19 lines · Yu Zhao
Quoted 18 lines · Ying
Hi, Yu,
Quoted 4 lines · Yu Zhao
To avoid confusions, the terms "promotion" and "demotion" will be applied to the multigenerational LRU, as a new convention; the terms "activation" and "deactivation" will be applied to the active/inactive LRU, as usual.
In the memory tiering related commits and patchset, for example as follows,
commit
668e4147d8850df32ca41e28f52c146025ca45c6Author: Yang Shi <[email protected]> Date: Thu Sep 2 14:59:19 2021 -0700mm/vmscan: add page demotion counter
Quoted 8 lines
Quoted 5 lines
https://lore.kernel.org/linux-mm/[email protected]/
"demote" and "promote" is used for migrating pages between different types of memory. Is it better for us to avoid overloading these words too much to avoid the possible confusion?
Given that LRU and migration are usually different contexts, I think we'd be fine, unless we want a third pair of terms.
This is true before memory tiering is introduced. In systems with multiple types memory (called memory tiering), LRU is used to identify pages to be migrated to the slow memory node. Please take a look at can_demote(), which is called in shrink_page_list().
Quoted 11 lines
Quoted 10 lines
mm/vmscan.c static struct lruvec *get_lruvec(struct mem_cgroup *memcg, int nid) · line 3113
Quoted 5 lines
mm/vmscan.c +5
-0+static int get_swappiness (struct mem_cgroup *memcg) +{ +return mem_cgroup_get_nr_swap_pages (memcg) >=MIN_LRU_BATCH ? +mem_cgroup_swappiness (memcg) :0 ; +}After we introduced demotion support in Linux kernel. The anonymous pages in the fast memory node could be demoted to the slow memory node via the page reclaiming mechanism as in the following commit. Can you consider that too?
Quoted 1 line
Sure. How do I check whether there is still space on the slow node?
You can always check the watermark of the slow node. But now, we actually don't check that (as in demote_page_list()), instead we will wake up kswapd of the slow node. The intended behavior is something like,
DRAM -> PMEM -> disk
Best Regards,
Huang, Ying
Re: [PATCH v7 05/12] mm: multigenerational LRU: minimal implementation
Quoted 28 lines · Ying
Quoted 25 lines · Yu Zhao
Quoted 23 lines · Ying
Hi, Yu,
Quoted 4 lines · Yu Zhao
To avoid confusions, the terms "promotion" and "demotion" will be applied to the multigenerational LRU, as a new convention; the terms "activation" and "deactivation" will be applied to the active/inactive LRU, as usual.
In the memory tiering related commits and patchset, for example as follows,
commit
668e4147d8850df32ca41e28f52c146025ca45c6Author: Yang Shi <[email protected]> Date: Thu Sep 2 14:59:19 2021 -0700mm/vmscan: add page demotion counter
https://lore.kernel.org/linux-mm/[email protected]/
"demote" and "promote" is used for migrating pages between different types of memory. Is it better for us to avoid overloading these words too much to avoid the possible confusion?
Quoted 7 lines
Quoted 2 lines
Given that LRU and migration are usually different contexts, I think we'd be fine, unless we want a third pair of terms.
This is true before memory tiering is introduced. In systems with multiple types memory (called memory tiering), LRU is used to identify pages to be migrated to the slow memory node. Please take a look at
can_demote(), which is called inshrink_page_list().
This sounds clearly two contexts to me. Promotion/demotion (move between generations) while pages are on LRU; or promotion/demotion (migration between nodes) after pages are taken off LRU.
Note that promotion/demotion are not used in function names. They are used to describe how MGLRU works, in comparison with the active/inactive LRU. Memory tiering is not within this context.
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
mm/vmscan.c static struct lruvec *get_lruvec(struct mem_cgroup *memcg, int nid) · line 3113
Quoted 5 lines
mm/vmscan.c +5
-0+static int get_swappiness (struct mem_cgroup *memcg) +{ +return mem_cgroup_get_nr_swap_pages (memcg) >=MIN_LRU_BATCH ? +mem_cgroup_swappiness (memcg) :0 ; +}After we introduced demotion support in Linux kernel. The anonymous pages in the fast memory node could be demoted to the slow memory node via the page reclaiming mechanism as in the following commit. Can you consider that too?
Quoted 8 lines
Quoted 1 line
Sure. How do I check whether there is still space on the slow node?
You can always check the watermark of the slow node. But now, we actually don't check that (as in
demote_page_list()), instead we will wake up kswapd of the slow node. The intended behavior is something like,DRAM -> PMEM -> disk
I'll look into this later -- for now, it's a low priority because there isn't much demand. I'll bump it up if anybody is interested in giving it a try. Meanwhile, please feel free to cook up something if you are interested.
Re: [PATCH v7 05/12] mm: multigenerational LRU: minimal implementation
Quoted 37 lines · Yu Zhao
Quoted 28 lines · Ying
Quoted 25 lines · Yu Zhao
Quoted 23 lines · Ying
Hi, Yu,
Quoted 4 lines · Yu Zhao
To avoid confusions, the terms "promotion" and "demotion" will be applied to the multigenerational LRU, as a new convention; the terms "activation" and "deactivation" will be applied to the active/inactive LRU, as usual.
In the memory tiering related commits and patchset, for example as follows,
commit
668e4147d8850df32ca41e28f52c146025ca45c6Author: Yang Shi <[email protected]> Date: Thu Sep 2 14:59:19 2021 -0700mm/vmscan: add page demotion counter
https://lore.kernel.org/linux-mm/[email protected]/
"demote" and "promote" is used for migrating pages between different types of memory. Is it better for us to avoid overloading these words too much to avoid the possible confusion?
Quoted 7 lines
Quoted 2 lines
Given that LRU and migration are usually different contexts, I think we'd be fine, unless we want a third pair of terms.
This is true before memory tiering is introduced. In systems with multiple types memory (called memory tiering), LRU is used to identify pages to be migrated to the slow memory node. Please take a look at
can_demote(), which is called inshrink_page_list().
Quoted 7 lines
This sounds clearly two contexts to me. Promotion/demotion (move between generations) while pages are on LRU; or promotion/demotion (migration between nodes) after pages are taken off LRU.
Note that promotion/demotion are not used in function names. They are used to describe how MGLRU works, in comparison with the active/inactive LRU. Memory tiering is not within this context.
Because we have used pgdemote_* in /proc/vmstat, "demotion_enabled" in /sys/kernel/mm/numa, and will use pgpromote_* in /proc/vmstat. It seems better to avoid to use promote/demote directly for MGLRU in ABI. A possible solution is to use "mglru" and "promote/demote" together (such as "mglru_promote_*" when it is needed?
Quoted 18 lines
Quoted 18 lines
Quoted 11 lines
Quoted 10 lines
mm/vmscan.c static struct lruvec *get_lruvec(struct mem_cgroup *memcg, int nid) · line 3113
Quoted 5 lines
mm/vmscan.c +5
-0+static int get_swappiness (struct mem_cgroup *memcg) +{ +return mem_cgroup_get_nr_swap_pages (memcg) >=MIN_LRU_BATCH ? +mem_cgroup_swappiness (memcg) :0 ; +}After we introduced demotion support in Linux kernel. The anonymous pages in the fast memory node could be demoted to the slow memory node via the page reclaiming mechanism as in the following commit. Can you consider that too?
Quoted 1 line
Sure. How do I check whether there is still space on the slow node?
You can always check the watermark of the slow node. But now, we actually don't check that (as in
demote_page_list()), instead we will wake up kswapd of the slow node. The intended behavior is something like,
Quoted 6 lines
Quoted 1 line
DRAM -> PMEM -> disk
I'll look into this later -- for now, it's a low priority because there isn't much demand. I'll bump it up if anybody is interested in giving it a try. Meanwhile, please feel free to cook up something if you are interested.
When we introduce a new feature, we shouldn't break an existing one. That is, not introducing regression. I think that it is a rule?
If my understanding were correct, MGLRU will ignore to scan anonymous page list even if there's demotion target for the node. This breaks the demotion feature in the upstream kernel. Right?
It's a new feature to check whether there is still space on the slow node. We can look at that later.
Best Regards,
Huang, Ying
Re: [PATCH v7 05/12] mm: multigenerational LRU: minimal implementation
Quoted 48 lines · Ying
Quoted 37 lines · Yu Zhao
Quoted 28 lines · Ying
Quoted 25 lines · Yu Zhao
Quoted 23 lines · Ying
Hi, Yu,
Quoted 4 lines · Yu Zhao
To avoid confusions, the terms "promotion" and "demotion" will be applied to the multigenerational LRU, as a new convention; the terms "activation" and "deactivation" will be applied to the active/inactive LRU, as usual.
In the memory tiering related commits and patchset, for example as follows,
commit
668e4147d8850df32ca41e28f52c146025ca45c6Author: Yang Shi <[email protected]> Date: Thu Sep 2 14:59:19 2021 -0700mm/vmscan: add page demotion counter
https://lore.kernel.org/linux-mm/[email protected]/
"demote" and "promote" is used for migrating pages between different types of memory. Is it better for us to avoid overloading these words too much to avoid the possible confusion?
Quoted 7 lines
Quoted 2 lines
Given that LRU and migration are usually different contexts, I think we'd be fine, unless we want a third pair of terms.
This is true before memory tiering is introduced. In systems with multiple types memory (called memory tiering), LRU is used to identify pages to be migrated to the slow memory node. Please take a look at
can_demote(), which is called inshrink_page_list().Quoted 7 lines
This sounds clearly two contexts to me. Promotion/demotion (move between generations) while pages are on LRU; or promotion/demotion (migration between nodes) after pages are taken off LRU.
Note that promotion/demotion are not used in function names. They are used to describe how MGLRU works, in comparison with the active/inactive LRU. Memory tiering is not within this context.
Quoted 5 lines
Because we have used pgdemote_* in
/proc/vmstat, "demotion_enabled" in/sys/kernel/mm/numa, and will use pgpromote_* in/proc/vmstat. It seems better to avoid to use promote/demote directly for MGLRU in ABI. A possible solution is to use "mglru" and "promote/demote" together (such as "mglru_promote_*" when it is needed?
*If* it is needed. Currently there are no such plans.
Quoted 25 lines
Quoted 18 lines
Quoted 18 lines
Quoted 11 lines
Quoted 10 lines
mm/vmscan.c static struct lruvec *get_lruvec(struct mem_cgroup *memcg, int nid) · line 3113
Quoted 5 lines
mm/vmscan.c +5
-0+static int get_swappiness (struct mem_cgroup *memcg) +{ +return mem_cgroup_get_nr_swap_pages (memcg) >=MIN_LRU_BATCH ? +mem_cgroup_swappiness (memcg) :0 ; +}After we introduced demotion support in Linux kernel. The anonymous pages in the fast memory node could be demoted to the slow memory node via the page reclaiming mechanism as in the following commit. Can you consider that too?
Quoted 1 line
Sure. How do I check whether there is still space on the slow node?
You can always check the watermark of the slow node. But now, we actually don't check that (as in
demote_page_list()), instead we will wake up kswapd of the slow node. The intended behavior is something like,Quoted 6 lines
Quoted 1 line
DRAM -> PMEM -> disk
I'll look into this later -- for now, it's a low priority because there isn't much demand. I'll bump it up if anybody is interested in giving it a try. Meanwhile, please feel free to cook up something if you are interested.
Quoted 6 lines
When we introduce a new feature, we shouldn't break an existing one. That is, not introducing regression. I think that it is a rule?
If my understanding were correct, MGLRU will ignore to scan anonymous page list even if there's demotion target for the node. This breaks the demotion feature in the upstream kernel. Right?
I'm not saying this shouldn't be fixed. I'm saying it's a low priority until somebody is interested in using/testing it (or making it work).
Regarding regressions, I'm sure MGLRU *will* regress many workloads. Its goal is to improve the majority of use cases, i.e., total net gain. Trying to improve everything is methodically wrong because the problem space is near infinite but the resource is limited. So we have to prioritize major use cases over minor ones. The bottom line is users have a choice not to use MGLRU.
Quoted 2 lines
It's a new feature to check whether there is still space on the slow node. We can look at that later.
SGTM.
Re: [PATCH v7 05/12] mm: multigenerational LRU: minimal implementation
Quoted 49 lines · Yu Zhao
Quoted 48 lines · Ying
Quoted 44 lines · Yu Zhao
Quoted 28 lines · Ying
Quoted 25 lines · Yu Zhao
Quoted 23 lines · Ying
Hi, Yu,
Quoted 4 lines · Yu Zhao
To avoid confusions, the terms "promotion" and "demotion" will be applied to the multigenerational LRU, as a new convention; the terms "activation" and "deactivation" will be applied to the active/inactive LRU, as usual.
In the memory tiering related commits and patchset, for example as follows,
commit
668e4147d8850df32ca41e28f52c146025ca45c6Author: Yang Shi <[email protected]> Date: Thu Sep 2 14:59:19 2021 -0700mm/vmscan: add page demotion counter
https://lore.kernel.org/linux-mm/[email protected]/
"demote" and "promote" is used for migrating pages between different types of memory. Is it better for us to avoid overloading these words too much to avoid the possible confusion?
Quoted 7 lines
Quoted 2 lines
Given that LRU and migration are usually different contexts, I think we'd be fine, unless we want a third pair of terms.
This is true before memory tiering is introduced. In systems with multiple types memory (called memory tiering), LRU is used to identify pages to be migrated to the slow memory node. Please take a look at
can_demote(), which is called inshrink_page_list().This sounds clearly two contexts to me. Promotion/demotion (move between generations) while pages are on LRU; or promotion/demotion (migration between nodes) after pages are taken off LRU.
Note that promotion/demotion are not used in function names. They are used to describe how MGLRU works, in comparison with the active/inactive LRU. Memory tiering is not within this context.
Quoted 7 lines
Quoted 5 lines
Because we have used pgdemote_* in
/proc/vmstat, "demotion_enabled" in/sys/kernel/mm/numa, and will use pgpromote_* in/proc/vmstat. It seems better to avoid to use promote/demote directly for MGLRU in ABI. A possible solution is to use "mglru" and "promote/demote" together (such as "mglru_promote_*" when it is needed?*If* it is needed. Currently there are no such plans.
OK.
Quoted 28 lines
Quoted 28 lines
Quoted 18 lines
Quoted 18 lines
Quoted 11 lines
Quoted 10 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+static int get_swappiness(struct mem_cgroup *memcg) +{ + return mem_cgroup_get_nr_swap_pages(memcg) >= MIN_LRU_BATCH ? + mem_cgroup_swappiness(memcg) : 0; +}After we introduced demotion support in Linux kernel. The anonymous pages in the fast memory node could be demoted to the slow memory node via the page reclaiming mechanism as in the following commit. Can you consider that too?
Quoted 1 line
Sure. How do I check whether there is still space on the slow node?
You can always check the watermark of the slow node. But now, we actually don't check that (as in
demote_page_list()), instead we will wake up kswapd of the slow node. The intended behavior is something like,Quoted 6 lines
Quoted 1 line
DRAM -> PMEM -> disk
I'll look into this later -- for now, it's a low priority because there isn't much demand. I'll bump it up if anybody is interested in giving it a try. Meanwhile, please feel free to cook up something if you are interested.
When we introduce a new feature, we shouldn't break an existing one. That is, not introducing regression. I think that it is a rule?
Quoted 6 lines
Quoted 3 lines
If my understanding were correct, MGLRU will ignore to scan anonymous page list even if there's demotion target for the node. This breaks the demotion feature in the upstream kernel. Right?
I'm not saying this shouldn't be fixed. I'm saying it's a low priority until somebody is interested in using/testing it (or making it work).
We are interested in this feature and can help to test it.
Quoted 6 lines
Regarding regressions, I'm sure MGLRU *will* regress many workloads. Its goal is to improve the majority of use cases, i.e., total net gain. Trying to improve everything is methodically wrong because the problem space is near infinite but the resource is limited. So we have to prioritize major use cases over minor ones. The bottom line is users have a choice not to use MGLRU.
This is a functionality regression, not performance regression. Without demotion support, some workloads will go OOM when DRAM is used up (while PMEM isn't) if PMEM is onlined in movable zone (as recommended).
Quoted 4 lines
Quoted 2 lines
It's a new feature to check whether there is still space on the slow node. We can look at that later.
SGTM.
Best Regards,
Huang, Ying
Re: [PATCH v7 05/12] mm: multigenerational LRU: minimal implementation
Quoted 90 lines · Ying
Quoted 49 lines · Yu Zhao
Quoted 48 lines · Ying
Quoted 44 lines · Yu Zhao
Quoted 28 lines · Ying
Quoted 25 lines · Yu Zhao
Quoted 23 lines · Ying
Hi, Yu,
Quoted 4 lines · Yu Zhao
To avoid confusions, the terms "promotion" and "demotion" will be applied to the multigenerational LRU, as a new convention; the terms "activation" and "deactivation" will be applied to the active/inactive LRU, as usual.
In the memory tiering related commits and patchset, for example as follows,
commit
668e4147d8850df32ca41e28f52c146025ca45c6Author: Yang Shi <[email protected]> Date: Thu Sep 2 14:59:19 2021 -0700mm/vmscan: add page demotion counter
https://lore.kernel.org/linux-mm/[email protected]/
"demote" and "promote" is used for migrating pages between different types of memory. Is it better for us to avoid overloading these words too much to avoid the possible confusion?
Quoted 7 lines
Quoted 2 lines
Given that LRU and migration are usually different contexts, I think we'd be fine, unless we want a third pair of terms.
This is true before memory tiering is introduced. In systems with multiple types memory (called memory tiering), LRU is used to identify pages to be migrated to the slow memory node. Please take a look at
can_demote(), which is called inshrink_page_list().This sounds clearly two contexts to me. Promotion/demotion (move between generations) while pages are on LRU; or promotion/demotion (migration between nodes) after pages are taken off LRU.
Note that promotion/demotion are not used in function names. They are used to describe how MGLRU works, in comparison with the active/inactive LRU. Memory tiering is not within this context.
Quoted 7 lines
Quoted 5 lines
Because we have used pgdemote_* in
/proc/vmstat, "demotion_enabled" in/sys/kernel/mm/numa, and will use pgpromote_* in/proc/vmstat. It seems better to avoid to use promote/demote directly for MGLRU in ABI. A possible solution is to use "mglru" and "promote/demote" together (such as "mglru_promote_*" when it is needed?*If* it is needed. Currently there are no such plans.
OK.
Quoted 28 lines
Quoted 28 lines
Quoted 18 lines
Quoted 18 lines
Quoted 11 lines
Quoted 10 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+static int get_swappiness(struct mem_cgroup *memcg) +{ + return mem_cgroup_get_nr_swap_pages(memcg) >= MIN_LRU_BATCH ? + mem_cgroup_swappiness(memcg) : 0; +}After we introduced demotion support in Linux kernel. The anonymous pages in the fast memory node could be demoted to the slow memory node via the page reclaiming mechanism as in the following commit. Can you consider that too?
Quoted 1 line
Sure. How do I check whether there is still space on the slow node?
You can always check the watermark of the slow node. But now, we actually don't check that (as in
demote_page_list()), instead we will wake up kswapd of the slow node. The intended behavior is something like,Quoted 6 lines
Quoted 1 line
DRAM -> PMEM -> disk
I'll look into this later -- for now, it's a low priority because there isn't much demand. I'll bump it up if anybody is interested in giving it a try. Meanwhile, please feel free to cook up something if you are interested.
When we introduce a new feature, we shouldn't break an existing one. That is, not introducing regression. I think that it is a rule?
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
If my understanding were correct, MGLRU will ignore to scan anonymous page list even if there's demotion target for the node. This breaks the demotion feature in the upstream kernel. Right?
I'm not saying this shouldn't be fixed. I'm saying it's a low priority until somebody is interested in using/testing it (or making it work).
We are interested in this feature and can help to test it.
That's great. I'll make sure it works in the next version.
PATCH v7 6/12 mm: multigenerational LRU: exploit locality in rmap
In reply to Yu ZhaoSearching the rmap for PTEs mapping each page on an LRU list (to test and clear the accessed bit) can be expensive because pages from different VMAs aren't cache friendly to the rmap. For workloads mostly using mapped pages, the rmap has a high CPU cost in the reclaim path.
This patch exploits spatial locality to reduce the trips into the rmap. When shrink_page_list() walks the rmap and finds a young PTE, a new function lru_gen_look_around() scans at most BITS_PER_LONG-1 adjacent PTEs. On finding another young PTE, it clears the accessed bit and updates the gen counter of the page mapped by this PTE to (max_seq%MAX_NR_GENS)+1.
Server benchmark results:
Single workload:
fio (buffered I/O): no change
Single workload:
memcached (anon): +[3.5, 5.5]%
Ops/sec KB/sec
patch1-5: 972526.07 37826.95
patch1-6: 1015292.83 39490.38 Configurations:
no change
Client benchmark results:
kswapd profiles:
patch1-5
39.73% lzo1x_1_do_compress (real work)
14.96% page_vma_mapped_walk
6.97% _raw_spin_unlock_irq
3.07% do_raw_spin_lock
2.53% anon_vma_interval_tree_iter_first
2.04% ptep_clear_flush
1.82% __zram_bvec_write
1.76% __anon_vma_interval_tree_subtree_search
1.57% memmove
1.45% free_unref_page_list
patch1-6
45.49% lzo1x_1_do_compress (real work)
7.38% page_vma_mapped_walk
7.24% _raw_spin_unlock_irq
2.64% ptep_clear_flush
2.31% __zram_bvec_write
2.13% do_raw_spin_lock
2.09% lru_gen_look_around
1.89% free_unref_page_list
1.85% memmove
1.74% obj_malloc Configurations:
no change
--- include/linux/memcontrol.h | 31 ++++++++ include/linux/mm.h | 5 ++ include/linux/mmzone.h | 6 ++ include/linux/swap.h | 1 + mm/memcontrol.c | 1 + mm/rmap.c | 7 ++ mm/swap.c | 4 +- mm/vmscan.c | 155 +++++++++++++++++++++++++++++++++++++ 8 files changed, 208 insertions(+), 2 deletions(-)
include/linux/memcontrol.h +31 -0
@@ -436,6 +436,7 @@ static inline struct obj_cgroup *__folio_objcg(struct folio *folio)* - LRU isolation * - lock_page_memcg() * - exclusive reference +* - mem_cgroup_trylock_pages() * * For a kmem folio a caller should hold an rcu read lock to protect memcg * associated with a kmem folio from being released.
@@ -497,6 +498,7 @@ static inline struct mem_cgroup *folio_memcg_rcu(struct folio *folio)* - LRU isolation * - lock_page_memcg() * - exclusive reference +* - mem_cgroup_trylock_pages() * * For a kmem page a caller should hold an rcu read lock to protect memcg * associated with a kmem page from being released.
@@ -934,6 +936,23 @@ void unlock_page_memcg(struct page *page);void __mod_memcg_state (struct mem_cgroup *memcg,int idx,int val); +/* try to stablize folio_memcg() for all the pages in a memcg */ +static inline bool mem_cgroup_trylock_pages (struct mem_cgroup *memcg) +{ +rcu_read_lock (); + +if (mem_cgroup_disabled () || !atomic_read (&memcg->moving_account)) +return true; + +rcu_read_unlock (); +return false; +} + +static inline void mem_cgroup_unlock_pages (void ) +{ +rcu_read_unlock (); +} +/* idx can be of type enum memcg_stat_item or node_stat_item */ static inline void mod_memcg_state (struct mem_cgroup *memcg,int idx,int val)
@@ -1371,6 +1390,18 @@ static inline void folio_memcg_unlock(struct folio *folio) { } +static inline bool mem_cgroup_trylock_pages (struct mem_cgroup *memcg) +{ +/* to match folio_memcg_rcu() */ +rcu_read_lock (); +return true; +} + +static inline void mem_cgroup_unlock_pages (void ) +{ +rcu_read_unlock (); +} +static inline void mem_cgroup_handle_over_high (void ) { }
include/linux/mm.h +5 -0
@@ -1588,6 +1588,11 @@ static inline unsigned long folio_pfn(struct folio *folio)return page_to_pfn (&folio->page); } +static inline struct folio *pfn_folio (unsigned long pfn) +{ +return page_folio (pfn_to_page (pfn)); +} +/* MIGRATE_CMA and ZONE_MOVABLE do not allow pin pages */ #ifdef CONFIG_MIGRATION static inline bool is_pinnable_page (struct page *page)
include/linux/mmzone.h +6 -0
@@ -304,6 +304,7 @@ enum lruvec_flags { };struct lruvec ; +struct page_vma_mapped_walk ;#define LRU_GEN_MASK ((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF)#define LRU_REFS_MASK ((BIT(LRU_REFS_WIDTH) - 1) << LRU_REFS_PGOFF)
@@ -386,6 +387,7 @@ struct lru_gen_struct { };void lru_gen_init_state (struct mem_cgroup *memcg,struct lruvec *lruvec); +void lru_gen_look_around (struct page_vma_mapped_walk *pvmw);#ifdef CONFIG_MEMCG void lru_gen_init_memcg (struct mem_cgroup *memcg);
@@ -398,6 +400,10 @@ static inline void lru_gen_init_state(struct mem_cgroup *memcg, struct lruvec *l { } +static inline void lru_gen_look_around (struct page_vma_mapped_walk *pvmw) +{ +} +#ifdef CONFIG_MEMCG static inline void lru_gen_init_memcg (struct mem_cgroup *memcg) {
include/linux/swap.h +1 -0
@@ -372,6 +372,7 @@ extern void lru_add_drain(void);extern void lru_add_drain_cpu (int cpu);extern void lru_add_drain_cpu_zone (struct zone *zone);extern void lru_add_drain_all (void ); +extern void folio_activate (struct folio *folio);extern void deactivate_file_page (struct page *page);extern void deactivate_page (struct page *page);extern void mark_page_lazyfree (struct page *page);
mm/memcontrol.c +1 -0
@@ -2744,6 +2744,7 @@ static void commit_charge(struct folio *folio, struct mem_cgroup *memcg)* - LRU isolation * - lock_page_memcg() * - exclusive reference +* - mem_cgroup_trylock_pages() */ folio->memcg_data = (unsigned long )memcg; }
mm/rmap.c +7 -0
@@ -73,6 +73,7 @@#include <linux/page_idle.h> #include <linux/memremap.h> #include <linux/userfaultfd_k.h> +#include <linux/mm_inline.h> #include <asm/tlbflush.h>
@@ -819,6 +820,12 @@ static bool page_referenced_one(struct page *page, struct vm_area_struct *vma, }if (pvmw.pte) { +if (lru_gen_enabled () &&pte_young (*pvmw.pte) && + !(vma->vm_flags & (VM_SEQ_READ |VM_RAND_READ ))) { +lru_gen_look_around (&pvmw); + referenced++; + } +if (ptep_clear_flush_young_notify (vma, address, pvmw.pte)) {/*
mm/swap.c +2 -2
@@ -344,7 +344,7 @@ static bool need_activate_page_drain(int cpu)return pagevec_count (&per_cpu (lru_pvecs.activate_page, cpu)) !=0 ; } -static void folio_activate (struct folio *folio) +void folio_activate (struct folio *folio) {if (folio_test_lru (folio) && !folio_test_active (folio) && !folio_test_unevictable (folio)) {
@@ -364,7 +364,7 @@ static inline void activate_page_drain(int cpu) { } -static void folio_activate (struct folio *folio) +void folio_activate (struct folio *folio) {struct lruvec *lruvec;
mm/vmscan.c +155 -0
185 lines not on this page · raw message
Signature · 1 line
2.35.0.263.gb82422642f-goog
Re: [PATCH v7 06/12] mm: multigenerational LRU: exploit locality in rmap
On Tue, Feb 08, 2022 at 01:18:56AM -0700, Yu Zhao wrote:
<snipped>
Quoted 30 lines
include/linux/memcontrol.h +8
-0@@ -436,6 +436,7 @@ static inline struct obj_cgroup *__folio_objcg(struct folio *folio)* - LRU isolation * - lock_page_memcg() * - exclusive reference +* - mem_cgroup_trylock_pages() * * For a kmem folio a caller should hold an rcu read lock to protect memcg * associated with a kmem folio from being released. @@ -497,6 +498,7 @@ static inline struct mem_cgroup *folio_memcg_rcu(struct folio *folio)* - LRU isolation * - lock_page_memcg() * - exclusive reference +* - mem_cgroup_trylock_pages() * * For a kmem page a caller should hold an rcu read lock to protect memcg * associated with a kmem page from being released. @@ -934,6 +936,23 @@ void unlock_page_memcg(struct page *page);void __mod_memcg_state (struct mem_cgroup *memcg,int idx,int val); +/* try to stablize folio_memcg() for all the pages in a memcg */ +static inline bool mem_cgroup_trylock_pages (struct mem_cgroup *memcg) +{ +rcu_read_lock (); + +if (mem_cgroup_disabled () || !atomic_read (&memcg->moving_account))
include/linux/memcontrol.h void unlock_page_memcg(struct page *page); · line 954
Quoted 10 lines
include/linux/memcontrol.h +10
-0+return true; + +rcu_read_unlock (); +return false; +} + +static inline void mem_cgroup_unlock_pages (void ) +{ +rcu_read_unlock (); +}
Replaced the open-coded folio_memcg() lock with a new function mem_cgroup_trylock_pages() as requested here:
https://lore.kernel.org/linux-mm/YeATr%2F%[email protected]/
<snipped>
PATCH v7 7/12 mm: multigenerational LRU: support page table walks
In reply to Yu ZhaoTo avoid confusions, the term "iteration" specifically means the traversal of an entire mm_struct list; the term "walk" will be applied to page tables and the rmap, as usual.
To further exploit spatial locality, the aging prefers to walk page tables to search for young PTEs and promote hot pages. A runtime switch will be added in the next patch to enable or disable this feature. Without it, the aging relies on the rmap only.
NB: this feature has nothing similar with the page table scanning in the 2.4 kernel [1], which searches page tables for old PTEs, adds cold pages to swapcache and unmap them.
An mm_struct list is maintained for each memcg, and an mm_struct follows its owner task to the new memcg when this task is migrated. Given an lruvec, the aging iterates lruvec_memcg()->mm_list and calls walk_page_range() with each mm_struct on this list to promote hot pages before it increments max_seq.
When multiple page table walkers (threads) iterate the same list, each of them gets a unique mm_struct; therefore they can run concurrently. Page table walkers ignore any misplaced pages, e.g., if an mm_struct was migrated, pages it left in the previous memcg won't be promoted when its current memcg is under reclaim. Similarly, page table walkers won't promote pages from nodes other than the one under reclaim.
This patch uses the following optimizations when walking page tables:
1) It tracks the usage of mm_struct's between context switches so that page table walkers can skip processes that have been sleeping since the last iteration.
2) It uses generational Bloom filters to record populated branches so that page table walkers can reduce their search space based on the query results, e.g., to skip page tables containing mostly holes or misplaced pages.
3) It takes advantage of the accessed bit in non-leaf PMD entries when CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG=y.
4) It doesn't zigzag between a PGD table and the same PMD table spanning multiple VMAs. IOW, it finishes all the VMAs within the range of the same PMD table before it returns to a PGD table. This improves the cache performance for workloads that have large numbers of tiny VMAs [2], especially when CONFIG_PGTABLE_LEVELS=5.
Server benchmark results:
Single workload:
fio (buffered I/O): no change
Single workload:
memcached (anon): +[5.5, 7.5]%
Ops/sec KB/sec
patch1-6: 1015292.83 39490.38
patch1-7: 1080856.82 42040.53 Configurations:
no change
Client benchmark results:
kswapd profiles:
patch1-6
45.49% lzo1x_1_do_compress (real work)
7.38% page_vma_mapped_walk
7.24% _raw_spin_unlock_irq
2.64% ptep_clear_flush
2.31% __zram_bvec_write
2.13% do_raw_spin_lock
2.09% lru_gen_look_around
1.89% free_unref_page_list
1.85% memmove
1.74% obj_malloc
patch1-7
47.73% lzo1x_1_do_compress (real work)
6.84% page_vma_mapped_walk
6.14% _raw_spin_unlock_irq
2.86% walk_pte_range
2.79% ptep_clear_flush
2.24% __zram_bvec_write
2.10% do_raw_spin_lock
1.94% free_unref_page_list
1.80% memmove
1.75% obj_malloc Configurations:
no change
[1] https://lwn.net/Articles/23732/
[2] https://source.android.com/devices/tech/debug/scudo
--- fs/exec.c | 2 + include/linux/memcontrol.h | 5 + include/linux/mm_types.h | 78 +++ include/linux/mmzone.h | 58 +++ include/linux/swap.h | 4 + kernel/exit.c | 1 + kernel/fork.c | 9 + kernel/sched/core.c | 1 + mm/memcontrol.c | 24 + mm/vmscan.c | 963 ++++++++++++++++++++++++++++++++++++- 10 files changed, 1132 insertions(+), 13 deletions(-)
fs/exec.c +2 -0
@@ -1006,6 +1006,7 @@ static int exec_mmap(struct mm_struct *mm) active_mm = tsk->active_mm; tsk->active_mm = mm; tsk->mm = mm; +lru_gen_add_mm (mm);/* * This prevents preemption while active_mm is being loaded and * it and mm are being updated, which could cause problems for
@@ -1016,6 +1017,7 @@ static int exec_mmap(struct mm_struct *mm)if (!IS_ENABLED (CONFIG_ARCH_WANT_IRQS_OFF_ACTIVATE_MM ))local_irq_enable ();activate_mm (active_mm, mm); +lru_gen_use_mm (mm);if (IS_ENABLED (CONFIG_ARCH_WANT_IRQS_OFF_ACTIVATE_MM ))local_irq_enable (); tsk->mm->vmacache_seqnum =0 ;
include/linux/memcontrol.h +5 -0
@@ -342,6 +342,11 @@ struct mem_cgroup {struct deferred_split deferred_split_queue;#endif +#ifdef CONFIG_LRU_GEN +/* per-memcg mm_struct list */ +struct lru_gen_mm_list mm_list; +#endif +struct mem_cgroup_per_node *nodeinfo[]; };
include/linux/mm_types.h +78 -0
102 lines not on this page · raw message
include/linux/mmzone.h +58 -0
76 lines not on this page · raw message
include/linux/swap.h +4 -0
@@ -137,6 +137,10 @@ union swap_header {*/ struct reclaim_state {unsigned long reclaimed_slab; +#ifdef CONFIG_LRU_GEN +/* per-thread mm walk data */ +struct lru_gen_mm_walk *mm_walk; +#endif };#ifdef __KERNEL__
kernel/exit.c +1 -0
@@ -463,6 +463,7 @@ void mm_update_next_owner(struct mm_struct *mm) goto retry; }WRITE_ONCE (mm->owner, c); +lru_gen_migrate_mm (mm);task_unlock (c);put_task_struct (c); }
kernel/fork.c +9 -0
@@ -1079,6 +1079,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, goto fail_nocontext; mm->user_ns =get_user_ns (user_ns); +lru_gen_init_mm (mm);return mm; fail_nocontext:
@@ -1121,6 +1122,7 @@ static inline void __mmput(struct mm_struct *mm) }if (mm->binfmt)module_put (mm->binfmt->module); +lru_gen_del_mm (mm);mmdrop (mm); }
@@ -2576,6 +2578,13 @@ pid_t kernel_clone(struct kernel_clone_args *args)get_task_struct (p); } +if (IS_ENABLED (CONFIG_LRU_GEN ) && !(clone_flags &CLONE_VM )) { +/* lock the task to synchronize with memcg migration */ +task_lock (p); +lru_gen_add_mm (p->mm); +task_unlock (p); + } +wake_up_new_task (p);/* forking complete and child started to run, tell ptracer */
kernel/sched/core.c +1 -0
@@ -4970,6 +4970,7 @@ context_switch(struct rq *rq, struct task_struct *prev,* finish_task_switch()'s mmdrop(). */ switch_mm_irqs_off (prev->active_mm, next->mm, next); +lru_gen_use_mm (next->mm);if (!prev->mm) {// from kernel /* will mmdrop() in finish_task_switch(). */
mm/memcontrol.c +24 -0
@@ -6155,6 +6155,29 @@ static void mem_cgroup_move_task(void) }#endif +#ifdef CONFIG_LRU_GEN +static void mem_cgroup_attach (struct cgroup_taskset *tset) +{ +struct cgroup_subsys_state *css; +struct task_struct *task =NULL ; + +cgroup_taskset_for_each_leader (task, css, tset) +break ; + +if (!task) +return ; + +task_lock (task); +if (task->mm && task->mm->owner == task) +lru_gen_migrate_mm (task->mm); +task_unlock (task); +} +#else +static void mem_cgroup_attach (struct cgroup_taskset *tset) +{ +} +#endif /* CONFIG_LRU_GEN */ +static int seq_puts_memcg_tunable (struct seq_file *m,unsigned long value) {if (value ==PAGE_COUNTER_MAX )
@@ -6500,6 +6523,7 @@ struct cgroup_subsys memory_cgrp_subsys = { .css_reset = mem_cgroup_css_reset, .css_rstat_flush = mem_cgroup_css_rstat_flush, .can_attach = mem_cgroup_can_attach, + .attach = mem_cgroup_attach, .cancel_attach = mem_cgroup_cancel_attach, .post_attach = mem_cgroup_move_task, .dfl_cftypes = memory_files,
mm/vmscan.c +950 -13
1,125 lines not on this page · raw message
Signature · 1 line
2.35.0.263.gb82422642f-goog
Re: [PATCH v7 07/12] mm: multigenerational LRU: support page table walks
Quoted 8 lines · Yu Zhao
To avoid confusions, the term "iteration" specifically means the traversal of an entire
mm_structlist; the term "walk" will be applied to page tables and the rmap, as usual.To further exploit spatial locality, the aging prefers to walk page tables to search for young PTEs and promote hot pages. A runtime switch will be added in the next patch to enable or disable this feature. Without it, the aging relies on the rmap only.
Clarified that page table scanning is optional as requested here:
https://lore.kernel.org/linux-mm/[email protected]/
Quoted 10 lines
NB: this feature has nothing similar with the page table scanning in the 2.4 kernel [1], which searches page tables for old PTEs, adds cold pages to swapcache and unmap them.
An
mm_structlist is maintained for each memcg, and anmm_structfollows its owner task to the new memcg when this task is migrated. Given an lruvec, the aging iterates lruvec_memcg()->mm_list and callswalk_page_range()with eachmm_structon this list to promote hot pages before it incrementsmax_seq.
Quoted 6 lines
When multiple page table walkers (threads) iterate the same list, each of them gets a unique
mm_struct; therefore they can run concurrently. Page table walkers ignore any misplaced pages, e.g., if anmm_structwas migrated, pages it left in the previous memcg won't be promoted when its current memcg is under reclaim. Similarly, page table walkers won't promote pages from nodes other than the one under reclaim.
Clarified the interaction between task migration and reclaim as requested here:
https://lore.kernel.org/linux-mm/[email protected]/
<snipped>
Quoted 39 lines
Server benchmark results:
Single workload:
fio (buffered I/O): no changeSingle workload:
memcached (anon): +[5.5, 7.5]% Ops/sec KB/sec patch1-6: 1015292.83 39490.38 patch1-7: 1080856.82 42040.53Configurations:
no changeClient benchmark results:
kswapd profiles:patch1-6 45.49% lzo1x_1_do_compress (real work) 7.38% page_vma_mapped_walk 7.24% _raw_spin_unlock_irq 2.64% ptep_clear_flush 2.31% __zram_bvec_write 2.13% do_raw_spin_lock 2.09% lru_gen_look_around 1.89% free_unref_page_list 1.85% memmove 1.74% obj_malloc patch1-7 47.73% lzo1x_1_do_compress (real work) 6.84% page_vma_mapped_walk 6.14% _raw_spin_unlock_irq 2.86% walk_pte_range 2.79% ptep_clear_flush 2.24% __zram_bvec_write 2.10% do_raw_spin_lock 1.94% free_unref_page_list 1.80% memmove 1.75% obj_malloc
Quoted 2 lines
Configurations:
no change
Added benchmark results to show the difference between page table scanning and no page table scanning, as requested here:
https://lore.kernel.org/linux-mm/[email protected]/
<snipped>
Quoted 8 lines
mm/vmscan.c +8
-0+static void walk_mm (struct lruvec *lruvec,struct mm_struct *mm,struct lru_gen_mm_walk *walk) +{ +static const struct mm_walk_ops mm_walk_ops = { + .test_walk = should_skip_vma, + .p4d_entry = walk_pud_range, + }; + +int err;
mm/vmscan.c static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool reclai · line 4078
Quoted 10 lines
mm/vmscan.c +10
-0+struct mem_cgroup *memcg =lruvec_memcg (lruvec); + + walk->next_addr =FIRST_USER_ADDRESS ; + + do { + err = -EBUSY; + + /* folio_update_gen() requires stable folio_memcg() */ + if (!mem_cgroup_trylock_pages(memcg)) + break;
Added a comment on the stable folio_memcg() requirement as requested here:
https://lore.kernel.org/linux-mm/[email protected]/
<snipped>
mm/vmscan.c static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool reclai · line 4109
Quoted 8 lines
mm/vmscan.c +8
-0+static struct lru_gen_mm_walk *alloc_mm_walk (void ) +{ +if (current->reclaim_state && current->reclaim_state->mm_walk) +return current->reclaim_state->mm_walk; + +return kzalloc (sizeof (struct lru_gen_mm_walk ), + __GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN); +}
Replaced kvzalloc() with kzalloc() as requested here:
https://lore.kernel.org/linux-mm/[email protected]/
Replaced GFP_KERNEL with __GFP_HIGH|__GFP_NOMEMALLOC|__GFP_NOWARN as requested here:
https://lore.kernel.org/linux-mm/[email protected]/
<snipped>
PATCH v7 8/12 mm: multigenerational LRU: optimize multiple memcgs
In reply to Yu ZhaoWhen multiple memcgs are available, it's possible to improve the overall performance under global memory pressure by making better choices based on generations and tiers. This patch adds a rudimentary optimization to select memcgs that can drop single-use unmapped clean pages first, and thus it reduces the chance of going into the aging path or swapping, which can be costly. Its goal is to improve the overall performance when there are mixed types of workloads, e.g., heavy anon workload in one memcg and heavy buffered I/O workload in the other.
Though this optimization can be applied to both kswapd and direct reclaim, it's only added to kswapd to keep the patchset manageable. Later improvements will cover the direct reclaim path.
Server benchmark results:
Mixed workloads:
fio (buffered I/O): -[28, 30]%
IOPS BW
patch1-7: 3117k 11.9GiB/s
patch1-8: 2217k 8661MiB/s
memcached (anon): +[247, 251]%
Ops/sec KB/sec
patch1-7: 563772.35 21900.01
patch1-8: 1968343.76 76461.24Mixed workloads:
fio (buffered I/O): -[4, 6]%
IOPS BW
5.17-rc2: 2338k 9133MiB/s
patch1-8: 2217k 8661MiB/s
memcached (anon): +[524, 530]%
Ops/sec KB/sec
5.17-rc2: 313821.65 12190.55
patch1-8: 1968343.76 76461.24Configurations:
(changes since patch 5)
cat combined.sh
modprobe brd rd_nr=2 rd_size=56623104
swapoff -a
mkswap /dev/ram0
swapon /dev/ram0
mkfs.ext4 /dev/ram1
mount -t ext4 /dev/ram1 /mnt
memtier_benchmark -S /var/run/memcached/memcached.sock \
-P memcache_binary -n allkeys --key-minimum=1 \
--key-maximum=50000000 --key-pattern=P:P -c 1 -t 36 \
--ratio 1:0 --pipeline 8 -d 2000
fio -name=mglru --numjobs=36 --directory=/mnt --size=1408m \
--buffered=1 --ioengine=io_uring --iodepth=128 \
--iodepth_batch_submit=32 --iodepth_batch_complete=32 \
--rw=randread --random_distribution=random --norandommap \
--time_based --ramp_time=10m --runtime=90m --group_reporting &
pid=$!
sleep 200
memtier_benchmark -S /var/run/memcached/memcached.sock \
-P memcache_binary -n allkeys --key-minimum=1 \
--key-maximum=50000000 --key-pattern=R:R -c 1 -t 36 \
--ratio 0:1 --pipeline 8 --randomize --distinct-client-seed
kill -INT $pid
waitClient benchmark results:
no change (CONFIG_MEMCG=n)
--- mm/vmscan.c | 45 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 41 insertions(+), 4 deletions(-)
mm/vmscan.c +41 -4
@@ -127,6 +127,13 @@ struct scan_control {/* Always discard instead of demoting to lower tier memory */ unsigned int no_demotion:1; +#ifdef CONFIG_LRU_GEN +/* help make better choices when multiple memcgs are available */ + unsigned int memcgs_need_aging:1; + unsigned int memcgs_need_swapping:1; + unsigned int memcgs_avoid_swapping:1; +#endif +/* Allocation order */ s8 order;
@@ -4343,6 +4350,22 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc)VM_BUG_ON (!current_is_kswapd ()); +/* +* To reduce the chance of going into the aging path or swapping, which +* can be costly, optimistically skip them unless their corresponding +* flags were cleared in the eviction path. This improves the overall +* performance when multiple memcgs are available. +*/ +if (!sc->memcgs_need_aging) { + sc->memcgs_need_aging = true; + sc->memcgs_avoid_swapping = !sc->memcgs_need_swapping; + sc->memcgs_need_swapping = true; +return ; + } + + sc->memcgs_need_swapping = true; + sc->memcgs_avoid_swapping = true; + current->reclaim_state->mm_walk = &pgdat->mm_walk; memcg =mem_cgroup_iter (NULL ,NULL ,NULL );
@@ -4745,7 +4768,8 @@ static int isolate_folios(struct lruvec *lruvec, struct scan_control *sc, int swreturn scanned; } -static int evict_folios (struct lruvec *lruvec,struct scan_control *sc,int swappiness) +static int evict_folios (struct lruvec *lruvec,struct scan_control *sc,int swappiness, +bool *swapped) {int type;int scanned;
@@ -4810,6 +4834,9 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap sc->nr_reclaimed += reclaimed; +if (!type && swapped) + *swapped = true; +return scanned; }
@@ -4838,8 +4865,10 @@ static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc, boolif (!nr_to_scan)return 0 ; -if (!need_aging) +if (!need_aging) { + sc->memcgs_need_aging = false;return nr_to_scan; + }/* leave the work to lru_gen_age_node() */ if (current_is_kswapd ())
@@ -4861,6 +4890,8 @@ static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc {struct blk_plug plug;long scanned =0 ; +bool swapped = false; +unsigned long reclaimed = sc->nr_reclaimed;struct mem_cgroup *memcg =lruvec_memcg (lruvec);struct pglist_data *pgdat =lruvec_pgdat (lruvec);
@@ -4887,13 +4918,19 @@ static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *scif (!nr_to_scan)break ; - delta =evict_folios (lruvec, sc, swappiness); + delta =evict_folios (lruvec, sc, swappiness, &swapped);if (!delta)break ; +if (sc->memcgs_avoid_swapping && swappiness <200 && swapped) +break ; + scanned += delta; -if (scanned >= nr_to_scan) +if (scanned >= nr_to_scan) { +if (!swapped && sc->nr_reclaimed - reclaimed >=MIN_LRU_BATCH ) + sc->memcgs_need_swapping = false;break ; + }cond_resched (); }
Signature · 1 line
2.35.0.263.gb82422642f-goog
PATCH v7 9/12 mm: multigenerational LRU: runtime switch
In reply to Yu ZhaoAdd /sys/kernel/mm/lru_gen/enabled as a runtime switch. Features that can be enabled or disabled include:
0x0001: the multigenerational LRU
0x0002: the page table walks, when arch_has_hw_pte_young() returns
true
0x0004: the use of the accessed bit in non-leaf PMD entries, when
CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG=y
[yYnN]: apply to all the features above
E.g.,
echo y >/sys/kernel/mm/lru_gen/enabled
cat /sys/kernel/mm/lru_gen/enabled
0x0007
echo 5 >/sys/kernel/mm/lru_gen/enabled
cat /sys/kernel/mm/lru_gen/enabled
0x0005
NB: the page table walks happen on the scale of seconds under heavy memory pressure. Under such a condition, the mmap_lock contention is a lesser concern, compared with the LRU lock contention and the I/O congestion. So far the only well-known case of the mmap_lock contention is Android, due to Scudo [1] which allocates several thousand VMAs for merely a few hundred MBs. The SPF and the Maple Tree also have provided their own assessments [2][3]. However, if the page table walks do worsen the mmap_lock contention, the runtime switch can be used to disable this feature. In this case the multigenerational LRU will suffer a minor performance degradation, as shown previously.
The use of the accessed bit in non-leaf PMD entries can also be disabled, since this feature wasn't tested on x86 varieties other than Intel and AMD.
[1] https://source.android.com/devices/tech/debug/scudo
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/
--- include/linux/cgroup.h | 15 +- include/linux/mm_inline.h | 10 +- include/linux/mmzone.h | 7 + kernel/cgroup/cgroup-internal.h | 1 - mm/Kconfig | 6 + mm/vmscan.c | 236 +++++++++++++++++++++++++++++++- 6 files changed, 267 insertions(+), 8 deletions(-)
include/linux/cgroup.h +14 -1
@@ -432,6 +432,18 @@ static inline void cgroup_put(struct cgroup *cgrp)css_put (&cgrp->self); } +extern struct mutex cgroup_mutex; + +static inline void cgroup_lock (void ) +{ +mutex_lock (&cgroup_mutex); +} + +static inline void cgroup_unlock (void ) +{ +mutex_unlock (&cgroup_mutex); +} +/** * task_css_set_check - obtain a task's css_set with extra access conditions * @task: the task to obtain css_set for
@@ -446,7 +458,6 @@ static inline void cgroup_put(struct cgroup *cgrp)* as locks used during the cgroup_subsys::attach() methods. */ #ifdef CONFIG_PROVE_RCU -extern struct mutex cgroup_mutex;extern spinlock_t css_set_lock;#define task_css_set_check (task, __c) \ rcu_dereference_check((task)->cgroups, \
@@ -707,6 +718,8 @@ struct cgroup;static inline u64 cgroup_id (const struct cgroup *cgrp) {return 1 ; }static inline void css_get (struct cgroup_subsys_state *css) {}static inline void css_put (struct cgroup_subsys_state *css) {} +static inline void cgroup_lock (void ) {} +static inline void cgroup_unlock (void ) {}static inline int cgroup_attach_task_all (struct task_struct *from,struct task_struct *t) {return 0 ; } static inline int cgroupstats_build(struct cgroupstats *stats,
include/linux/mm_inline.h +9 -1
@@ -96,7 +96,15 @@ static __always_inline enum lru_list folio_lru_list(struct folio *folio)static inline bool lru_gen_enabled (void ) { -return true; +#ifdef CONFIG_LRU_GEN_ENABLED +DECLARE_STATIC_KEY_TRUE (lru_gen_caps[NR_LRU_GEN_CAPS ]); + +return static_branch_likely (&lru_gen_caps[LRU_GEN_CORE ]); +#else +DECLARE_STATIC_KEY_FALSE (lru_gen_caps[NR_LRU_GEN_CAPS ]); + +return static_branch_unlikely (&lru_gen_caps[LRU_GEN_CORE ]); +#endif }static inline bool lru_gen_in_fault (void )
include/linux/mmzone.h +7 -0
@@ -311,6 +311,13 @@ struct page_vma_mapped_walk;#ifdef CONFIG_LRU_GEN +enum { +LRU_GEN_CORE , +LRU_GEN_MM_WALK , +LRU_GEN_NONLEAF_YOUNG , +NR_LRU_GEN_CAPS +}; +#define MIN_LRU_BATCH BITS_PER_LONG#define MAX_LRU_BATCH (MIN_LRU_BATCH * 128)
kernel/cgroup/cgroup-internal.h +0 -1
@@ -165,7 +165,6 @@ struct cgroup_mgctx {#define DEFINE_CGROUP_MGCTX (name) \ struct cgroup_mgctx name = CGROUP_MGCTX_INIT(name) -extern struct mutex cgroup_mutex;extern spinlock_t css_set_lock;extern struct cgroup_subsys *cgroup_subsys[];extern struct list_head cgroup_roots;
mm/Kconfig +6 -0
@@ -903,6 +903,12 @@ config LRU_GEN Documentation/admin-guide/mm/multigen_lru.rst and Documentation/vm/multigen_lru.rst for details. +config LRU_GEN_ENABLED + bool "Enable by default" + depends on LRU_GEN + help + This option enables the multigenerational LRU by default. + config NR_LRU_GENS int "Max number of generations" depends on LRU_GEN
mm/vmscan.c +231 -5
279 lines not on this page · raw message
Signature · 1 line
2.35.0.263.gb82422642f-goog
Re: [PATCH v7 09/12] mm: multigenerational LRU: runtime switch
Quoted 16 lines · Yu Zhao
Add
/sys/kernel/mm/lru_gen/enabledas a runtime switch. Features that can be enabled or disabled include:
0x0001: the multigenerational LRU
0x0002: the page table walks, whenarch_has_hw_pte_young()returns
true
0x0004: the use of the accessed bit in non-leaf PMD entries, when
CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG=y
[yYnN]: apply to all the features above
E.g.,
echo y >/sys/kernel/mm/lru_gen/enabled
cat/sys/kernel/mm/lru_gen/enabled
0x0007
echo 5 >/sys/kernel/mm/lru_gen/enabled
cat/sys/kernel/mm/lru_gen/enabled
0x0005
Quoted 10 lines
NB: the page table walks happen on the scale of seconds under heavy memory pressure. Under such a condition, the
mmap_lockcontention is a lesser concern, compared with the LRU lock contention and the I/O congestion. So far the only well-known case of themmap_lockcontention is Android, due to Scudo [1] which allocates several thousand VMAs for merely a few hundred MBs. The SPF and the Maple Tree also have provided their own assessments [2][3]. However, if the page table walks do worsen themmap_lockcontention, the runtime switch can be used to disable this feature. In this case the multigenerational LRU will suffer a minor performance degradation, as shown previously.
Clarified the potential impact from the mmap_lock contention as requested here:
https://lore.kernel.org/linux-mm/[email protected]/
<snipped>
Quoted 8 lines
mm/vmscan.c +8
-0+static void lru_gen_change_state (bool enable) +{ +static DEFINE_MUTEX (state_mutex); + +struct mem_cgroup *memcg; + +cgroup_lock (); +cpus_read_lock ();
mm/vmscan.c static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc · line 5069
Quoted 10 lines
mm/vmscan.c +10
-0+get_online_mems (); +mutex_lock (&state_mutex); + +if (enable ==lru_gen_enabled ()) + goto unlock; + +if (enable) +static_branch_enable_cpuslocked (&lru_gen_caps[LRU_GEN_CORE ]); +else +static_branch_disable_cpuslocked (&lru_gen_caps[LRU_GEN_CORE ]);
Fixed the lockdep warning for memory hotplug:
https://lore.kernel.org/linux-mm/[email protected]/
<snipped>
PATCH v7 10/12 mm: multigenerational LRU: thrashing prevention
In reply to Yu ZhaoAdd /sys/kernel/mm/lru_gen/min_ttl_ms for thrashing prevention, as requested by many desktop users [1].
When set to value N, it prevents the working set of N milliseconds from getting evicted. The OOM killer is triggered if this working set can't be kept in memory. Based on the average human detectable lag (~100ms), N=1000 usually eliminates intolerable lags due to thrashing. Larger values like N=3000 make lags less noticeable at the risk of premature OOM kills.
Compared with the size-based approach, e.g., [2], this time-based approach has the following advantages:
1) It's easier to configure because it's agnostic to applications and memory sizes.
2) It's more reliable because it's directly wired to the OOM killer.
[1] https://lore.kernel.org/lkml/Ydza%[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
--- mm/vmscan.c | 64 +++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 60 insertions(+), 4 deletions(-)
mm/vmscan.c +60 -4
@@ -4335,7 +4335,8 @@ static long get_nr_evictable(struct lruvec *lruvec, unsigned long max_seq,return total >0 ? total :0 ; } -static void age_lruvec (struct lruvec *lruvec,struct scan_control *sc) +static bool age_lruvec (struct lruvec *lruvec,struct scan_control *sc, +unsigned long min_ttl) {bool need_aging;long nr_to_scan;
@@ -4344,14 +4345,22 @@ static void age_lruvec(struct lruvec *lruvec, struct scan_control *sc)DEFINE_MAX_SEQ (lruvec);DEFINE_MIN_SEQ (lruvec); +if (min_ttl) { +int gen =lru_gen_from_seq (min_seq[TYPE_FILE ]); +unsigned long birth =READ_ONCE (lruvec->lrugen.timestamps[gen]); + +if (time_is_after_jiffies (birth + min_ttl)) +return false; + } +mem_cgroup_calculate_protection (NULL , memcg);if (mem_cgroup_below_min (memcg)) -return ; +return false; nr_to_scan =get_nr_evictable (lruvec, max_seq, min_seq, swappiness, &need_aging);if (!nr_to_scan) -return ; +return false; nr_to_scan >>= sc->priority;
@@ -4360,11 +4369,18 @@ static void age_lruvec(struct lruvec *lruvec, struct scan_control *sc)if (nr_to_scan && need_aging && (!mem_cgroup_below_low (memcg) || sc->memcg_low_reclaim))try_to_inc_max_seq (lruvec, max_seq, sc, swappiness, false); + +return true; } +/* to protect the working set of the last N jiffies */ +static unsigned long lru_gen_min_ttl__read_mostly ; +static void lru_gen_age_node (struct pglist_data *pgdat,struct scan_control *sc) {struct mem_cgroup *memcg; +bool success = false; +unsigned long min_ttl =READ_ONCE (lru_gen_min_ttl);VM_BUG_ON (!current_is_kswapd ());
@@ -4390,11 +4406,28 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc)do {struct lruvec *lruvec =mem_cgroup_lruvec (memcg, pgdat); -age_lruvec (lruvec, sc); +if (age_lruvec (lruvec, sc, min_ttl)) + success = true;cond_resched (); }while ((memcg =mem_cgroup_iter (NULL , memcg,NULL ))); +/* +* The main goal is to OOM kill if every generation from all memcgs is +* younger than min_ttl. However, another theoretical possibility is all +* memcgs are either below min or empty. +*/ +if (!success &&mutex_trylock (&oom_lock)) { +struct oom_control oc = { + .gfp_mask = sc->gfp_mask, + .order = sc->order, + }; + +out_of_memory (&oc); + +mutex_unlock (&oom_lock); + } + current->reclaim_state->mm_walk =NULL ; }
@@ -5107,6 +5140,28 @@ static void lru_gen_change_state(bool enable)* sysfs interface ******************************************************************************/ +static ssize_t show_min_ttl (struct kobject *kobj,struct kobj_attribute *attr,char *buf) +{ +return sprintf (buf,"%u\n" ,jiffies_to_msecs (READ_ONCE (lru_gen_min_ttl))); +} + +static ssize_t store_min_ttl (struct kobject *kobj,struct kobj_attribute *attr, +const char *buf,size_t len) +{ +unsigned int msecs; + +if (kstrtouint (buf,0 , &msecs)) +return -EINVAL ; + +WRITE_ONCE (lru_gen_min_ttl,msecs_to_jiffies (msecs)); + +return len; +} + +static struct kobj_attribute lru_gen_min_ttl_attr =__ATTR ( + min_ttl_ms,0644 , show_min_ttl, store_min_ttl +); +static ssize_t show_enable (struct kobject *kobj,struct kobj_attribute *attr,char *buf) {unsigned int caps =0 ;
@@ -5155,6 +5210,7 @@ static struct kobj_attribute lru_gen_enabled_attr = __ATTR( );static struct attribute *lru_gen_attrs[] = { + &lru_gen_min_ttl_attr.attr, &lru_gen_enabled_attr.attr,NULL };
Signature · 1 line
2.35.0.263.gb82422642f-goog
Re: [PATCH v7 10/12] mm: multigenerational LRU: thrashing prevention
Quoted 9 lines · Yu Zhao
Add
/sys/kernel/mm/lru_gen/min_ttl_msfor thrashing prevention, as requested by many desktop users [1].When set to value N, it prevents the working set of N milliseconds from getting evicted. The OOM killer is triggered if this working set can't be kept in memory. Based on the average human detectable lag (~100ms), N=1000 usually eliminates intolerable lags due to thrashing. Larger values like N=3000 make lags less noticeable at the risk of premature OOM kills.
Refactored min_ttl into a separate patch as requested here:
https://lore.kernel.org/linux-mm/[email protected]/
<snipped>
PATCH v7 11/12 mm: multigenerational LRU: debugfs interface
In reply to Yu ZhaoAdd /sys/kernel/debug/lru_gen for working set estimation and proactive reclaim. These features are required to optimize job scheduling (bin packing) in data centers [1][2].
Compared with the page table-based approach and the PFN-based approach, e.g., mm/damon/[vp]addr.c, this lruvec-based approach has the following advantages:
1) It offers better choices because it's aware of memcgs, NUMA nodes, shared mappings and unmapped page cache.
2) It's more scalable because it's O(nr_hot_pages), whereas the PFN-based approach is O(nr_total_pages).
Add /sys/kernel/debug/lru_gen_full for debugging.
[1] https://research.google/pubs/pub48551/
[2] https://www.cs.cmu.edu/~dskarlat/publications/tmo_asplos22.pdf
--- include/linux/nodemask.h | 1 + mm/vmscan.c | 353 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 354 insertions(+)
include/linux/nodemask.h +1 -0
@@ -486,6 +486,7 @@ static inline int num_node_state(enum node_states state)#define first_online_node 0#define first_memory_node 0#define next_online_node (nid) (MAX_NUMNODES) +#define next_memory_node (nid) (MAX_NUMNODES)#define nr_node_ids 1U#define nr_online_nodes 1U
mm/vmscan.c +353 -0
371 lines not on this page · raw message
Signature · 1 line
2.35.0.263.gb82422642f-goog
Re: [page-reclaim] [PATCH v7 11/12] mm: multigenerational LRU: debugfs interface
Quoted 4 lines · Yu Zhao
Add
/sys/kernel/debug/lru_genfor working set estimation and proactive reclaim. These features are required to optimize job scheduling (bin packing) in data centers [1][2].
Johannes, do you believe this interface is sufficient to induce memcg based reclaim or do you have plans to propose a memory.reclaim extension to memcg?
I assume that lru_gen could be used even if memcg is disabled whereas memory.reclaim would only be available on systems where the controller is mounted through cgroups.
Yet a question would probably arise about the stability of this interface since it lives in debugfs: can we plan on /sys/kernel/debug/lru_gen being long-term supported?
PATCH v7 12/12 mm: multigenerational LRU: documentation
In reply to Yu ZhaoAdd a design doc and an admin guide.
--- Documentation/admin-guide/mm/index.rst | 1 + Documentation/admin-guide/mm/multigen_lru.rst | 121 ++++++++++++++ Documentation/vm/index.rst | 1 + Documentation/vm/multigen_lru.rst | 152 ++++++++++++++++++ 4 files changed, 275 insertions(+) create mode 100644 Documentation/admin-guide/mm/multigen_lru.rst create mode 100644 Documentation/vm/multigen_lru.rst
Documentation/admin-guide/mm/index.rst +1 -0
@@ -32,6 +32,7 @@ the Linux memory management. idle_page_tracking ksm memory-hotplug + multigen_lru nommu-mmap numa_memory_policy numaperf
Documentation/admin-guide/mm/multigen_lru.rst new file +121 -0
121 lines not on this page · raw message
Documentation/vm/index.rst +1 -0
@@ -25,6 +25,7 @@ algorithms. If you are looking for advice on simply allocating memory, see the ksm memory-model mmu_notifier + multigen_lru numa overcommit-accounting page_migration
Documentation/vm/multigen_lru.rst new file +152 -0
152 lines not on this page · raw message
Signature · 1 line
2.35.0.263.gb82422642f-goog
Re: [PATCH v7 12/12] mm: multigenerational LRU: documentation
Quoted 14 lines · Yu Zhao
Add a design doc and an admin guide.
Signed-off-by Yu Zhao <[email protected]>Acked-by Brian Geffon <[email protected]>Acked-by Steven Barrett <[email protected]>Acked-by Suleiman Souhlal <[email protected]>Tested-by Daniel Byrne <[email protected]>Tested-by Donald Carr <[email protected]>Tested-by Holger Hoffstätte <[email protected]>Tested-by Konstantin Kharlamov <[email protected]>Tested-by Shuang Zhai <[email protected]>Tested-by Sofia Trinh <[email protected]>
Quoted 8 lines
--- Documentation/admin-guide/mm/index.rst | 1 + Documentation/admin-guide/mm/multigen_lru.rst | 121 ++++++++++++++ Documentation/vm/index.rst | 1 + Documentation/vm/multigen_lru.rst | 152 ++++++++++++++++++ 4 files changed, 275 insertions(+) create mode 100644 Documentation/admin-guide/mm/multigen_lru.rst create mode 100644 Documentation/vm/multigen_lru.rst
Refactored the doc into a separate patch as requested here:
https://lore.kernel.org/linux-mm/[email protected]/
Reworked the doc as requested here:
https://lore.kernel.org/linux-mm/[email protected]/
<snipped>
Mike Rapoport
maintainerRe: [PATCH v7 12/12] mm: multigenerational LRU: documentation
In reply to Yu ZhaoHi,
Quoted 14 lines · Yu Zhao
Add a design doc and an admin guide.
Signed-off-by Yu Zhao <[email protected]>Acked-by Brian Geffon <[email protected]>Acked-by Steven Barrett <[email protected]>Acked-by Suleiman Souhlal <[email protected]>Tested-by Daniel Byrne <[email protected]>Tested-by Donald Carr <[email protected]>Tested-by Holger Hoffstätte <[email protected]>Tested-by Konstantin Kharlamov <[email protected]>Tested-by Shuang Zhai <[email protected]>Tested-by Sofia Trinh <[email protected]>
Quoted 5 lines
--- Documentation/admin-guide/mm/index.rst | 1 + Documentation/admin-guide/mm/multigen_lru.rst | 121 ++++++++++++++ Documentation/vm/index.rst | 1 + Documentation/vm/multigen_lru.rst | 152 ++++++++++++++++++
Please consider splitting this patch into Documentation/admin-guide and Documentation/vm parts.
For now I only had time to review the admin-guide part.
Quoted 22 lines
4 files changed, 275 insertions(+) create mode 100644 Documentation/admin-guide/mm/multigen_lru.rst create mode 100644 Documentation/vm/multigen_lru.rstDocumentation/admin-guide/mm/index.rst +1
-0@@ -32,6 +32,7 @@ the Linux memory management. idle_page_tracking ksm memory-hotplug + multigen_lru nommu-mmap numa_memory_policy numaperfDocumentation/admin-guide/mm/multigen_lru.rst new file +0
-0
Documentation/admin-guide/mm/multigen_lru.rst line 5
Quoted 5 lines
Documentation/admin-guide/mm/multigen_lru.rst new file +5
-0@@ -0,0 +1,121 @@ +.. SPDX-License-Identifier: GPL-2.0 + +===================== +Multigenerational LRU +=====================
+
Documentation/admin-guide/mm/multigen_lru.rst line 8
Quoted 2 lines
Documentation/admin-guide/mm/multigen_lru.rst new file +2
-0+Quick start +===========
There is no explanation why one would want to use multigenerational LRU until the next section.
I think there should be an overview that explains why users would want to enable multigenerational LRU.
Documentation/admin-guide/mm/multigen_lru.rst line 11
Quoted 3 lines
Documentation/admin-guide/mm/multigen_lru.rst new file +3
-0+Build configurations +-------------------- +:Required: Set ``CONFIG_LRU_GEN=y``.
Maybe
Set ``CONFIG_LRU_GEN=y`` to build kernel with multigenerational LRU
Documentation/admin-guide/mm/multigen_lru.rst line 22
Quoted 11 lines
Documentation/admin-guide/mm/multigen_lru.rst new file +11
-0+ +:Optional: Set ``CONFIG_LRU_GEN_ENABLED=y`` to enable the + multigenerational LRU by default. + +Runtime configurations +---------------------- +:Required: Write ``y`` to ``/sys/kernel/mm/lru_gen/enable`` if + ``CONFIG_LRU_GEN_ENABLED=n``. + +This file accepts different values to enabled or disabled the +following features:
Maybe
After multigenerational LRU is enabled, this file accepts different values to enable or disable the following feaures:
Documentation/admin-guide/mm/multigen_lru.rst line 27
Quoted 4 lines
Documentation/admin-guide/mm/multigen_lru.rst new file +4
-0+====== ======== +Values Features +====== ======== +0x0001 the multigenerational LRU
The multigenerational LRU what?
What will happen if I write 0x2 to this file? Please consider splitting "enable" and "features" attributes.
Documentation/admin-guide/mm/multigen_lru.rst line 29
Quoted 2 lines
Documentation/admin-guide/mm/multigen_lru.rst new file +2
-0+0x0002 clear the accessed bit in leaf page table entries **in large + batches**, when MMU sets it (e.g., on x86)
Is extra markup really needed here...
Documentation/admin-guide/mm/multigen_lru.rst line 31
Quoted 2 lines
Documentation/admin-guide/mm/multigen_lru.rst new file +2
-0+0x0004 clear the accessed bit in non-leaf page table entries **as + well**, when MMU sets it (e.g., on x86)
... and here?
As for the descriptions, what is the user-visible effect of these features? How different modes of clearing the access bit are reflected in, say, GUI responsiveness, database TPS, or probability of OOM?
Quoted 14 lines
Documentation/admin-guide/mm/multigen_lru.rst new file +14
-0+[yYnN] apply to all the features above +====== ======== + +E.g., +:: + + echo y >/sys/kernel/mm/lru_gen/enabled + cat /sys/kernel/mm/lru_gen/enabled + 0x0007 + echo 5 >/sys/kernel/mm/lru_gen/enabled + cat /sys/kernel/mm/lru_gen/enabled + 0x0005 + +Most users should enable or disable all the features unless some of
Documentation/admin-guide/mm/multigen_lru.rst line 55
Quoted 10 lines
Documentation/admin-guide/mm/multigen_lru.rst new file +10
-0+them have unforeseen side effects. + +Recipes +======= +Personal computers +------------------ +Personal computers are more sensitive to thrashing because it can +cause janks (lags when rendering UI) and negatively impact user +experience. The multigenerational LRU offers thrashing prevention to +the majority of laptop and desktop users who don't have oomd.
I'd expect something like this paragraph in overview.
Quoted 9 lines
Documentation/admin-guide/mm/multigen_lru.rst new file +8
-0+ +:Thrashing prevention: Write ``N`` to + ``/sys/kernel/mm/lru_gen/min_ttl_ms`` to prevent the working set of + ``N`` milliseconds from getting evicted. The OOM killer is triggered + if this working set can't be kept in memory. Based on the average + human detectable lag (~100ms), ``N=1000`` usually eliminates + intolerable janks due to thrashing. Larger values like ``N=3000`` + make janks less noticeable at the risk of premature OOM kills.
Documentation/admin-guide/mm/multigen_lru.rst line 73
Quoted 10 lines
Documentation/admin-guide/mm/multigen_lru.rst new file +10
-0+ +Data centers +------------ +Data centers want to optimize job scheduling (bin packing) to improve +memory utilizations. Job schedulers need to estimate whether a server +can allocate a certain amount of memory for a new job, and this step +is known as working set estimation, which doesn't impact the existing +jobs running on this server. They also want to attempt freeing some +cold memory from the existing jobs, and this step is known as proactive +reclaim, which improves the chance of landing a new job successfully.
This paragraph also fits overview.
Quoted 3 lines
+
+:Optional: Increase ``CONFIG_NR_LRU_GENS`` to support more generations
+ for working set estimation and proactive reclaim.
Please add a note that this is build time option.
Quoted 2 lines
+
+:Debugfs interface: ``/sys/kernel/debug/lru_gen`` has the following
Is debugfs interface relevant only for datacenters?
Documentation/admin-guide/mm/multigen_lru.rst line 89
Quoted 11 lines
+ format:
+ ::Documentation/admin-guide/mm/multigen_lru.rst new file +9
-0+ + memcg memcg_id memcg_path + node node_id + min_gen birth_time anon_size file_size + ... + max_gen birth_time anon_size file_size + + ``min_gen`` is the oldest generation number and ``max_gen`` is the + youngest generation number. ``birth_time`` is in milliseconds.
It's unclear what is birth_time reference point. Is it milliseconds from the system start or it is measured some other way?
Quoted 3 lines
+ ``anon_size`` and ``file_size`` are in pages. The youngest generation
+ represents the group of the MRU pages and the oldest generation
+ represents the group of the LRU pages. For working set estimation, a
Please spell out MRU and LRU fully.
Quoted 7 lines
+ job scheduler writes to this file at a certain time interval to
+ create new generations, and it ranks available servers based on the
+ sizes of their cold memory defined by this time interval. For
+ proactive reclaim, a job scheduler writes to this file before it
+ tries to land a new job, and if it fails to materialize the cold
+ memory without impacting the existing jobs, it retries on the next
+ server according to the ranking result.
Is this knob only relevant for a job scheduler? Or it can be used in other use-cases as well?
Quoted 2 lines
+
+ This file accepts commands in the following subsections. Multiple
^ described
Quoted 10 lines
+ command lines are supported, so does concatenation with delimiters
+ ``,`` and ``;``.Documentation/admin-guide/mm/multigen_lru.rst new file +8
-0+ + ``/sys/kernel/debug/lru_gen_full`` contains additional stats for + debugging. + +:Working set estimation: Write ``+ memcg_id node_id max_gen + [can_swap [full_scan]]`` to ``/sys/kernel/debug/lru_gen`` to invoke + the aging. It scans PTEs for hot pages and promotes them to the + youngest generation ``max_gen``. Then it creates a new generation
Documentation/admin-guide/mm/multigen_lru.rst line 121
Quoted 10 lines
Documentation/admin-guide/mm/multigen_lru.rst new file +10
-0+ ``max_gen+1``. Set ``can_swap`` to ``1`` to scan for hot anon pages + when swap is off. Set ``full_scan`` to ``0`` to reduce the overhead + as well as the coverage when scanning PTEs. + +:Proactive reclaim: Write ``- memcg_id node_id min_gen [swappiness + [nr_to_reclaim]]`` to ``/sys/kernel/debug/lru_gen`` to invoke the + eviction. It evicts generations less than or equal to ``min_gen``. + ``min_gen`` should be less than ``max_gen-1`` as ``max_gen`` and + ``max_gen-1`` aren't fully aged and therefore can't be evicted. Use + ``nr_to_reclaim`` to limit the number of pages to evict.
I feel that /sys/kernel/debug/lru_gen is too overloaded.
Quoted 12 lines
Documentation/vm/index.rst +1
-0@@ -25,6 +25,7 @@ algorithms. If you are looking for advice on simply allocating memory, see the ksm memory-model mmu_notifier + multigen_lru numa overcommit-accounting page_migration
Signature · 2 lines
Sincerely yours, Mike.
Re: [PATCH v7 12/12] mm: multigenerational LRU: documentation
On Mon, Feb 14, 2022 at 12:28:56PM +0200, Mike Rapoport wrote:
Thanks for reviewing.
Quoted 7 lines
Quoted 4 lines
Documentation/admin-guide/mm/index.rst | 1 + Documentation/admin-guide/mm/multigen_lru.rst | 121 ++++++++++++++ Documentation/vm/index.rst | 1 + Documentation/vm/multigen_lru.rst | 152 ++++++++++++++++++Please consider splitting this patch into
Documentation/admin-guideandDocumentation/vmparts.
Will do.
Quoted 4 lines
Quoted 3 lines
+=====================
+Multigenerational LRU
+=====================+
Quoted 8 lines
Quoted 2 lines
+Quick start
+===========There is no explanation why one would want to use multigenerational LRU until the next section.
I think there should be an overview that explains why users would want to enable multigenerational LRU.
Will do.
Quoted 7 lines
Quoted 3 lines
+Build configurations
+--------------------
+:Required: Set ``CONFIG_LRU_GEN=y``.Maybe
Set ``CONFIG_LRU_GEN=y`` to build kernel with multigenerational LRU
Will do.
Quoted 13 lines
Documentation/admin-guide/mm/multigen_lru.rst line 22
Quoted 10 lines
+:Optional: Set ``CONFIG_LRU_GEN_ENABLED=y`` to enable the
+ multigenerational LRU by default.Documentation/admin-guide/mm/multigen_lru.rst new file +8
-0+ +Runtime configurations +---------------------- +:Required: Write ``y`` to ``/sys/kernel/mm/lru_gen/enable`` if + ``CONFIG_LRU_GEN_ENABLED=n``. + +This file accepts different values to enabled or disabled the +following features:Maybe
Quoted 2 lines
After multigenerational LRU is enabled, this file accepts different values to enable or disable the following feaures:
Will do.
Quoted 6 lines
Quoted 4 lines
+====== ========
+Values Features
+====== ========
+0x0001 the multigenerational LRUThe multigenerational LRU what?
Itself? This depends on the POV, and I'm trying to determine what would be the natural way to present it.
MGLRU itself could be seen as an add-on atop the existing page reclaim or an alternative in parallel. The latter would be similar to sl[aou]b, and that's how I personally see it.
But here I presented it more like the former because I feel this way is more natural to users because they are like switches on a single panel.
Quoted 1 line
What will happen if I write
0x2to this file?
Just like turning on a branch breaker while leaving the main breaker off in a circuit breaker box. This is how I see it, and I'm totally fine with changing it to whatever you'd recommend.
Quoted 1 line
Please consider splitting "enable" and "features" attributes.
How about s/Features/Components/?
Quoted 9 lines
Quoted 2 lines
+0x0002 clear the accessed bit in leaf page table entries **in large
+ batches**, when MMU sets it (e.g., on x86)Is extra markup really needed here...
Quoted 2 lines
+0x0004 clear the accessed bit in non-leaf page table entries **as
+ well**, when MMU sets it (e.g., on x86)... and here?
Will do.
Quoted 3 lines
As for the descriptions, what is the user-visible effect of these features? How different modes of clearing the access bit are reflected in, say, GUI responsiveness, database TPS, or probability of OOM?
These remain to be seen :) I just added these switches in v7, per Mel's request from the meeting we had. These were never tested in the field.
Quoted 36 lines
Quoted 14 lines
+[yYnN] apply to all the features above
+====== ========Documentation/admin-guide/mm/multigen_lru.rst new file +12
-0+ +E.g., +:: + + echo y >/sys/kernel/mm/lru_gen/enabled + cat /sys/kernel/mm/lru_gen/enabled + 0x0007 + echo 5 >/sys/kernel/mm/lru_gen/enabled + cat /sys/kernel/mm/lru_gen/enabled + 0x0005 + +Most users should enable or disable all the features unless some ofDocumentation/admin-guide/mm/multigen_lru.rst line 55
Quoted 10 lines
Documentation/admin-guide/mm/multigen_lru.rst new file +10
-0+them have unforeseen side effects. + +Recipes +======= +Personal computers +------------------ +Personal computers are more sensitive to thrashing because it can +cause janks (lags when rendering UI) and negatively impact user +experience. The multigenerational LRU offers thrashing prevention to +the majority of laptop and desktop users who don't have oomd.I'd expect something like this paragraph in overview.
Quoted 8 lines
+
+:Thrashing prevention: Write ``N`` to
+ ``/sys/kernel/mm/lru_gen/min_ttl_ms`` to prevent the working set of
+ ``N`` milliseconds from getting evicted. The OOM killer is triggered
+ if this working set can't be kept in memory. Based on the average
+ human detectable lag (~100ms), ``N=1000`` usually eliminates
+ intolerable janks due to thrashing. Larger values like ``N=3000``
+ make janks less noticeable at the risk of premature OOM kills.
Quoted 12 lines
Documentation/admin-guide/mm/multigen_lru.rst line 73
Quoted 10 lines
Documentation/admin-guide/mm/multigen_lru.rst new file +10
-0+ +Data centers +------------ +Data centers want to optimize job scheduling (bin packing) to improve +memory utilizations. Job schedulers need to estimate whether a server +can allocate a certain amount of memory for a new job, and this step +is known as working set estimation, which doesn't impact the existing +jobs running on this server. They also want to attempt freeing some +cold memory from the existing jobs, and this step is known as proactive +reclaim, which improves the chance of landing a new job successfully.This paragraph also fits overview.
Will do.
Quoted 4 lines
Quoted 2 lines
+:Optional: Increase ``CONFIG_NR_LRU_GENS`` to support more generations
+ for working set estimation and proactive reclaim.Please add a note that this is build time option.
Will do.
Quoted 3 lines
Quoted 1 line
+:Debugfs interface: ``/sys/kernel/debug/lru_gen`` has the following
Is debugfs interface relevant only for datacenters?
For the moment, yes.
Quoted 6 lines
Quoted 6 lines
+ format:
+ ::Documentation/admin-guide/mm/multigen_lru.rst new file +4
-0+ + memcg memcg_id memcg_path + node node_id + min_gen birth_time anon_size file_size
Quoted 8 lines
Documentation/admin-guide/mm/multigen_lru.rst line 89
Quoted 5 lines
Documentation/admin-guide/mm/multigen_lru.rst new file +5
-0+ ... + max_gen birth_time anon_size file_size + + ``min_gen`` is the oldest generation number and ``max_gen`` is the + youngest generation number. ``birth_time`` is in milliseconds.It's unclear what is
birth_timereference point. Is it milliseconds from the system start or it is measured some other way?
Good point. Will clarify.
Quoted 5 lines
Quoted 3 lines
+ ``anon_size`` and ``file_size`` are in pages. The youngest generation
+ represents the group of the MRU pages and the oldest generation
+ represents the group of the LRU pages. For working set estimation, aPlease spell out MRU and LRU fully.
Will do.
Quoted 10 lines
Quoted 7 lines
+ job scheduler writes to this file at a certain time interval to
+ create new generations, and it ranks available servers based on the
+ sizes of their cold memory defined by this time interval. For
+ proactive reclaim, a job scheduler writes to this file before it
+ tries to land a new job, and if it fails to materialize the cold
+ memory without impacting the existing jobs, it retries on the next
+ server according to the ranking result.Is this knob only relevant for a job scheduler? Or it can be used in other use-cases as well?
There are other concrete use cases but I'm not ready to discuss them yet.
Quoted 3 lines
Quoted 1 line
+ This file accepts commands in the following subsections. Multiple
^ described
Will do.
Mike Rapoport
maintainerRe: [PATCH v7 12/12] mm: multigenerational LRU: documentation
Quoted 12 lines · Yu Zhao
Quoted 6 lines · Mike Rapoport
Quoted 4 lines
+====== ========
+Values Features
+====== ========
+0x0001 the multigenerational LRUThe multigenerational LRU what?
Itself? This depends on the POV, and I'm trying to determine what would be the natural way to present it.
Quoted 6 lines
MGLRU itself could be seen as an add-on atop the existing page reclaim or an alternative in parallel. The latter would be similar to sl[aou]b, and that's how I personally see it.
But here I presented it more like the former because I feel this way is more natural to users because they are like switches on a single panel.
Than I think it should be described as "enable multigenerational LRU" or something like this.
Quoted 5 lines
Quoted 1 line
What will happen if I write
0x2to this file?Just like turning on a branch breaker while leaving the main breaker off in a circuit breaker box. This is how I see it, and I'm totally fine with changing it to whatever you'd recommend.
That was my guess that when bit 0 is clear the rest do not matter :) What's important, IMO, is that it is stated explicitly in the description.
Quoted 3 lines
Quoted 1 line
Please consider splitting "enable" and "features" attributes.
How about s/Features/Components/?
I meant to use two attributes:
/sys/kernel/mm/lru_gen/enable for the main breaker, and /sys/kernel/mm/lru_gen/features (or components) for the branch breakers
Quoted 10 lines
Quoted 9 lines
Quoted 2 lines
+0x0002 clear the accessed bit in leaf page table entries **in large
+ batches**, when MMU sets it (e.g., on x86)Is extra markup really needed here...
Quoted 2 lines
+0x0004 clear the accessed bit in non-leaf page table entries **as
+ well**, when MMU sets it (e.g., on x86)... and here?
Quoted 8 lines
Will do.
Quoted 3 lines
As for the descriptions, what is the user-visible effect of these features? How different modes of clearing the access bit are reflected in, say, GUI responsiveness, database TPS, or probability of OOM?
These remain to be seen :) I just added these switches in v7, per Mel's request from the meeting we had. These were never tested in the field.
I see :)
It would be nice to have a description or/and examples of user-visible effects when there will be some insight on what these features do.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
+:Debugfs interface: ``/sys/kernel/debug/lru_gen`` has the following
Is debugfs interface relevant only for datacenters?
For the moment, yes.
And what will happen if somebody uses these interfaces outside datacenters? As soon as there is a sysfs intefrace, somebody will surely play with it.
I think the job schedulers might be the most important user of that interface, but the documentation should not presume it is the only user.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
+ job scheduler writes to this file at a certain time interval to
+ create new generations, and it ranks available servers based on the
+ sizes of their cold memory defined by this time interval. For
+ proactive reclaim, a job scheduler writes to this file before it
+ tries to land a new job, and if it fails to materialize the cold
+ memory without impacting the existing jobs, it retries on the next
+ server according to the ranking result.
Quoted 5 lines
Quoted 2 lines
Is this knob only relevant for a job scheduler? Or it can be used in other use-cases as well?
There are other concrete use cases but I'm not ready to discuss them yet.
Here as well, as soon as there is an interface it's not necessarily "job scheduler" that will "write to this file", anybody can write to that file. Please adjust the documentation to be more neutral regarding the use-cases.
Signature · 2 lines
Sincerely yours, Mike.
Re: [PATCH v7 12/12] mm: multigenerational LRU: documentation
Quoted 18 lines · Mike Rapoport
Quoted 16 lines · Yu Zhao
Quoted 6 lines · Mike Rapoport
Quoted 4 lines
+====== ========
+Values Features
+====== ========
+0x0001 the multigenerational LRUThe multigenerational LRU what?
Itself? This depends on the POV, and I'm trying to determine what would be the natural way to present it.
MGLRU itself could be seen as an add-on atop the existing page reclaim or an alternative in parallel. The latter would be similar to sl[aou]b, and that's how I personally see it.
Quoted 5 lines
Quoted 2 lines
But here I presented it more like the former because I feel this way is more natural to users because they are like switches on a single panel.
Than I think it should be described as "enable multigenerational LRU" or something like this.
Will do.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
What will happen if I write
0x2to this file?Just like turning on a branch breaker while leaving the main breaker off in a circuit breaker box. This is how I see it, and I'm totally fine with changing it to whatever you'd recommend.
That was my guess that when bit 0 is clear the rest do not matter :) What's important, IMO, is that it is stated explicitly in the description.
Will do.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Please consider splitting "enable" and "features" attributes.
How about s/Features/Components/?
I meant to use two attributes:
/sys/kernel/mm/lru_gen/enablefor the main breaker, and/sys/kernel/mm/lru_gen/features(or components) for the branch breakers
It's a bit superfluous for my taste. I generally consider multiple items to fall into the same category if they can be expressed by a type of array, and I usually pack an array into a single file.
From your last review, I gauged this would be too overloaded for your taste. So I'd be happy to make the change if you think two files look more intuitive from user's perspective.
Quoted 16 lines
Quoted 16 lines
Quoted 9 lines
Quoted 2 lines
+0x0002 clear the accessed bit in leaf page table entries **in large
+ batches**, when MMU sets it (e.g., on x86)Is extra markup really needed here...
Quoted 2 lines
+0x0004 clear the accessed bit in non-leaf page table entries **as
+ well**, when MMU sets it (e.g., on x86)... and here?
Will do.
Quoted 3 lines
As for the descriptions, what is the user-visible effect of these features? How different modes of clearing the access bit are reflected in, say, GUI responsiveness, database TPS, or probability of OOM?
Quoted 7 lines
Quoted 2 lines
These remain to be seen :) I just added these switches in v7, per Mel's request from the meeting we had. These were never tested in the field.
I see :)
It would be nice to have a description or/and examples of user-visible effects when there will be some insight on what these features do.
How does the following sound?
Clearing the accessed bit in large batches can theoretically cause lock contention (mmap_lock), and if it happens the 0x0002 switch can disable this feature. In this case the multigenerational LRU suffers a minor performance degradation.
Clearing the accessed bit in non-leaf page table entries was only verified on Intel and AMD, and if it causes problems on other x86 varieties the 0x0004 switch can disable this feature. In this case the multigenerational LRU suffers a negligible performance degradation.
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
+:Debugfs interface: ``/sys/kernel/debug/lru_gen`` has the following
Is debugfs interface relevant only for datacenters?
Quoted 8 lines
Quoted 1 line
For the moment, yes.
And what will happen if somebody uses these interfaces outside datacenters? As soon as there is a sysfs intefrace, somebody will surely play with it.
I think the job schedulers might be the most important user of that interface, but the documentation should not presume it is the only user.
Other ideas are more like brainstorming than concrete use cases, e.g., for desktop users, these interface can in theory speed up hibernation (suspend to disk); for VM users, they can again in theory support auto ballooning. These niches are really minor and less explored compared with the data center use cases which have been dominant.
I was hoping we could focus on the essential and take one step at a time. Later on, if there is additional demand and resource, then we expand to cover more use cases.
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines
+ job scheduler writes to this file at a certain time interval to
+ create new generations, and it ranks available servers based on the
+ sizes of their cold memory defined by this time interval. For
+ proactive reclaim, a job scheduler writes to this file before it
+ tries to land a new job, and if it fails to materialize the cold
+ memory without impacting the existing jobs, it retries on the next
+ server according to the ranking result.Is this knob only relevant for a job scheduler? Or it can be used in other use-cases as well?
Quoted 6 lines
Quoted 2 lines
There are other concrete use cases but I'm not ready to discuss them yet.
Here as well, as soon as there is an interface it's not necessarily "job scheduler" that will "write to this file", anybody can write to that file. Please adjust the documentation to be more neutral regarding the use-cases.
Will do.
Mike Rapoport
maintainerRe: [PATCH v7 12/12] mm: multigenerational LRU: documentation
Quoted 12 lines · Yu Zhao
Quoted 10 lines · Mike Rapoport
Quoted 3 lines · Yu Zhao
Quoted 1 line
Please consider splitting "enable" and "features" attributes.
How about s/Features/Components/?
I meant to use two attributes:
/sys/kernel/mm/lru_gen/enablefor the main breaker, and/sys/kernel/mm/lru_gen/features(or components) for the branch breakers
Quoted 7 lines
It's a bit superfluous for my taste. I generally consider multiple items to fall into the same category if they can be expressed by a type of array, and I usually pack an array into a single file.
From your last review, I gauged this would be too overloaded for your taste. So I'd be happy to make the change if you think two files look more intuitive from user's perspective.
I do think that two attributes are more user-friendly, but I don't feel strongly about it.
Quoted 14 lines
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines
As for the descriptions, what is the user-visible effect of these features? How different modes of clearing the access bit are reflected in, say, GUI responsiveness, database TPS, or probability of OOM?
These remain to be seen :) I just added these switches in v7, per Mel's request from the meeting we had. These were never tested in the field.
I see :)
It would be nice to have a description or/and examples of user-visible effects when there will be some insight on what these features do.
How does the following sound?
Quoted 8 lines
Clearing the accessed bit in large batches can theoretically cause lock contention (
mmap_lock), and if it happens the0x0002switch can disable this feature. In this case the multigenerational LRU suffers a minor performance degradation.
Clearing the accessed bit in non-leaf page table entries was only verified on Intel and AMD, and if it causes problems on other x86 varieties the0x0004switch can disable this feature. In this case the multigenerational LRU suffers a negligible performance degradation.
LGTM
Quoted 19 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
+:Debugfs interface: ``/sys/kernel/debug/lru_gen`` has the following
Is debugfs interface relevant only for datacenters?
Quoted 8 lines
Quoted 1 line
For the moment, yes.
And what will happen if somebody uses these interfaces outside datacenters? As soon as there is a sysfs intefrace, somebody will surely play with it.
I think the job schedulers might be the most important user of that interface, but the documentation should not presume it is the only user.
Other ideas are more like brainstorming than concrete use cases, e.g., for desktop users, these interface can in theory speed up hibernation (suspend to disk); for VM users, they can again in theory support auto ballooning. These niches are really minor and less explored compared with the data center use cases which have been dominant.
Quoted 3 lines
I was hoping we could focus on the essential and take one step at a time. Later on, if there is additional demand and resource, then we expand to cover more use cases.
Apparently I was not clear :)
I didn't mean that you should describe other use-cases, I rather suggested to make the documentation more neutral, e.g. using "a user writes to this file ..." instead of "job scheduler writes to a file ...". Or maybe add a sentence in the beginning of the "Data centers" section, for instance:
Data centers
------------
+ A representative example of multigenerational LRU users are job schedulers.
Data centers want to optimize job scheduling (bin packing) to improve memory utilizations. Job schedulers need to estimate whether a server
Signature · 2 lines
Sincerely yours, Mike.
Re: [PATCH v7 12/12] mm: multigenerational LRU: documentation
Quoted 80 lines · Mike Rapoport
Quoted 12 lines · Yu Zhao
Quoted 10 lines · Mike Rapoport
Quoted 3 lines · Yu Zhao
Quoted 1 line
Please consider splitting "enable" and "features" attributes.
How about s/Features/Components/?
I meant to use two attributes:
/sys/kernel/mm/lru_gen/enablefor the main breaker, and/sys/kernel/mm/lru_gen/features(or components) for the branch breakersQuoted 7 lines
It's a bit superfluous for my taste. I generally consider multiple items to fall into the same category if they can be expressed by a type of array, and I usually pack an array into a single file.
From your last review, I gauged this would be too overloaded for your taste. So I'd be happy to make the change if you think two files look more intuitive from user's perspective.
I do think that two attributes are more user-friendly, but I don't feel strongly about it.
Quoted 14 lines
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines
As for the descriptions, what is the user-visible effect of these features? How different modes of clearing the access bit are reflected in, say, GUI responsiveness, database TPS, or probability of OOM?
These remain to be seen :) I just added these switches in v7, per Mel's request from the meeting we had. These were never tested in the field.
I see :)
It would be nice to have a description or/and examples of user-visible effects when there will be some insight on what these features do.
How does the following sound?
Quoted 8 lines
Clearing the accessed bit in large batches can theoretically cause lock contention (
mmap_lock), and if it happens the0x0002switch can disable this feature. In this case the multigenerational LRU suffers a minor performance degradation.
Clearing the accessed bit in non-leaf page table entries was only verified on Intel and AMD, and if it causes problems on other x86 varieties the0x0004switch can disable this feature. In this case the multigenerational LRU suffers a negligible performance degradation.LGTM
Quoted 19 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
+:Debugfs interface: ``/sys/kernel/debug/lru_gen`` has the following
Is debugfs interface relevant only for datacenters?
Quoted 8 lines
Quoted 1 line
For the moment, yes.
And what will happen if somebody uses these interfaces outside datacenters? As soon as there is a sysfs intefrace, somebody will surely play with it.
I think the job schedulers might be the most important user of that interface, but the documentation should not presume it is the only user.
Other ideas are more like brainstorming than concrete use cases, e.g., for desktop users, these interface can in theory speed up hibernation (suspend to disk); for VM users, they can again in theory support auto ballooning. These niches are really minor and less explored compared with the data center use cases which have been dominant.
Quoted 3 lines
I was hoping we could focus on the essential and take one step at a time. Later on, if there is additional demand and resource, then we expand to cover more use cases.
Apparently I was not clear :)
I didn't mean that you should describe other use-cases, I rather suggested to make the documentation more neutral, e.g. using "a user writes to this file ..." instead of "job scheduler writes to a file ...". Or maybe add a sentence in the beginning of the "Data centers" section, for instance:
Quoted 8 lines
Data centers
------------+ A representative example of multigenerational LRU users are job schedulers.
Data centers want to optimize job scheduling (bin packing) to improve memory utilizations. Job schedulers need to estimate whether a server
Yes, that makes sense. Will do. Thanks.
Hello.
Quoted 293 lines · On úterý 8. února 2022 9:18:50 CET Yu Zhao wrote:
What's new
==========
1) Addressed all the comments received on the mailing list and in the meeting with the stakeholders (will note on individual patches).
2) Measured the performance improvements for each patch between 5-8 (reported in the commit messages).TLDR
====
The current page reclaim is too expensive in terms of CPU usage and it often makes poor choices about what to evict. This patchset offers an alternative solution that is performant, versatile and straightforward.Patchset overview
=================
The design and implementation overview was moved to patch 12 so that people can finish reading this cover letter.1. mm: x86, arm64: add
arch_has_hw_pte_young()
2. mm: x86: addCONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG
Using hardware optimizations when trying to clear the accessed bit in many PTEs.3.
mm/vmscan.c: refactorshrink_node()
A minor refactor.4. mm: multigenerational LRU: groundwork
Adding the basic data structure and the functions that insert/remove pages to/from the multigenerational LRU (MGLRU) lists.5. mm: multigenerational LRU: minimal implementation
A minimal (functional) implementation without any optimizations.6. mm: multigenerational LRU: exploit locality in rmap Improving the efficiency when using the rmap.
7. mm: multigenerational LRU: support page table walks Adding the (optional) page table scanning.
8. mm: multigenerational LRU: optimize multiple memcgs Optimizing the overall performance for multiple memcgs running mixed types of workloads.
9. mm: multigenerational LRU: runtime switch Adding a runtime switch to enable or disable MGLRU.
10. mm: multigenerational LRU: thrashing prevention
11. mm: multigenerational LRU: debugfs interface
Providing userspace with additional features like thrashing prevention, working set estimation and proactive reclaim.12. mm: multigenerational LRU: documentation
Adding a design doc and an admin guide.Benchmark results
=================
Independent lab results
-----------------------
Based on the popularity of searches [01] and the memory usage in Google's public cloud, the most popular open-source memory-hungry applications, in alphabetical order, are:Apache Cassandra Memcached Apache Hadoop MongoDB Apache Spark PostgreSQL MariaDB (MySQL) RedisAn independent lab evaluated MGLRU with the most widely used benchmark suites for the above applications. They posted 960 data points along with kernel metrics and perf profiles collected over more than 500 hours of total benchmark time. Their final reports show that, with 95% confidence intervals (CIs), the above applications all performed significantly better for at least part of their benchmark matrices.
On 5.14:
1. Apache Spark [02] took 95% CIs [9.28, 11.19]% and [12.20, 14.93]% less wall time to sort three billion random integers, respectively, under the medium- and the high-concurrency conditions, when overcommitting memory. There were no statistically significant changes in wall time for the rest of the benchmark matrix.
2. MariaDB [03] achieved 95% CIs [5.24, 10.71]% and [20.22, 25.97]% more transactions per minute (TPM), respectively, under the medium- and the high-concurrency conditions, when overcommitting memory. There were no statistically significant changes in TPM for the rest of the benchmark matrix.
3. Memcached [04] achieved 95% CIs [23.54, 32.25]%, [20.76, 41.61]% and [21.59, 30.02]% more operations per second (OPS), respectively, for sequential access, random access and Gaussian (distribution) access, when THP=always; 95% CIs [13.85, 15.97]% and [23.94, 29.92]% more OPS, respectively, for random access and Gaussian access, when THP=never. There were no statistically significant changes in OPS for the rest of the benchmark matrix.
4. MongoDB [05] achieved 95% CIs [2.23, 3.44]%, [6.97, 9.73]% and [2.16, 3.55]% more operations per second (OPS), respectively, for exponential (distribution) access, random access and Zipfian (distribution) access, when underutilizing memory; 95% CIs [8.83, 10.03]%, [21.12, 23.14]% and [5.53, 6.46]% more OPS, respectively, for exponential access, random access and Zipfian access, when overcommitting memory.On 5.15:
5. Apache Cassandra [06] achieved 95% CIs [1.06, 4.10]%, [1.94, 5.43]% and [4.11, 7.50]% more operations per second (OPS), respectively, for exponential (distribution) access, random access and Zipfian (distribution) access, when swap was off; 95% CIs [0.50, 2.60]%, [6.51, 8.77]% and [3.29, 6.75]% more OPS, respectively, for exponential access, random access and Zipfian access, when swap was on.
6. Apache Hadoop [07] took 95% CIs [5.31, 9.69]% and [2.02, 7.86]% less average wall time to finish twelve parallel TeraSort jobs, respectively, under the medium- and the high-concurrency conditions, when swap was on. There were no statistically significant changes in average wall time for the rest of the benchmark matrix.
7. PostgreSQL [08] achieved 95% CI [1.75, 6.42]% more transactions per minute (TPM) under the high-concurrency condition, when swap was off; 95% CIs [12.82, 18.69]% and [22.70, 46.86]% more TPM, respectively, under the medium- and the high-concurrency conditions, when swap was on. There were no statistically significant changes in TPM for the rest of the benchmark matrix.
8. Redis [09] achieved 95% CIs [0.58, 5.94]%, [6.55, 14.58]% and [11.47, 19.36]% more total operations per second (OPS), respectively, for sequential access, random access and Gaussian (distribution) access, when THP=always; 95% CIs [1.27, 3.54]%, [10.11, 14.81]% and [8.75, 13.64]% more total OPS, respectively, for sequential access, random access and Gaussian access, when THP=never.Our lab results
---------------
To supplement the above results, we ran the following benchmark suites on 5.16-rc7 and found no regressions [10]. (These synthetic benchmarks are popular among MM developers, but we prefer large-scale A/B experiments to validate improvements.)fs_fio_bench_hdd_mq pft fs_lmbench pgsql-hammerdb fs_parallelio redis fs_postmark stream hackbench sysbenchthread kernbench tpcc_spark memcached unixbench multichase vm-scalability mutilate will-it-scalenginx
[01] https://trends.google.com
[02] https://lore.kernel.org/lkml/[email protected]/
[03] https://lore.kernel.org/lkml/[email protected]/
[04] https://lore.kernel.org/lkml/[email protected]/
[05] https://lore.kernel.org/lkml/[email protected]/
[06] https://lore.kernel.org/lkml/[email protected]/
[07] https://lore.kernel.org/lkml/[email protected]/
[08] https://lore.kernel.org/lkml/[email protected]/
[09] https://lore.kernel.org/lkml/[email protected]/
[10] https://lore.kernel.org/lkml/[email protected]/Read-world applications
=======================
Third-party testimonials
------------------------
Konstantin wrote [11]:
I have Archlinux with 8G RAM + zswap + swap. While developing, I have lots of apps opened such as multiple LSP-servers for different langs, chats, two browsers, etc... Usually, my system gets quickly to a point of SWAP-storms, where I have to kill LSP-servers, restart browsers to free memory, etc, otherwise the system lags heavily and is barely usable.1.5 day ago I migrated from 5.11.15 kernel to 5.12 + the LRU patchset, and I started up by opening lots of apps to create memory pressure, and worked for a day like this. Till now I had *not a single SWAP-storm*, and mind you I got 3.4G in SWAP. I was never getting to the point of 3G in SWAP before without a single SWAP-storm.
An anonymous user wrote [12]:
Using that v5 for some time and confirm that difference under heavy load and memory pressure is significant.Shuang wrote [13]:
With the MGLRU, fio achieved 95% CIs [38.95, 40.26]%, [4.12, 6.64]% and [9.26, 10.36]% higher throughput, respectively, for random access, Zipfian (distribution) access and Gaussian (distribution) access, when the average number of jobs per CPU is 1; 95% CIs [42.32, 49.15]%, [9.44, 9.89]% and [20.99, 22.86]% higher throughput, respectively, for random access, Zipfian access and Gaussian access, when the average number of jobs per CPU is 2.Daniel wrote [14]:
With memcached allocating ~100GB of byte-addressable Optante, performance improvement in terms of throughput (measured as queries per second) was about 10% for a series of workloads.Large-scale deployments
-----------------------
The downstream kernels that have been using MGLRU include:
1. Android ARCVM [15]
2. Arch Linux Zen [16]
3. Chrome OS [17]
4. Liquorix [18]
5. post-factum [19]
6. XanMod [20]We've rolled out MGLRU to tens of millions of Chrome OS users and about a million Android users. Google's fleetwide profiling [21] shows an overall 40% decrease in kswapd CPU usage, in addition to improvements in other UX metrics, e.g., an 85% decrease in the number of low-memory kills at the 75th percentile and an 18% decrease in rendering latency at the 50th percentile.
[11] https://lore.kernel.org/lkml/[email protected]/
[12] https://phoronix.com/forums/forum/software/general-linux-open-source/1301258-mglru-is-a-very-enticing-enhancement-for-linux-in-2022?p=1301275#post1301275
[13] https://lore.kernel.org/lkml/[email protected]/
[14] https://lore.kernel.org/linux-mm/CA+4-3vksGvKd18FgRinxhqHetBS1hQekJE2gwco8Ja-bJWKtFw@mail.gmail.com/
[15] https://chromium.googlesource.com/chromiumos/third_party/kernel
[16] https://archlinux.org
[17] https://chromium.org
[18] https://liquorix.net
[19] https://gitlab.com/post-factum/pf-kernel
[20] https://xanmod.org
[21] https://research.google/pubs/pub44271/Summery
=======
The facts are:
1. The independent lab results and the real-world applications indicate substantial improvements; there are no known regressions.
2. Thrashing prevention, working set estimation and proactive reclaim work out of the box; there are no equivalent solutions.
3. There is a lot of new code; nobody has demonstrated smaller changes with similar effects.Our options, accordingly, are:
1. Given the amount of evidence, the reported improvements will likely materialize for a wide range of workloads.
2. Gauging the interest from the past discussions [22][23][24], the new features will likely be put to use for both personal computers and data centers.
3. Based on Google's track record, the new code will likely be well maintained in the long term. It'd be more difficult if not impossible to achieve similar effects on top of the existing design.[22] https://lore.kernel.org/lkml/[email protected]/
[23] https://lore.kernel.org/lkml/[email protected]/
[24] https://lore.kernel.org/lkml/[email protected]/Yu Zhao (12):
mm: x86, arm64: addarch_has_hw_pte_young()
mm: x86: addCONFIG_ARCH_HAS_NONLEAF_PMD_YOUNGmm/vmscan.c: refactorshrink_node()
mm: multigenerational LRU: groundwork
mm: multigenerational LRU: minimal implementation
mm: multigenerational LRU: exploit locality in rmap
mm: multigenerational LRU: support page table walks
mm: multigenerational LRU: optimize multiple memcgs
mm: multigenerational LRU: runtime switch
mm: multigenerational LRU: thrashing prevention
mm: multigenerational LRU: debugfs interface
mm: multigenerational LRU: documentationDocumentation/admin-guide/mm/index.rst | 1 + Documentation/admin-guide/mm/multigen_lru.rst | 121 + Documentation/vm/index.rst | 1 + Documentation/vm/multigen_lru.rst | 152 + arch/Kconfig | 9 + arch/arm64/include/asm/pgtable.h | 14 +- arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 9 +- arch/x86/mm/pgtable.c | 5 +- fs/exec.c | 2 + fs/fuse/dev.c | 3 +- include/linux/cgroup.h | 15 +- include/linux/memcontrol.h | 36 + include/linux/mm.h | 8 + include/linux/mm_inline.h | 214 ++ include/linux/mm_types.h | 78 + include/linux/mmzone.h | 182 ++ include/linux/nodemask.h | 1 + include/linux/page-flags-layout.h | 19 +- include/linux/page-flags.h | 4 +- include/linux/pgtable.h | 17 +- include/linux/sched.h | 4 + include/linux/swap.h | 5 + kernel/bounds.c | 3 + kernel/cgroup/cgroup-internal.h | 1 - kernel/exit.c | 1 + kernel/fork.c | 9 + kernel/sched/core.c | 1 + mm/Kconfig | 50 + mm/huge_memory.c | 3 +- mm/memcontrol.c | 27 + mm/memory.c | 39 +- mm/mm_init.c | 6 +-
Quoted 8 lines
mm/page_alloc.c | 1 + mm/rmap.c | 7 + mm/swap.c | 55 +- mm/vmscan.c | 2831 ++++++++++++++++- mm/workingset.c | 119 +- 38 files changed, 3908 insertions(+), 146 deletions(-) create mode 100644 Documentation/admin-guide/mm/multigen_lru.rst create mode 100644 Documentation/vm/multigen_lru.rst
Thanks for the new spin.
Is the patch submission broken for everyone, or for me only? I see raw emails cluttered with some garbage like =2D, and hence I cannot apply those neither from my email client nor from lore.
Probably, you've got a git repo where things can be pulled from so that we do not depend on mailing systems and/or tools breaking plaintext?
Thanks.
Signature · 1 line
Oleksandr Natalenko (post-factum)
Michal Hocko
reviewerOn Tue 08-02-22 11:11:02, Oleksandr Natalenko wrote:
[...]
Quoted 3 lines
Is the patch submission broken for everyone, or for me only? I see raw emails cluttered with some garbage like =2D, and hence I cannot apply those neither from my email client nor from lore.
The patchset seems to be OK both in my inbox and b4[1] has downloaded the full thread without any issues and I could apply all the patches just fine
[1] https://git.kernel.org/pub/scm/utils/b4/b4.git
Signature · 2 lines
Michal Hocko SUSE Labs
Hello.
Quoted 11 lines · On úterý 8. února 2022 12:14:00 CET Michal Hocko wrote:
On Tue 08-02-22 11:11:02, Oleksandr Natalenko wrote:
[...]Quoted 3 lines
Is the patch submission broken for everyone, or for me only? I see raw emails cluttered with some garbage like =2D, and hence I cannot apply those neither from my email client nor from lore.
The patchset seems to be OK both in my inbox and b4[1] has downloaded the full thread without any issues and I could apply all the patches just fine
Thanks, b4 worked for me as well.
Signature · 1 line
Oleksandr Natalenko (post-factum)
Aggressive swapping even with vm.swappiness=1 with MGLRU
========================================================
Reading a large mmapped file leads to a super agressive swapping. Reducing vm.swappiness even to 1 does not have effect.
Demo: https://www.youtube.com/watch?v=J81kwJeuW58
Linux 5.17-rc3, Multigenerational LRU v7,
vm.swappiness=1, MemTotal: 11.5 GiB.
$ cache-bench -r 35000 -m1 -b1 -p1 -f test20000 Reading mmapped file (file size: 20000 MiB) cache-bench v0.2.0: https://github.com/hakavlad/cache-bench
Swapping started with MemAvailable=71%.
At the end 33 GiB was swapped out when MemAvailable=60%.
Is it OK?
Quoted 5 lines · Alexey Avramov
Aggressive swapping even with vm.swappiness=1 with MGLRU
========================================================Reading a large mmapped file leads to a super agressive swapping. Reducing vm.swappiness even to 1 does not have effect.
Mind explaining why you think it's "super agressive"? I assume you expected a different behavior that would perform better. If so, please spell it out.
Quoted 8 lines
Demo: https://www.youtube.com/watch?v=J81kwJeuW58
Linux 5.17-rc3, Multigenerational LRU v7,
vm.swappiness=1, MemTotal: 11.5 GiB.$ cache-bench -r 35000 -m1 -b1 -p1 -f test20000 Reading mmapped file (file size: 20000 MiB) cache-bench v0.2.0: https://github.com/hakavlad/cache-bench
Writing your own benchmark is a good exercise but fio is the standard benchmark in this case. Please use it with --ioengine=mmap.
Quoted 4 lines
Swapping started with MemAvailable=71%.
At the end 33 GiB was swapped out when MemAvailable=60%.Is it OK?
MemAvailable is an estimate (free + page cache), and it doesn't imply any reclaim preferences. In the worst case scenario, e.g., out of swap space, MemAvailable *may* be reclaimed.
Here is my benchmark result with file mmap + *high* swap usage. Ram disk was used to reduce the variance in the result (and SSD wear out if you care). More details on additional configurations here:
https://lore.kernel.org/linux-mm/[email protected]/
Mixed workloads:
fio (buffered I/O): +13%
IOPS BW
5.17-rc3: 275k 1075MiB/s
v7: 313k 1222MiB/s
memcached (anon): +12%
Ops/sec KB/sec
5.17-rc3: 511282.72 19861.04
v7: 572408.80 22235.49 cat mmap.sh
systemctl restart memcached
swapoff -a
umount /mnt
rmmod brd
modprobe brd rd_nr=2 rd_size=56623104
mkswap /dev/ram0
swapon /dev/ram0
mkfs.ext4 /dev/ram1
mount -t ext4 /dev/ram1 /mnt
memtier_benchmark -S /var/run/memcached/memcached.sock \ -P memcache_binary -n allkeys --key-minimum=1 \ --key-maximum=50000000 --key-pattern=P:P -c 1 \ -t 36 --ratio 1:0 --pipeline 8 -d 2000
sysctl vm.overcommit_memory=1
fio -name=mglru --numjobs=36 --directory=/mnt --size=1408m \ --buffered=1 --ioengine=mmap --iodepth=128 --iodepth_batch_submit=32 \ --iodepth_batch_complete=32 --rw=randread --random_distribution=random \ --norandommap --time_based --ramp_time=10m --runtime=990m \ --group_reporting & pid=$!
sleep 200
memcached.sock -P memcache_binary -n allkeys --key-minimum=1 \ --key-maximum=50000000 --key-pattern=R:R -c 1 -t 36 --ratio 0:1 \ --pipeline 8 --randomize --distinct-client-seed
kill -INT $pid
wait
In a synthetic MongoDB Benchmark (YCSB) seeing an average of ~19% throughput improvement on POWER10(Radix MMU + 64K Page Size) with MGLRU patches on top of v5.16 kernel for MongoDB + YCSB bench across three different request distriburions namely Exponential,Uniform and Zipfan
Test-Results
============
Average YCSB reported throughput (95% Confidence Interval):
|---------------------+---------------------+---------------------+---------------------| | Kernel-Type | Exponential | Uniform | Zipfan | |---------------------+---------------------+---------------------+---------------------| | Base Kernel (v5.16) | 27324.701 ± 759.652 | 20671.590 ± 412.974 | 37713.761 ± 621.213 | | v5.16 + MGLRU | 32702.231 ± 287.957 | 24916.239 ± 217.977 | 44308.839 ± 701.829 | |---------------------+---------------------+---------------------+---------------------| | Speedup | 19.68% ± 4.03% | 20.11% ± 2.95% | 17.49% ± 2.82% | |---------------------+---------------------+---------------------+---------------------|
n = 11 Samples x 3 (Distributions) x 2 (Kernels) = 66 Observations
Test Environment
================
Cpu: POWER10 (architected), altivec supported
platform: pSeries
CPUs: 32
MMU: Radix
Page-Size: 64K
Total-Memory: 64G
Distro
-------
# cat /etc/os-release NAME="Red Hat Enterprise Linux" VERSION="8.4 (Ootpa)" ID="rhel" ID_LIKE="fedora" VERSION_ID="8.4" PLATFORM_ID="platform:el8" PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" HOME_URL="https://www.redhat.com/" DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
System-config
-------------
# cat /sys/kernel/mm/transparent_hugepage/enabled always madvise [never]
# cat /proc/swaps Filename Type Size Used Priority /dev/dm-5 partition 10485696 940864 -2
# cat /proc/sys/vm/overcommit_memory 0
#cat /proc/cmdline
<existing parameters> systemd.unified_cgroup_hierarchy=1 transparent_hugepage=never
MongoDB data partition
----------------------
lsblk /dev/sdb
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 0 128G 0 disk <home>/data/mongodb
mount | grep /dev/sdb/dev/sdb on /root/vajain21/mglru/data/mongodb type ext4 (rw,relatime)
Testing Artifacts
==================
MongoDB-configuration
---------------------
MongoDB Commounity Server built from https://github.com/mongodb/mongo release v5.0.6
# mongod --version
db version v5.0.6
Build Info: {
"version": "5.0.6",
"gitVersion": "212a8dbb47f07427dae194a9c75baec1d81d9259",
"openSSLVersion": "OpenSSL 1.1.1g FIPS 21 Apr 2020",
"modules": [],
"allocator": "tcmalloc",
"environment": {
"distarch": "ppc64le",
"target_arch": "ppc64le"
}
}# cat /etc/mongod.conf
storage:
dbPath: <home-path>/data/mongodb
journal:
enabled: true
engine: wiredTiger
wiredTiger:
engineConfig:
cacheSizeGB: 50
net:
bindIp: 127.0.0.1
unixDomainSocket:
enabled: true
pathPrefix: /run/mongodb
setParameter:
enableLocalhostAuthBypass: trueYCSB (https://github.com/vaibhav92/YCSB/tree/mongodb-domain-sockets)
--------------------------------------------------------------------
YCSB forked from https://github.com/brianfrankcooper/YCSB.git. This fixes a problem with YCSB when trying to connect to MongoDB on a unix domain socket. PR raised to the project at https://github.com/brianfrankcooper/YCSB/pull/1587
Head Commit: fb2555a77005ae70c26e4adc46c945caf4daa2f9(" [core] Generate classpath from all dependencies rather than just compile scoped")
Kernel-Config
-------------
Base-Kernel: https://github.com/torvalds/linux/ v5.16 Base-Kernel-Config:
https://github.com/vaibhav92/mglru-benchmark/blob/auto_build/config-non-mglru
Test-Kernel: https://linux-mm.googlesource.com/page-reclaim refs/changes/49/1549/1 Test-Kernel-Config:
https://github.com/vaibhav92/mglru-benchmark/blob/auto_build/config-mglru
CONFIG_LRU_GEN=yCONFIG_LRU_GEN_ENABLED=yCONFIG_NR_LRU_GENS=4CONFIG_TIERS_PER_GEN=4
YCSB:
recordcount=80000000
operationcount=80000000
readproportion=0.8
updateproportion=0.2 workload=site.ycsb.workloads.CoreWorkload threads=64
requestdistributions={uniform, exponential, zipfian}
Test-Bench
===========
Source: https://github.com/vaibhav92/mglru-benchmark/tree/auto_build
Invoked via following command that will *destroy* contents of /dev/sdd and use it as data disk for MongoDB:
$ export MONGODB_DISK=/dev/sdd; curl \ https://raw.githubusercontent.com/vaibhav92/mglru-benchmark/auto_build/build.sh \ | sudo bash -s
Test-Methodology
================
Setup
-----
1. Pull & Build testing artifact v5.16 Base Kernel, MGLRU Kernel,
MongoDB, YCSB & Qemu for qemu-img tools
2. Format and mount provided MongoDB Data disk with ext4.
3. Generate Systemd service/slice files for MongoDB and place them into /etc/systemd/system/
4. Generate MongoDB configration pointing to the data disk mount.
5. Start the built MongoDB instance.
6. Ensure that MongoDB is running.
Load Test Data
---------------
1. Ensure that MongoDB instance is stopped.
2. Unmount the data disk and reformat it with ext4.
3. Restart MongoDB.
4. Spin off YCSB to load data into the Mongo instance.
5. Stop MongoDB + Unmount data Disk
6. Create a qcow2 image of the data disk and store it with test data.
7. Kexec into base kernel.
Test Phase (Happens at each boot)
---------------------------------
1. Select the distribution to be used for YCSB from {"Uniform","Exponential","Zipfan"}
2. Restore the MongoDB qcow2 data disk Image to the disk
3. Mount the data disk and restart MongoDB daemon.
4. Start YCSB to generate the workload on MongoDB.
5. Once finished collect results.
6. Kexec into next-kernel which keeps switching between Base-Kernel & MGLRU-Kernel when all three distriutions have been tested.
Setup and Load Test Data stages can be accomplished by following command:
#export MONGODB_DISK=/dev/sdd; \
curl https://raw.githubusercontent.com/vaibhav92/mglru-benchmark/auto_build/build.sh | bash -s
Once completed successfully it will kexec into the base kernel and start the Test phase on boot via systemd service named 'mglru-benchmark'
Based on above results,
Quoted 306 lines · Yu Zhao
What's new
==========
1) Addressed all the comments received on the mailing list and in the meeting with the stakeholders (will note on individual patches).
2) Measured the performance improvements for each patch between 5-8 (reported in the commit messages).TLDR
====
The current page reclaim is too expensive in terms of CPU usage and it often makes poor choices about what to evict. This patchset offers an alternative solution that is performant, versatile and straightforward.Patchset overview
=================
The design and implementation overview was moved to patch 12 so that people can finish reading this cover letter.1. mm: x86, arm64: add
arch_has_hw_pte_young()
2. mm: x86: addCONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG
Using hardware optimizations when trying to clear the accessed bit in many PTEs.3.
mm/vmscan.c: refactorshrink_node()
A minor refactor.4. mm: multigenerational LRU: groundwork
Adding the basic data structure and the functions that insert/remove pages to/from the multigenerational LRU (MGLRU) lists.5. mm: multigenerational LRU: minimal implementation
A minimal (functional) implementation without any optimizations.6. mm: multigenerational LRU: exploit locality in rmap Improving the efficiency when using the rmap.
7. mm: multigenerational LRU: support page table walks Adding the (optional) page table scanning.
8. mm: multigenerational LRU: optimize multiple memcgs Optimizing the overall performance for multiple memcgs running mixed types of workloads.
9. mm: multigenerational LRU: runtime switch Adding a runtime switch to enable or disable MGLRU.
10. mm: multigenerational LRU: thrashing prevention
11. mm: multigenerational LRU: debugfs interface
Providing userspace with additional features like thrashing prevention, working set estimation and proactive reclaim.12. mm: multigenerational LRU: documentation
Adding a design doc and an admin guide.Benchmark results
=================
Independent lab results
-----------------------
Based on the popularity of searches [01] and the memory usage in Google's public cloud, the most popular open-source memory-hungry applications, in alphabetical order, are:Apache Cassandra Memcached Apache Hadoop MongoDB Apache Spark PostgreSQL MariaDB (MySQL) RedisAn independent lab evaluated MGLRU with the most widely used benchmark suites for the above applications. They posted 960 data points along with kernel metrics and perf profiles collected over more than 500 hours of total benchmark time. Their final reports show that, with 95% confidence intervals (CIs), the above applications all performed significantly better for at least part of their benchmark matrices.
On 5.14:
1. Apache Spark [02] took 95% CIs [9.28, 11.19]% and [12.20, 14.93]% less wall time to sort three billion random integers, respectively, under the medium- and the high-concurrency conditions, when overcommitting memory. There were no statistically significant changes in wall time for the rest of the benchmark matrix.
2. MariaDB [03] achieved 95% CIs [5.24, 10.71]% and [20.22, 25.97]% more transactions per minute (TPM), respectively, under the medium- and the high-concurrency conditions, when overcommitting memory. There were no statistically significant changes in TPM for the rest of the benchmark matrix.
3. Memcached [04] achieved 95% CIs [23.54, 32.25]%, [20.76, 41.61]% and [21.59, 30.02]% more operations per second (OPS), respectively, for sequential access, random access and Gaussian (distribution) access, when THP=always; 95% CIs [13.85, 15.97]% and [23.94, 29.92]% more OPS, respectively, for random access and Gaussian access, when THP=never. There were no statistically significant changes in OPS for the rest of the benchmark matrix.
4. MongoDB [05] achieved 95% CIs [2.23, 3.44]%, [6.97, 9.73]% and [2.16, 3.55]% more operations per second (OPS), respectively, for exponential (distribution) access, random access and Zipfian (distribution) access, when underutilizing memory; 95% CIs [8.83, 10.03]%, [21.12, 23.14]% and [5.53, 6.46]% more OPS, respectively, for exponential access, random access and Zipfian access, when overcommitting memory.On 5.15:
5. Apache Cassandra [06] achieved 95% CIs [1.06, 4.10]%, [1.94, 5.43]% and [4.11, 7.50]% more operations per second (OPS), respectively, for exponential (distribution) access, random access and Zipfian (distribution) access, when swap was off; 95% CIs [0.50, 2.60]%, [6.51, 8.77]% and [3.29, 6.75]% more OPS, respectively, for exponential access, random access and Zipfian access, when swap was on.
6. Apache Hadoop [07] took 95% CIs [5.31, 9.69]% and [2.02, 7.86]% less average wall time to finish twelve parallel TeraSort jobs, respectively, under the medium- and the high-concurrency conditions, when swap was on. There were no statistically significant changes in average wall time for the rest of the benchmark matrix.
7. PostgreSQL [08] achieved 95% CI [1.75, 6.42]% more transactions per minute (TPM) under the high-concurrency condition, when swap was off; 95% CIs [12.82, 18.69]% and [22.70, 46.86]% more TPM, respectively, under the medium- and the high-concurrency conditions, when swap was on. There were no statistically significant changes in TPM for the rest of the benchmark matrix.
8. Redis [09] achieved 95% CIs [0.58, 5.94]%, [6.55, 14.58]% and [11.47, 19.36]% more total operations per second (OPS), respectively, for sequential access, random access and Gaussian (distribution) access, when THP=always; 95% CIs [1.27, 3.54]%, [10.11, 14.81]% and [8.75, 13.64]% more total OPS, respectively, for sequential access, random access and Gaussian access, when THP=never.Our lab results
---------------
To supplement the above results, we ran the following benchmark suites on 5.16-rc7 and found no regressions [10]. (These synthetic benchmarks are popular among MM developers, but we prefer large-scale A/B experiments to validate improvements.)fs_fio_bench_hdd_mq pft fs_lmbench pgsql-hammerdb fs_parallelio redis fs_postmark stream hackbench sysbenchthread kernbench tpcc_spark memcached unixbench multichase vm-scalability mutilate will-it-scalenginx
[01] https://trends.google.com
[02] https://lore.kernel.org/lkml/[email protected]/
[03] https://lore.kernel.org/lkml/[email protected]/
[04] https://lore.kernel.org/lkml/[email protected]/
[05] https://lore.kernel.org/lkml/[email protected]/
[06] https://lore.kernel.org/lkml/[email protected]/
[07] https://lore.kernel.org/lkml/[email protected]/
[08] https://lore.kernel.org/lkml/[email protected]/
[09] https://lore.kernel.org/lkml/[email protected]/
[10] https://lore.kernel.org/lkml/[email protected]/Read-world applications
=======================
Third-party testimonials
------------------------
Konstantin wrote [11]:
I have Archlinux with 8G RAM + zswap + swap. While developing, I have lots of apps opened such as multiple LSP-servers for different langs, chats, two browsers, etc... Usually, my system gets quickly to a point of SWAP-storms, where I have to kill LSP-servers, restart browsers to free memory, etc, otherwise the system lags heavily and is barely usable.1.5 day ago I migrated from 5.11.15 kernel to 5.12 + the LRU patchset, and I started up by opening lots of apps to create memory pressure, and worked for a day like this. Till now I had *not a single SWAP-storm*, and mind you I got 3.4G in SWAP. I was never getting to the point of 3G in SWAP before without a single SWAP-storm.
An anonymous user wrote [12]:
Using that v5 for some time and confirm that difference under heavy load and memory pressure is significant.Shuang wrote [13]:
With the MGLRU, fio achieved 95% CIs [38.95, 40.26]%, [4.12, 6.64]% and [9.26, 10.36]% higher throughput, respectively, for random access, Zipfian (distribution) access and Gaussian (distribution) access, when the average number of jobs per CPU is 1; 95% CIs [42.32, 49.15]%, [9.44, 9.89]% and [20.99, 22.86]% higher throughput, respectively, for random access, Zipfian access and Gaussian access, when the average number of jobs per CPU is 2.Daniel wrote [14]:
With memcached allocating ~100GB of byte-addressable Optante, performance improvement in terms of throughput (measured as queries per second) was about 10% for a series of workloads.Large-scale deployments
-----------------------
The downstream kernels that have been using MGLRU include:
1. Android ARCVM [15]
2. Arch Linux Zen [16]
3. Chrome OS [17]
4. Liquorix [18]
5. post-factum [19]
6. XanMod [20]We've rolled out MGLRU to tens of millions of Chrome OS users and about a million Android users. Google's fleetwide profiling [21] shows an overall 40% decrease in kswapd CPU usage, in addition to improvements in other UX metrics, e.g., an 85% decrease in the number of low-memory kills at the 75th percentile and an 18% decrease in rendering latency at the 50th percentile.
[11] https://lore.kernel.org/lkml/[email protected]/
[12] https://phoronix.com/forums/forum/software/general-linux-open-source/1301258-mglru-is-a-very-enticing-enhancement-for-linux-in-2022?p=1301275#post1301275
[13] https://lore.kernel.org/lkml/[email protected]/
[14] https://lore.kernel.org/linux-mm/CA+4-3vksGvKd18FgRinxhqHetBS1hQekJE2gwco8Ja-bJWKtFw@mail.gmail.com/
[15] https://chromium.googlesource.com/chromiumos/third_party/kernel
[16] https://archlinux.org
[17] https://chromium.org
[18] https://liquorix.net
[19] https://gitlab.com/post-factum/pf-kernel
[20] https://xanmod.org
[21] https://research.google/pubs/pub44271/Summery
=======
The facts are:
1. The independent lab results and the real-world applications indicate substantial improvements; there are no known regressions.
2. Thrashing prevention, working set estimation and proactive reclaim work out of the box; there are no equivalent solutions.
3. There is a lot of new code; nobody has demonstrated smaller changes with similar effects.Our options, accordingly, are:
1. Given the amount of evidence, the reported improvements will likely materialize for a wide range of workloads.
2. Gauging the interest from the past discussions [22][23][24], the new features will likely be put to use for both personal computers and data centers.
3. Based on Google's track record, the new code will likely be well maintained in the long term. It'd be more difficult if not impossible to achieve similar effects on top of the existing design.[22] https://lore.kernel.org/lkml/[email protected]/
[23] https://lore.kernel.org/lkml/[email protected]/
[24] https://lore.kernel.org/lkml/[email protected]/Yu Zhao (12):
mm: x86, arm64: addarch_has_hw_pte_young()
mm: x86: addCONFIG_ARCH_HAS_NONLEAF_PMD_YOUNGmm/vmscan.c: refactorshrink_node()
mm: multigenerational LRU: groundwork
mm: multigenerational LRU: minimal implementation
mm: multigenerational LRU: exploit locality in rmap
mm: multigenerational LRU: support page table walks
mm: multigenerational LRU: optimize multiple memcgs
mm: multigenerational LRU: runtime switch
mm: multigenerational LRU: thrashing prevention
mm: multigenerational LRU: debugfs interface
mm: multigenerational LRU: documentationDocumentation/admin-guide/mm/index.rst | 1 + Documentation/admin-guide/mm/multigen_lru.rst | 121 + Documentation/vm/index.rst | 1 + Documentation/vm/multigen_lru.rst | 152 + arch/Kconfig | 9 + arch/arm64/include/asm/pgtable.h | 14 +- arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 9 +- arch/x86/mm/pgtable.c | 5 +- fs/exec.c | 2 + fs/fuse/dev.c | 3 +- include/linux/cgroup.h | 15 +- include/linux/memcontrol.h | 36 + include/linux/mm.h | 8 + include/linux/mm_inline.h | 214 ++ include/linux/mm_types.h | 78 + include/linux/mmzone.h | 182 ++ include/linux/nodemask.h | 1 + include/linux/page-flags-layout.h | 19 +- include/linux/page-flags.h | 4 +- include/linux/pgtable.h | 17 +- include/linux/sched.h | 4 + include/linux/swap.h | 5 + kernel/bounds.c | 3 + kernel/cgroup/cgroup-internal.h | 1 - kernel/exit.c | 1 + kernel/fork.c | 9 + kernel/sched/core.c | 1 + mm/Kconfig | 50 + mm/huge_memory.c | 3 +- mm/memcontrol.c | 27 + mm/memory.c | 39 +- mm/mm_init.c | 6 +- mm/page_alloc.c | 1 + mm/rmap.c | 7 + mm/swap.c | 55 +- mm/vmscan.c | 2831 ++++++++++++++++- mm/workingset.c | 119 +- 38 files changed, 3908 insertions(+), 146 deletions(-) create mode 100644 Documentation/admin-guide/mm/multigen_lru.rst create mode 100644 Documentation/vm/multigen_lru.rstSignature · 1 line
2.35.0.263.gb82422642f-goog
Signature · 2 lines
Cheers ~ Vaibhav
Quoted 5 lines · Vaibhav Jain
In a synthetic MongoDB Benchmark (YCSB) seeing an average of ~19% throughput improvement on POWER10(Radix MMU + 64K Page Size) with MGLRU patches on top of v5.16 kernel for MongoDB + YCSB bench across three different request distriburions namely Exponential,Uniform and Zipfan
Thanks, Vaibhav. I'll post the next version in a few days and include your tested-by tag.