Skip to contentSkip to conversation

LUF(Lazy Unmap Flush) reducing tlb numbers over 90%

PATCH Superseded Byungchul Park 7 participants 36 messages May 31, 2024 – Jun 14, 2024

Subsystems (13)
Memory management x86 VFS MM core +9 more subsystems

v11 of 12 Superseded Latest is v12 →

lore
Share & tools
lore

Byungchul Park

author reviewer
#0 lore raw

v11 cover LUF(Lazy Unmap Flush) reducing tlb numbers over 90%

Hi everyone,

While I'm working with a tiered memory system e.g. CXL memory, I have been facing migration overhead esp. tlb shootdown on promotion or demotion between different tiers. Yeah.. most tlb shootdowns on migration through hinting fault can be avoided thanks to Huang Ying's work, commit 4d4b6d66db ("mm,unmap: avoid flushing tlb in batch if PTE is inaccessible").

However, it's only for migration through hinting fault. I thought it'd be much better if we have a general mechanism to reduce all the tlb numbers that we can apply to any unmap code, that we normally believe tlb flush should be followed.

I'm suggesting a new mechanism, LUF(Lazy Unmap Flush), that defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read-only and were unmapped, as long as the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.

tlb flush can be defered when folios get unmapped as long as it guarantees to perform tlb flush needed, before the folios actually become used, of course, only if all the corresponding ptes don't have write permission. Otherwise, the system will get messed up.

To achieve that, for the folios that map only to non-writable tlb entries, prevent tlb flush during unmapping but perform it just before the folios actually become used, out of buddy or pcp.

However, we should cancel the pending by LUF and perform the deferred TLB flush right away when:

1. a writable pte is newly set through fault handler
2. a file is updated
3. kasan needs poisoning on free
4. the kernel wants to init pages on free

No matter what type of workload is used for performance evaluation, the result would be positive thanks to the unconditional reduction of tlb flushes, tlb misses and interrupts. For the test, I picked up one of the most popular and heavy workload, llama.cpp that is a LLM(Large Language Model) inference engine.

The result would depend on memory latency and how often reclaim runs, which implies tlb miss overhead and how many times unmapping happens. In my system, the result shows:

1. tlb shootdown interrupts are reduced about 97%.
2. The test program runtime is reduced about 4.5%.

The test environment and the result is like:

Machine: bare metal, x86_64, Intel(R) Xeon(R) Gold 6430

   CPU: 1 socket 64 core with hyper thread on
   Numa: 2 nodes (64 CPUs DRAM 42GB, no CPUs CXL expander 98GB)
   Config: swap off, numa balancing tiering on, demotion enabled

The test set:

llama.cpp/main -m $(70G_model1) -p "who are you?" -s 1 -t 15 -n 20 & llama.cpp/main -m $(70G_model2) -p "who are you?" -s 1 -t 15 -n 20 & llama.cpp/main -m $(70G_model3) -p "who are you?" -s 1 -t 15 -n 20 & wait

where -t: nr of threads, -s: seed used to make the runtime stable, -n: nr of tokens that determines the runtime, -p: prompt to ask, -m: LLM model to use.

Run the test set 5 times successively with caches dropped every run via 'echo 3 > /proc/sys/vm/drop_caches'. Each inference prints its runtime at the end of each.

1. Runtime from the output of llama.cpp:

BEFORE
------

   llama_print_timings:       total time =  883450.54 ms /    24 tokens
   llama_print_timings:       total time =  861665.91 ms /    24 tokens
   llama_print_timings:       total time =  898079.02 ms /    24 tokens
   llama_print_timings:       total time =  879897.69 ms /    24 tokens
   llama_print_timings:       total time =  892360.75 ms /    24 tokens
   llama_print_timings:       total time =  884587.85 ms /    24 tokens
   llama_print_timings:       total time =  861023.19 ms /    24 tokens
   llama_print_timings:       total time =  900022.18 ms /    24 tokens
   llama_print_timings:       total time =  878771.88 ms /    24 tokens
   llama_print_timings:       total time =  889027.98 ms /    24 tokens
   llama_print_timings:       total time =  880783.90 ms /    24 tokens
   llama_print_timings:       total time =  856475.29 ms /    24 tokens
   llama_print_timings:       total time =  896842.21 ms /    24 tokens
   llama_print_timings:       total time =  878883.53 ms /    24 tokens
   llama_print_timings:       total time =  890122.10 ms /    24 tokens

AFTER
-----

   llama_print_timings:       total time =  871060.86 ms /    24 tokens
   llama_print_timings:       total time =  825609.53 ms /    24 tokens
   llama_print_timings:       total time =  836854.81 ms /    24 tokens
   llama_print_timings:       total time =  843147.99 ms /    24 tokens
   llama_print_timings:       total time =  831426.65 ms /    24 tokens
   llama_print_timings:       total time =  873939.23 ms /    24 tokens
   llama_print_timings:       total time =  826127.69 ms /    24 tokens
   llama_print_timings:       total time =  835489.26 ms /    24 tokens
   llama_print_timings:       total time =  842589.62 ms /    24 tokens
   llama_print_timings:       total time =  833700.66 ms /    24 tokens
   llama_print_timings:       total time =  875996.19 ms /    24 tokens
   llama_print_timings:       total time =  826401.73 ms /    24 tokens
   llama_print_timings:       total time =  839341.28 ms /    24 tokens
   llama_print_timings:       total time =  841075.10 ms /    24 tokens
   llama_print_timings:       total time =  835136.41 ms /    24 tokens

2. tlb shootdowns from 'cat /proc/interrupts':

BEFORE
------
TLB:

    80911532   93691786  100296251  111062810  109769109  109862429
   108968588  119175230  115779676  118377498  119325266  120300143
   124514185  116697222  121068466  118031913  122660681  117494403
   121819907  116960596  120936335  117217061  118630217  122322724
   119595577  111693298  119232201  120030377  115334687  113179982
   118808254  116353592  140987367  137095516  131724276  139742240
   136501150  130428761  127585535  132483981  133430250  133756207
   131786710  126365824  129812539  133850040  131742690  125142213
   128572830  132234350  131945922  128417707  133355434  129972846
   126331823  134050849  133991626  121129038  124637283  132830916
   126875507  122322440  125776487  124340278   TLB shootdowns

AFTER
-----
TLB:

     2121206    2615108    2983494    2911950    3055086    3092672
     3204894    3346082    3286744    3307310    3357296    3315940
     3428034    3112596    3143325    3185551    3186493    3322314
     3330523    3339663    3156064    3272070    3296309    3198962
     3332662    3315870    3234467    3353240    3281234    3300666
     3345452    3173097    4009196    3932215    3898735    3726531
     3717982    3671726    3728788    3724613    3799147    3691764
     3620630    3684655    3666688    3393974    3448651    3487593
     3446357    3618418    3671920    3712949    3575264    3715385
     3641513    3630897    3691047    3630690    3504933    3662647
     3629926    3443044    3832970    3548813   TLB shootdowns

---

Changes from v10:

1. Rebase on akpm/mm.git mm-unstable as of May 28, 2024.
2. Cancel LUF on file_end_write() when updating a file. (feedbacked by Dave Hansen)
3. Cancel LUF after every update_mmu_tlb*() in fault handler if it's going to set the pte to writable.
4. Cancel LUF on freeing pages if kasan needs poisoning. (feedbacked by David Hildenbrand)
5. Cancel LUF on freeing pages if want_init_on_free(). (feedbacked by David Hildenbrand)
6. Change test iteration from 10 times to 5 times.
7. Not include perf result. (I will add it if needed.)
8. Trivial optimization.

Changes from v9:

1. Expand the candidate to apply this mechanism:
BEFORE - The souce folios at any type of migration. AFTER - Any folios that have been unmapped and freed.
2. Change the workload for test:
BEFORE - XSBench
AFTER - llama.cpp (one of the most popluar real workload)
3. Change the test environment:
BEFORE - qemu machine, too small DRAM(1GB), large remote mem AFTER - bare metal, real CXL memory, practical memory size
4. Rename the mechanism from MIGRC(Migration Read Copy) to LUF(Lazy Unmap Flush) to reflect the current version of the mechanism can be applied not only to unmap during migration but any unmap code e.g. unmap in shrink_folio_list().
5. Fix build error for riscv. (feedbacked by kernel test bot)
6. Supplement commit messages to describe what this mechanism is for, especially in the patches for arch code. (feedbacked by Thomas Gleixner)
7. Clean up some trivial things.

Changes from v8:

1. Rebase on akpm/mm.git mm-unstable as of April 18, 2024.
2. Supplement comments and commit message.
3. Change the candidate to apply migrc mechanism:
BEFORE - The source folios at demotion and promotion. AFTER - The souce folios at any type of migration.
4. Change how migrc mechanism works:
BEFORE - Reduce tlb flushes by deferring folio_free() for
source folios during demotion and promotion.
AFTER - Reduce tlb flushes by deferring tlb flush until they
actually become used, out of pcp or buddy. The current version of migrc does *not* defer calling folio_free() but let it go as it is as the same as vanilla kernel, with the folios marked kind of 'need to tlb flush'. And then handle the flush when the page exits from pcp or buddy so as to prevent changing vm stats e.g. free pages.

Changes from v7:

1. Rewrite cover letter to explain what 'migrc' mechasism is. (feedbacked by Andrew Morton)
2. Supplement the commit message of a patch 'mm: Add APIs to free a folio directly to the buddy bypassing pcp'. (feedbacked by Andrew Morton)

Changes from v6:

1. Fix build errors in case of CONFIG_ARCH_WANT_BATCHED_UNMAP_tlb_FLUSH disabled by moving migrc_flush_{start,end}() calls from arch code to try_to_unmap_flush() in mm/rmap.c.

Changes from v5:

1. Fix build errors in case of CONFIG_MIGRATION disabled or CONFIG_HWPOISON_INJECT moduled. (feedbacked by kernel test bot and Raymond Jay Golo)
2. Organize migrc code with two kconfigs, CONFIG_MIGRATION and CONFIG_ARCH_WANT_BATCHED_UNMAP_tlb_FLUSH.

Changes from v4:

1. Rebase on v6.7.
2. Fix build errors in arm64 that is doing nothing for tlb flush but has CONFIG_ARCH_WANT_BATCHED_UNMAP_tlb_FLUSH. (reported by kernel test robot)
3. Don't use any page flag. So the system would give up migrc mechanism more often but it's okay. The final improvement is good enough.
4. Instead, optimize full tlb flush(arch_tlbbatch_flush()) by avoiding redundant CPUs from tlb flush.

Changes from v3:

1. Don't use the kconfig, CONFIG_MIGRC, and remove sysctl knob, migrc_enable. (feedbacked by Nadav)
2. Remove the optimization skipping CPUs that have already performed tlb flushes needed by any reason when performing tlb flushes by migrc because I can't tell the performance difference between w/ the optimization and w/o that. (feedbacked by Nadav)
3. Minimize arch-specific code. While at it, move all the migrc declarations and inline functions from include/linux/mm.h to mm/internal.h (feedbacked by Dave Hansen, Nadav)
4. Separate a part making migrc paused when the system is in high memory pressure to another patch. (feedbacked by Nadav)
5. Rename:
a. arch_tlbbatch_clean() to arch_tlbbatch_clear(), b. tlb_ubc_nowr to tlb_ubc_ro,
c. migrc_try_flush_free_folios() to migrc_flush_free_folios(), d. migrc_stop to migrc_pause.
(feedbacked by Nadav)
6. Use ->lru list_head instead of introducing a new llist_head. (feedbacked by Nadav)
7. Use non-atomic operations of page-flag when it's safe. (feedbacked by Nadav)
8. Use stack instead of keeping a pointer of 'struct migrc_req' in struct task, which is for manipulating it locally. (feedbacked by Nadav)
9. Replace a lot of simple functions to inline functions placed in a header, mm/internal.h. (feedbacked by Nadav)
10. Add additional sufficient comments. (feedbacked by Nadav)
11. Remove a lot of wrapper functions. (feedbacked by Nadav)

Changes from RFC v2:

1. Remove additional occupation in struct page. To do that, unioned with lru field for migrc's list and added a page flag. I know page flag is a thing that we don't like to add but no choice because migrc should distinguish folios under migrc's control from others. Instead, I force migrc to be used only on 64 bit system to mitigate you guys from getting angry.
2. Remove meaningless internal object allocator that I introduced to minimize impact onto the system. However, a ton of tests showed there was no difference.
3. Stop migrc from working when the system is in high memory pressure like about to perform direct reclaim. At the condition where the swap mechanism is heavily used, I found the system suffered from regression without this control.
4. Exclude folios that pte_dirty() == true from migrc's interest so that migrc can work simpler.
5. Combine several patches that work tightly coupled to one.
6. Add sufficient comments for better review.
7. Manage migrc's request in per-node manner (from globally).
8. Add tlb miss improvement in commit message.
9. Test with more CPUs(4 -> 16) to see bigger improvement.

Changes from RFC:

1. Fix a bug triggered when a destination folio at the previous migration becomes a source folio at the next migration, before the folio gets handled properly so that the folio can play with another migration. There was inconsistency in the folio's state. Fixed it.
2. Split the patch set into more pieces so that the folks can review better. (Feedbacked by Nadav Amit)
3. Fix a wrong usage of barrier e.g. smp_mb__after_atomic(). (Feedbacked by Nadav Amit)
4. Tried to add sufficient comments to explain the patch set better. (Feedbacked by Nadav Amit)

Byungchul Park (12):
x86/tlb: add APIs manipulating tlb batch's arch data
arm64: tlbflush: add APIs manipulating tlb batch's arch data riscv, tlb: add APIs manipulating tlb batch's arch data x86/tlb, riscv/tlb, mm/rmap: separate arch_tlbbatch_clear() out of
arch_tlbbatch_flush()
mm: buddy: make room for a new variable, ugen, in struct page
mm: add folio_put_ugen() to deliver unmap generation number to pcp or
buddy
mm: add a parameter, unmap generation number, to free_unref_folios() mm/rmap: recognize read-only tlb entries during batched tlb flush
mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get
unmapped
mm: separate move/undo parts from migrate_pages_batch() mm, migrate: apply luf mechanism to unmapping during migration mm, vmscan: apply luf mechanism to unmapping during folio reclaim

 arch/arm64/include/asm/tlbflush.h |  18 ++
 arch/riscv/include/asm/tlbflush.h |  21 ++
 arch/riscv/mm/tlbflush.c          |   1 -
 arch/x86/include/asm/tlbflush.h   |  18 ++
 arch/x86/mm/tlb.c                 |   2 -
 include/linux/fs.h                |   6 +
 include/linux/mm.h                |  22 ++
 include/linux/mm_types.h          |  48 +++-
 include/linux/rmap.h              |   7 +-
 include/linux/sched.h             |  11 +
 mm/compaction.c                   |  12 +-
 mm/internal.h                     | 115 +++++++++-
 mm/memory.c                       |  39 +++-
 mm/migrate.c                      | 184 ++++++++++------
 mm/page_alloc.c                   | 174 ++++++++++++---
 mm/page_isolation.c               |   6 +
 mm/page_reporting.c               |  10 +
 mm/rmap.c                         | 352 +++++++++++++++++++++++++++++-
 mm/swap.c                         |  18 +-
 mm/vmscan.c                       |  29 ++-
 20 files changed, 959 insertions(+), 134 deletions(-)

base-commit: b610f75d19a34b488021b9a4d2e3bd1cf34fc200

Signature · 1 line
2.17.1
author reviewer
#1 lore raw

PATCH v11 1/12 x86/tlb: add APIs manipulating tlb batch's arch data

A new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read-only and were unmapped, since the contents of the folios wouldn't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.

This is a preparation for the mechanism that needs to recognize read-only tlb entries by separating tlb batch arch data into two, one is for read-only entries and the other is for writable ones, and merging those two when needed.

It also optimizes tlb shootdown by skipping CPUs that have already performed tlb flush needed since. To support it, added APIs manipulating arch data for x86.

---
 arch/x86/include/asm/tlbflush.h | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
arch/x86/include/asm/tlbflush.h +18 -0
index 25726893c6f4..a14f77c5cdde 100644
@@ -5,6 +5,7 @@
 #include <linux/mm_types.h>
 #include <linux/mmu_notifier.h>
 #include <linux/sched.h>
+#include <linux/cpumask.h>
 
 #include <asm/processor.h>
 #include <asm/cpufeature.h>
@@ -293,6 +294,23 @@ static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm)
 
 extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch);
 
+static inline void arch_tlbbatch_clear(struct arch_tlbflush_unmap_batch *batch)
+{
+	cpumask_clear(&batch->cpumask);
+}
+
+static inline void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst,
+		struct arch_tlbflush_unmap_batch *bsrc)
+{
+	cpumask_or(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask);
+}
+
+static inline bool arch_tlbbatch_done(struct arch_tlbflush_unmap_batch *bdst,
+		struct arch_tlbflush_unmap_batch *bsrc)
+{
+	return !cpumask_andnot(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask);
+}
+
 static inline bool pte_flags_need_flush(unsigned long oldflags,
 					unsigned long newflags,
 					bool ignore_access)
Signature · 1 line
2.17.1
author reviewer
#2 lore raw

PATCH v11 2/12 arm64: tlbflush: add APIs manipulating tlb batch's arch data

In reply to Byungchul Park

A new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read only and were unmapped, since the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.

This is a preparation for the mechanism that requires to manipulate tlb batch's arch data. Even though arm64 does nothing for tlb things, arch with CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH should provide the APIs.

---
 arch/arm64/include/asm/tlbflush.h | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
arch/arm64/include/asm/tlbflush.h +18 -0
index 95fbc8c05607..4fefc1f90304 100644
@@ -354,6 +354,24 @@ static inline void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
 	dsb(ish);
 }
 
+static inline void arch_tlbbatch_clear(struct arch_tlbflush_unmap_batch *batch)
+{
+	/* nothing to do */
+}
+
+static inline void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst,
+			       struct arch_tlbflush_unmap_batch *bsrc)
+{
+	/* nothing to do */
+}
+
+static inline bool arch_tlbbatch_done(struct arch_tlbflush_unmap_batch *bdst,
+			       struct arch_tlbflush_unmap_batch *bsrc)
+{
+	/* Kernel can consider tlb batch always has been done. */
+	return true;
+}
+
 /*
  * This is meant to avoid soft lock-ups on large TLB flushing ranges and not
  * necessarily a performance improvement.
Signature · 1 line
2.17.1
author reviewer
#3 lore raw

PATCH v11 3/12 riscv, tlb: add APIs manipulating tlb batch's arch data

In reply to Byungchul Park

A new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read only and were unmapped, since the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.

This is a preparation for the mechanism that needs to recognize read-only tlb entries by separating tlb batch arch data into two, one is for read-only entries and the other is for writable ones, and merging those two when needed.

It also optimizes tlb shootdown by skipping CPUs that have already performed tlb flush needed since. To support it, added APIs manipulating arch data for riscv.

---
 arch/riscv/include/asm/tlbflush.h | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
arch/riscv/include/asm/tlbflush.h +21 -0
index 72e559934952..c0bfb9b2bf54 100644
@@ -8,6 +8,7 @@
 #define _ASM_RISCV_TLBFLUSH_H
 
 #include <linux/mm_types.h>
+#include <linux/cpumask.h>
 #include <asm/smp.h>
 #include <asm/errata_list.h>
 
@@ -65,6 +66,26 @@ void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch,
 void arch_flush_tlb_batched_pending(struct mm_struct *mm);
 void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch);
 
+static inline void arch_tlbbatch_clear(struct arch_tlbflush_unmap_batch *batch)
+{
+	cpumask_clear(&batch->cpumask);
+
+}
+
+static inline void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst,
+		struct arch_tlbflush_unmap_batch *bsrc)
+{
+	cpumask_or(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask);
+
+}
+
+static inline bool arch_tlbbatch_done(struct arch_tlbflush_unmap_batch *bdst,
+		struct arch_tlbflush_unmap_batch *bsrc)
+{
+	return !cpumask_andnot(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask);
+
+}
+
 extern unsigned long tlb_flush_all_threshold;
 #else /* CONFIG_MMU */
 #define local_flush_tlb_all()			do { } while (0)
Signature · 1 line
2.17.1
author reviewer
#4 lore raw

PATCH v11 4/12 x86/tlb, riscv/tlb, mm/rmap: separate arch_tlbbatch_clear() out of arch_tlbbatch_flush()

In reply to Byungchul Park

A new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read only and were unmapped, since the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.

This is a preparation for the mechanism that requires to avoid redundant tlb flush by manipulating tlb batch's arch data. To achieve that, we need to separate the part clearing the tlb batch's arch data out of arch_tlbbatch_flush().

---
 arch/riscv/mm/tlbflush.c | 1 -
 arch/x86/mm/tlb.c        | 2 --
 mm/rmap.c                | 1 +
 3 files changed, 1 insertion(+), 3 deletions(-)
arch/riscv/mm/tlbflush.c +0 -1
index 9b6e86ce3867..36f996af6256 100644
@@ -201,5 +201,4 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
 {
 	__flush_tlb_range(&batch->cpumask, FLUSH_TLB_NO_ASID, 0,
 			  FLUSH_TLB_MAX_SIZE, PAGE_SIZE);
-	cpumask_clear(&batch->cpumask);
 }
arch/x86/mm/tlb.c +0 -2
index 44ac64f3a047..24bce69222cd 100644
@@ -1265,8 +1265,6 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
 		local_irq_enable();
 	}
 
-	cpumask_clear(&batch->cpumask);
-
 	put_flush_tlb_info();
 	put_cpu();
 }
mm/rmap.c +1 -0
index 52357d79917c..a65a94aada8d 100644
@@ -648,6 +648,7 @@ void try_to_unmap_flush(void)
 		return;
 
 	arch_tlbbatch_flush(&tlb_ubc->arch);
+	arch_tlbbatch_clear(&tlb_ubc->arch);
 	tlb_ubc->flush_required = false;
 	tlb_ubc->writable = false;
 }
Signature · 1 line
2.17.1
author reviewer
#5 lore raw

PATCH v11 5/12 mm: buddy: make room for a new variable, ugen, in struct page

In reply to Byungchul Park

Functionally, no change. This is a preparation for luf mechanism that tracks need of tlb flush for each page residing in buddy, using a generation number in struct page.

Fortunately, since the private field in struct page is used only to store page order in buddy, ranging from 0 to MAX_PAGE_ORDER, that can be covered with unsigned short int. So splitted it into two smaller ones, order and ugen, so that the both can be used in buddy at the same time.

---
 include/linux/mm_types.h | 40 +++++++++++++++++++++++++++++++++-------
 mm/internal.h            |  4 ++--
 mm/page_alloc.c          | 13 ++++++++-----
 3 files changed, 43 insertions(+), 14 deletions(-)
include/linux/mm_types.h +33 -7
index 24323c7d0bd4..37eb3000267c 100644
@@ -108,13 +108,25 @@ struct page {
 				pgoff_t index;		/* Our offset within mapping. */
 				unsigned long share;	/* share count for fsdax */
 			};
-			/**
-			 * @private: Mapping-private opaque data.
-			 * Usually used for buffer_heads if PagePrivate.
-			 * Used for swp_entry_t if PageSwapCache.
-			 * Indicates order in the buddy system if PageBuddy.
-			 */
-			unsigned long private;
+			union {
+				/**
+				 * @private: Mapping-private opaque data.
+				 * Usually used for buffer_heads if PagePrivate.
+				 * Used for swp_entry_t if PageSwapCache.
+				 */
+				unsigned long private;
+				struct {
+					/*
+					 * Indicates order in the buddy system if PageBuddy.
+					 */
+					unsigned short int order;
+					/*
+					 * Tracks need of tlb flush used by luf,
+					 * which stands for lazy unmap flush.
+					 */
+					unsigned short int ugen;
+				};
+			};
 		};
 		struct {	/* page_pool used by netstack */
 			/**
@@ -521,6 +533,20 @@ static inline void set_page_private(struct page *page, unsigned long private)
 	page->private = private;
 }
 
+#define page_buddy_order(page)		((page)->order)
+
+static inline void set_page_buddy_order(struct page *page, unsigned int order)
+{
+	page->order = (unsigned short int)order;
+}
+
+#define page_buddy_ugen(page)		((page)->ugen)
+
+static inline void set_page_buddy_ugen(struct page *page, unsigned short int ugen)
+{
+	page->ugen = ugen;
+}
+
 static inline void *folio_get_private(struct folio *folio)
 {
 	return folio->private;
mm/internal.h +2 -2
index bbec99cc9d9d..552e1061d36d 100644
@@ -461,7 +461,7 @@ struct alloc_context {
 static inline unsigned int buddy_order(struct page *page)
 {
 	/* PageBuddy() must be checked by the caller */
-	return page_private(page);
+	return page_buddy_order(page);
 }
 
 /*
@@ -475,7 +475,7 @@ static inline unsigned int buddy_order(struct page *page)
  * times, potentially observing different values in the tests and the actual
  * use of the result.
  */
-#define buddy_order_unsafe(page)	READ_ONCE(page_private(page))
+#define buddy_order_unsafe(page)	READ_ONCE(page_buddy_order(page))
 
 /*
  * This function checks whether a page is free && is the buddy
mm/page_alloc.c +8 -5
index b1e3eb5787de..ae57dd8718fe 100644
@@ -565,9 +565,12 @@ void prep_compound_page(struct page *page, unsigned int order)
 	prep_compound_head(page, order);
 }
 
-static inline void set_buddy_order(struct page *page, unsigned int order)
+static inline void set_buddy_order_ugen(struct page *page,
+					unsigned int order,
+					unsigned short int ugen)
 {
-	set_page_private(page, order);
+	set_page_buddy_order(page, order);
+	set_page_buddy_ugen(page, order);
 	__SetPageBuddy(page);
 }
 
@@ -826,7 +829,7 @@ static inline void __free_one_page(struct page *page,
 	}
 
 done_merging:
-	set_buddy_order(page, order);
+	set_buddy_order_ugen(page, order, 0);
 
 	if (fpi_flags & FPI_TO_TAIL)
 		to_tail = true;
@@ -1336,7 +1339,7 @@ static inline void expand(struct zone *zone, struct page *page,
 			continue;
 
 		__add_to_free_list(&page[size], zone, high, migratetype, false);
-		set_buddy_order(&page[size], high);
+		set_buddy_order_ugen(&page[size], high, 0);
 		nr_added += size;
 	}
 	account_freepages(zone, nr_added, migratetype);
@@ -6801,7 +6804,7 @@ static void break_down_buddy_pages(struct zone *zone, struct page *page,
 			continue;
 
 		add_to_free_list(current_buddy, zone, high, migratetype, false);
-		set_buddy_order(current_buddy, high);
+		set_buddy_order_ugen(current_buddy, high, 0);
 	}
 }
 
Signature · 1 line
2.17.1
author reviewer
#6 lore raw

PATCH v11 6/12 mm: add folio_put_ugen() to deliver unmap generation number to pcp or buddy

In reply to Byungchul Park

Introduced a new API, folio_put_ugen(), to deliver unmap generation number to pcp or buddy that will be used by luf mechanism to track need of tlb flush for each page residing in pcp or buddy.

For now, the delivery should work for the following call path that is of releasing source folios during migration:

	folio_put_ugen()
	   __folio_put_ugen()
	      free_unref_page()
	         free_unref_page_commit()
	         free_one_page()
	            __free_one_page()

The generation number should be handed over properly when pages travel between pcp and buddy, and must do necessary handling on exit from pcp or buddy. This patch doesn't include actual body for tlb flush on the exit, which will be filled by the main patch of luf mechanism.

---
 include/linux/mm.h    |  22 +++++++
 include/linux/sched.h |   1 +
 mm/compaction.c       |  10 +++
 mm/internal.h         |  71 ++++++++++++++++++++-
 mm/page_alloc.c       | 144 ++++++++++++++++++++++++++++++++++--------
 mm/page_isolation.c   |   6 ++
 mm/page_reporting.c   |  10 +++
 mm/swap.c             |  12 +++-
 8 files changed, 248 insertions(+), 28 deletions(-)
include/linux/mm.h +22 -0
index 3aa1b6889bcc..54cb6316a76d 100644
@@ -1316,6 +1316,7 @@ static inline struct folio *virt_to_folio(const void *x)
 }
 
 void __folio_put(struct folio *folio);
+void __folio_put_ugen(struct folio *folio, unsigned short int ugen);
 
 void put_pages_list(struct list_head *pages);
 
@@ -1508,6 +1509,27 @@ static inline void folio_put(struct folio *folio)
 		__folio_put(folio);
 }
 
+/**
+ * folio_put_ugen - Decrement the last reference count on a folio.
+ * @folio: The folio.
+ * @ugen: The unmap generation # of TLB flush that the folio requires.
+ *
+ * The folio's reference count should be one since the only user, folio
+ * migration code, calls folio_put_ugen() only when the folio has no
+ * reference else.  The memory will be released back to the page
+ * allocator and may be used by another allocation immediately.  Do not
+ * access the memory or the struct folio after calling folio_put_ugen().
+ *
+ * Context: May be called in process or interrupt context, but not in NMI
+ * context.  May be called while holding a spinlock.
+ */
+static inline void folio_put_ugen(struct folio *folio, unsigned short int ugen)
+{
+	if (WARN_ON(!folio_put_testzero(folio)))
+		return;
+	__folio_put_ugen(folio, ugen);
+}
+
 /**
  * folio_put_refs - Reduce the reference count on a folio.
  * @folio: The folio.
include/linux/sched.h +1 -0
index 61591ac6eab6..ab5a2ed79b88 100644
@@ -1340,6 +1340,7 @@ struct task_struct {
 #endif
 
 	struct tlbflush_unmap_batch	tlb_ubc;
+	unsigned short int		ugen;
 
 	/* Cache last used pipe for splice(): */
 	struct pipe_inode_info		*splice_pipe;
mm/compaction.c +10 -0
index e731d45befc7..13799fbb2a9a 100644
@@ -701,6 +701,11 @@ static unsigned long isolate_freepages_block(struct compact_control *cc,
 	if (locked)
 		spin_unlock_irqrestore(&cc->zone->lock, flags);
 
+	/*
+	 * Check and flush before using the isolated pages.
+	 */
+	check_flush_task_ugen();
+
 	/*
 	 * Be careful to not go outside of the pageblock.
 	 */
@@ -1673,6 +1678,11 @@ static void fast_isolate_freepages(struct compact_control *cc)
 
 		spin_unlock_irqrestore(&cc->zone->lock, flags);
 
+		/*
+		 * Check and flush before using the isolated pages.
+		 */
+		check_flush_task_ugen();
+
 		/* Skip fast search if enough freepages isolated */
 		if (cc->nr_freepages >= cc->nr_migratepages)
 			break;
mm/internal.h +70 -1

83 lines not on this page · raw message

mm/page_alloc.c +118 -26

405 lines not on this page · raw message

mm/page_isolation.c +6 -0
index 042937d5abe4..5823da60a621 100644
@@ -260,6 +260,12 @@ static void unset_migratetype_isolate(struct page *page, int migratetype)
 	zone->nr_isolate_pageblock--;
 out:
 	spin_unlock_irqrestore(&zone->lock, flags);
+
+	/*
+	 * Check and flush for the pages that have been isolated.
+	 */
+	if (isolated_page)
+		check_flush_task_ugen();
 }
 
 static inline struct page *
mm/page_reporting.c +10 -0
index e4c428e61d8c..4f94a3ea1b22 100644
@@ -221,6 +221,11 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,
 		/* release lock before waiting on report processing */
 		spin_unlock_irq(&zone->lock);
 
+		/*
+		 * Check and flush before using the isolated pages.
+		 */
+		check_flush_task_ugen();
+
 		/* begin processing pages in local list */
 		err = prdev->report(prdev, sgl, PAGE_REPORTING_CAPACITY);
 
@@ -253,6 +258,11 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,
 
 	spin_unlock_irq(&zone->lock);
 
+	/*
+	 * Check and flush before using the isolated pages.
+	 */
+	check_flush_task_ugen();
+
 	return err;
 }
 
mm/swap.c +11 -1
index dc205bdfbbd4..dae169b19ab9 100644
@@ -125,10 +125,20 @@ void __folio_put(struct folio *folio)
 	page_cache_release(folio);
 	folio_undo_large_rmappable(folio);
 	mem_cgroup_uncharge(folio);
-	free_unref_page(&folio->page, folio_order(folio));
+	free_unref_page(&folio->page, folio_order(folio), 0);
 }
 EXPORT_SYMBOL(__folio_put);
 
+void __folio_put_ugen(struct folio *folio, unsigned short int ugen)
+{
+	if (WARN_ON(!can_luf_folio(folio)))
+		return;
+
+	page_cache_release(folio);
+	mem_cgroup_uncharge(folio);
+	free_unref_page(&folio->page, 0, ugen);
+}
+
 /**
  * put_pages_list() - release a list of pages
  * @pages: list of pages threaded on page->lru
Signature · 1 line
2.17.1
author reviewer
#7 lore raw

PATCH v11 7/12 mm: add a parameter, unmap generation number, to free_unref_folios()

In reply to Byungchul Park

Unmap generation number is used by luf mechanism to track need of tlb flush for each page residing in pcp or buddy.

The number should be delivered to pcp or buddy via free_unref_folios() that is for releasing folios that have been unmapped during reclaim in shrink_folio_list().

---
 mm/internal.h   |  2 +-
 mm/page_alloc.c | 10 +++++-----
 mm/swap.c       |  6 +++---
 mm/vmscan.c     |  8 ++++----
 4 files changed, 13 insertions(+), 13 deletions(-)
mm/internal.h +1 -1
index 380ae980e4f9..dba6d0eb7b6d 100644
@@ -662,7 +662,7 @@ extern bool free_pages_prepare(struct page *page, unsigned int order);
 extern int user_min_free_kbytes;
 
 void free_unref_page(struct page *page, unsigned int order, unsigned short int ugen);
-void free_unref_folios(struct folio_batch *fbatch);
+void free_unref_folios(struct folio_batch *fbatch, unsigned short int ugen);
 
 extern void zone_pcp_reset(struct zone *zone);
 extern void zone_pcp_disable(struct zone *zone);
mm/page_alloc.c +5 -5
index 6fbbe45be5ae..c9acb4da91e0 100644
@@ -2698,7 +2698,7 @@ void free_unref_page(struct page *page, unsigned int order,
 /*
  * Free a batch of folios
  */
-void free_unref_folios(struct folio_batch *folios)
+void free_unref_folios(struct folio_batch *folios, unsigned short int ugen)
 {
 	unsigned long __maybe_unused UP_flags;
 	struct per_cpu_pages *pcp = NULL;
@@ -2720,7 +2720,7 @@ void free_unref_folios(struct folio_batch *folios)
 		 */
 		if (!pcp_allowed_order(order)) {
 			free_one_page(folio_zone(folio), &folio->page,
-				      pfn, order, FPI_NONE, 0);
+				      pfn, order, FPI_NONE, ugen);
 			continue;
 		}
 		folio->private = (void *)(unsigned long)order;
@@ -2756,7 +2756,7 @@ void free_unref_folios(struct folio_batch *folios)
 			 */
 			if (is_migrate_isolate(migratetype)) {
 				free_one_page(zone, &folio->page, pfn,
-					      order, FPI_NONE, 0);
+					      order, FPI_NONE, ugen);
 				continue;
 			}
 
@@ -2769,7 +2769,7 @@ void free_unref_folios(struct folio_batch *folios)
 			if (unlikely(!pcp)) {
 				pcp_trylock_finish(UP_flags);
 				free_one_page(zone, &folio->page, pfn,
-					      order, FPI_NONE, 0);
+					      order, FPI_NONE, ugen);
 				continue;
 			}
 			locked_zone = zone;
@@ -2784,7 +2784,7 @@ void free_unref_folios(struct folio_batch *folios)
 
 		trace_mm_page_free_batched(&folio->page);
 		free_unref_page_commit(zone, pcp, &folio->page, migratetype,
-				order, 0);
+				order, ugen);
 	}
 
 	if (pcp) {
mm/swap.c +3 -3
index dae169b19ab9..67605bbfc95c 100644
@@ -161,11 +161,11 @@ void put_pages_list(struct list_head *pages)
 		/* LRU flag must be clear because it's passed using the lru */
 		if (folio_batch_add(&fbatch, folio) > 0)
 			continue;
-		free_unref_folios(&fbatch);
+		free_unref_folios(&fbatch, 0);
 	}
 
 	if (fbatch.nr)
-		free_unref_folios(&fbatch);
+		free_unref_folios(&fbatch, 0);
 	INIT_LIST_HEAD(pages);
 }
 EXPORT_SYMBOL(put_pages_list);
@@ -1027,7 +1027,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs)
 
 	folios->nr = j;
 	mem_cgroup_uncharge_folios(folios);
-	free_unref_folios(folios);
+	free_unref_folios(folios, 0);
 }
 EXPORT_SYMBOL(folios_put_refs);
 
mm/vmscan.c +4 -4
index b9170f767353..15efe6f0edce 100644
@@ -1461,7 +1461,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 		if (folio_batch_add(&free_folios, folio) == 0) {
 			mem_cgroup_uncharge_folios(&free_folios);
 			try_to_unmap_flush();
-			free_unref_folios(&free_folios);
+			free_unref_folios(&free_folios, 0);
 		}
 		continue;
 
@@ -1528,7 +1528,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 
 	mem_cgroup_uncharge_folios(&free_folios);
 	try_to_unmap_flush();
-	free_unref_folios(&free_folios);
+	free_unref_folios(&free_folios, 0);
 
 	list_splice(&ret_folios, folio_list);
 	count_vm_events(PGACTIVATE, pgactivate);
@@ -1868,7 +1868,7 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec,
 			if (folio_batch_add(&free_folios, folio) == 0) {
 				spin_unlock_irq(&lruvec->lru_lock);
 				mem_cgroup_uncharge_folios(&free_folios);
-				free_unref_folios(&free_folios);
+				free_unref_folios(&free_folios, 0);
 				spin_lock_irq(&lruvec->lru_lock);
 			}
 
@@ -1890,7 +1890,7 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec,
 	if (free_folios.nr) {
 		spin_unlock_irq(&lruvec->lru_lock);
 		mem_cgroup_uncharge_folios(&free_folios);
-		free_unref_folios(&free_folios);
+		free_unref_folios(&free_folios, 0);
 		spin_lock_irq(&lruvec->lru_lock);
 	}
 
Signature · 1 line
2.17.1
author reviewer
#8 lore raw

PATCH v11 8/12 mm/rmap: recognize read-only tlb entries during batched tlb flush

In reply to Byungchul Park

Functionally, no change. This is a preparation for luf mechanism that requires to recognize read-only tlb entries and handle them in a different way. The newly introduced API in this patch, fold_ubc(), will be used by luf mechanism.

---
 include/linux/sched.h |  1 +
 mm/internal.h         |  4 ++++
 mm/rmap.c             | 34 ++++++++++++++++++++++++++++++++--
 3 files changed, 37 insertions(+), 2 deletions(-)
include/linux/sched.h +1 -0
index ab5a2ed79b88..d9722c014157 100644
@@ -1340,6 +1340,7 @@ struct task_struct {
 #endif
 
 	struct tlbflush_unmap_batch	tlb_ubc;
+	struct tlbflush_unmap_batch	tlb_ubc_ro;
 	unsigned short int		ugen;
 
 	/* Cache last used pipe for splice(): */
mm/internal.h +4 -0
index dba6d0eb7b6d..ca6fb5b2a640 100644
@@ -1124,6 +1124,7 @@ extern struct workqueue_struct *mm_percpu_wq;
 void try_to_unmap_flush(void);
 void try_to_unmap_flush_dirty(void);
 void flush_tlb_batched_pending(struct mm_struct *mm);
+void fold_ubc(struct tlbflush_unmap_batch *dst, struct tlbflush_unmap_batch *src);
 #else
 static inline void try_to_unmap_flush(void)
 {
@@ -1134,6 +1135,9 @@ static inline void try_to_unmap_flush_dirty(void)
 static inline void flush_tlb_batched_pending(struct mm_struct *mm)
 {
 }
+static inline void fold_ubc(struct tlbflush_unmap_batch *dst, struct tlbflush_unmap_batch *src)
+{
+}
 #endif /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */
 
 extern const struct trace_print_flags pageflag_names[];
mm/rmap.c +32 -2
index a65a94aada8d..1a246788e867 100644
@@ -634,6 +634,28 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio,
 }
 
 #ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH
+
+void fold_ubc(struct tlbflush_unmap_batch *dst,
+	      struct tlbflush_unmap_batch *src)
+{
+	if (!src->flush_required)
+		return;
+
+	/*
+	 * Fold src to dst.
+	 */
+	arch_tlbbatch_fold(&dst->arch, &src->arch);
+	dst->writable = dst->writable || src->writable;
+	dst->flush_required = true;
+
+	/*
+	 * Reset src.
+	 */
+	arch_tlbbatch_clear(&src->arch);
+	src->flush_required = false;
+	src->writable = false;
+}
+
 /*
  * Flush TLB entries for recently unmapped pages from remote CPUs. It is
  * important if a PTE was dirty when it was unmapped that it's flushed
@@ -643,7 +665,9 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio,
 void try_to_unmap_flush(void)
 {
 	struct tlbflush_unmap_batch *tlb_ubc = &current->tlb_ubc;
+	struct tlbflush_unmap_batch *tlb_ubc_ro = &current->tlb_ubc_ro;
 
+	fold_ubc(tlb_ubc, tlb_ubc_ro);
 	if (!tlb_ubc->flush_required)
 		return;
 
@@ -657,8 +681,9 @@ void try_to_unmap_flush(void)
 void try_to_unmap_flush_dirty(void)
 {
 	struct tlbflush_unmap_batch *tlb_ubc = &current->tlb_ubc;
+	struct tlbflush_unmap_batch *tlb_ubc_ro = &current->tlb_ubc_ro;
 
-	if (tlb_ubc->writable)
+	if (tlb_ubc->writable || tlb_ubc_ro->writable)
 		try_to_unmap_flush();
 }
 
@@ -675,13 +700,18 @@ void try_to_unmap_flush_dirty(void)
 static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval,
 				      unsigned long uaddr)
 {
-	struct tlbflush_unmap_batch *tlb_ubc = &current->tlb_ubc;
+	struct tlbflush_unmap_batch *tlb_ubc;
 	int batch;
 	bool writable = pte_dirty(pteval);
 
 	if (!pte_accessible(mm, pteval))
 		return;
 
+	if (pte_write(pteval))
+		tlb_ubc = &current->tlb_ubc;
+	else
+		tlb_ubc = &current->tlb_ubc_ro;
+
 	arch_tlbbatch_add_pending(&tlb_ubc->arch, mm, uaddr);
 	tlb_ubc->flush_required = true;
 
Signature · 1 line
2.17.1
author reviewer
#9 lore raw

PATCH v11 9/12 mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped

In reply to Byungchul Park

A new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read-only and were unmapped, as long as the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.

tlb flush can be defered when folios get unmapped as long as it guarantees to perform tlb flush needed, before the folios actually become used, of course, only if all the corresponding ptes don't have write permission. Otherwise, the system will get messed up.

To achieve that, for the folios that map only to non-writable tlb entries, prevent tlb flush during unmapping but perform it just before the folios actually become used, out of buddy or pcp.

However, we should cancel the pending by LUF and perform the deferred TLB flush right away when:

1. a writable pte is newly set through fault handler
2. a file is updated
3. kasan needs poisoning on free
4. the kernel wants to init pages on free

No matter what type of workload is used for performance evaluation, the result would be positive thanks to the unconditional reduction of tlb flushes, tlb misses and interrupts. For the test, I picked up one of the most popular and heavy workload, llama.cpp that is a LLM(Large Language Model) inference engine.

The result would depend on memory latency and how often reclaim runs, which implies tlb miss overhead and how many times unmapping happens. In my system, the result shows:

1. tlb shootdown interrupts are reduced about 97%.
2. The test program runtime is reduced about 4.5%.

The test environment and the result is like:

Machine: bare metal, x86_64, Intel(R) Xeon(R) Gold 6430

   CPU: 1 socket 64 core with hyper thread on
   Numa: 2 nodes (64 CPUs DRAM 42GB, no CPUs CXL expander 98GB)
   Config: swap off, numa balancing tiering on, demotion enabled

The test set:

llama.cpp/main -m $(70G_model1) -p "who are you?" -s 1 -t 15 -n 20 & llama.cpp/main -m $(70G_model2) -p "who are you?" -s 1 -t 15 -n 20 & llama.cpp/main -m $(70G_model3) -p "who are you?" -s 1 -t 15 -n 20 & wait

where -t: nr of threads, -s: seed used to make the runtime stable, -n: nr of tokens that determines the runtime, -p: prompt to ask, -m: LLM model to use.

Run the test set 5 times successively with caches dropped every run via 'echo 3 > /proc/sys/vm/drop_caches'. Each inference prints its runtime at the end of each.

1. Runtime from the output of llama.cpp:

BEFORE
------

   llama_print_timings:       total time =  883450.54 ms /    24 tokens
   llama_print_timings:       total time =  861665.91 ms /    24 tokens
   llama_print_timings:       total time =  898079.02 ms /    24 tokens
   llama_print_timings:       total time =  879897.69 ms /    24 tokens
   llama_print_timings:       total time =  892360.75 ms /    24 tokens
   llama_print_timings:       total time =  884587.85 ms /    24 tokens
   llama_print_timings:       total time =  861023.19 ms /    24 tokens
   llama_print_timings:       total time =  900022.18 ms /    24 tokens
   llama_print_timings:       total time =  878771.88 ms /    24 tokens
   llama_print_timings:       total time =  889027.98 ms /    24 tokens
   llama_print_timings:       total time =  880783.90 ms /    24 tokens
   llama_print_timings:       total time =  856475.29 ms /    24 tokens
   llama_print_timings:       total time =  896842.21 ms /    24 tokens
   llama_print_timings:       total time =  878883.53 ms /    24 tokens
   llama_print_timings:       total time =  890122.10 ms /    24 tokens

AFTER
-----

   llama_print_timings:       total time =  871060.86 ms /    24 tokens
   llama_print_timings:       total time =  825609.53 ms /    24 tokens
   llama_print_timings:       total time =  836854.81 ms /    24 tokens
   llama_print_timings:       total time =  843147.99 ms /    24 tokens
   llama_print_timings:       total time =  831426.65 ms /    24 tokens
   llama_print_timings:       total time =  873939.23 ms /    24 tokens
   llama_print_timings:       total time =  826127.69 ms /    24 tokens
   llama_print_timings:       total time =  835489.26 ms /    24 tokens
   llama_print_timings:       total time =  842589.62 ms /    24 tokens
   llama_print_timings:       total time =  833700.66 ms /    24 tokens
   llama_print_timings:       total time =  875996.19 ms /    24 tokens
   llama_print_timings:       total time =  826401.73 ms /    24 tokens
   llama_print_timings:       total time =  839341.28 ms /    24 tokens
   llama_print_timings:       total time =  841075.10 ms /    24 tokens
   llama_print_timings:       total time =  835136.41 ms /    24 tokens

2. tlb shootdowns from 'cat /proc/interrupts':

BEFORE
------
TLB:

    80911532   93691786  100296251  111062810  109769109  109862429
   108968588  119175230  115779676  118377498  119325266  120300143
   124514185  116697222  121068466  118031913  122660681  117494403
   121819907  116960596  120936335  117217061  118630217  122322724
   119595577  111693298  119232201  120030377  115334687  113179982
   118808254  116353592  140987367  137095516  131724276  139742240
   136501150  130428761  127585535  132483981  133430250  133756207
   131786710  126365824  129812539  133850040  131742690  125142213
   128572830  132234350  131945922  128417707  133355434  129972846
   126331823  134050849  133991626  121129038  124637283  132830916
   126875507  122322440  125776487  124340278   TLB shootdowns

AFTER
-----
TLB:

     2121206    2615108    2983494    2911950    3055086    3092672
     3204894    3346082    3286744    3307310    3357296    3315940
     3428034    3112596    3143325    3185551    3186493    3322314
     3330523    3339663    3156064    3272070    3296309    3198962
     3332662    3315870    3234467    3353240    3281234    3300666
     3345452    3173097    4009196    3932215    3898735    3726531
     3717982    3671726    3728788    3724613    3799147    3691764
     3620630    3684655    3666688    3393974    3448651    3487593
     3446357    3618418    3671920    3712949    3575264    3715385
     3641513    3630897    3691047    3630690    3504933    3662647
     3629926    3443044    3832970    3548813   TLB shootdowns
---
 include/linux/fs.h       |   6 +
 include/linux/mm_types.h |   8 +
 include/linux/sched.h    |   9 ++
 mm/compaction.c          |   2 +-
 mm/internal.h            |  42 +++++-
 mm/memory.c              |  39 ++++-
 mm/page_alloc.c          |  17 ++-
 mm/rmap.c                | 315 ++++++++++++++++++++++++++++++++++++++-
 8 files changed, 420 insertions(+), 18 deletions(-)
include/linux/fs.h +6 -0
index 0283cf366c2a..03683bf66031 100644
@@ -2872,6 +2872,12 @@ static inline void file_end_write(struct file *file)
 	if (!S_ISREG(file_inode(file)->i_mode))
 		return;
 	sb_end_write(file_inode(file)->i_sb);
+
+	/*
+	 * XXX: If needed, can be optimized by avoiding luf_flush() if
+	 * the address space of the file has never been involved by luf.
+	 */
+	luf_flush();
 }
 
 /**
include/linux/mm_types.h +8 -0
index 37eb3000267c..cd52c996e8aa 100644
@@ -1223,6 +1223,14 @@ static inline unsigned int mm_cid_size(void)
 }
 #endif /* CONFIG_SCHED_MM_CID */
 
+#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH)
+void check_luf_flush(unsigned short int ugen);
+void luf_flush(void);
+#else
+static inline void check_luf_flush(unsigned short int ugen) {}
+static inline void luf_flush(void) {}
+#endif
+
 struct mmu_gather;
 extern void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm);
 extern void tlb_gather_mmu_fullmm(struct mmu_gather *tlb, struct mm_struct *mm);
include/linux/sched.h +9 -0
index d9722c014157..613ed175e5f2 100644
@@ -1341,8 +1341,17 @@ struct task_struct {
 
 	struct tlbflush_unmap_batch	tlb_ubc;
 	struct tlbflush_unmap_batch	tlb_ubc_ro;
+	struct tlbflush_unmap_batch	tlb_ubc_luf;
 	unsigned short int		ugen;
 
+#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH)
+	/*
+	 * whether all the mappings of a folio during unmap are read-only
+	 * so that luf can work on the folio
+	 */
+	bool				can_luf;
+#endif
+
 	/* Cache last used pipe for splice(): */
 	struct pipe_inode_info		*splice_pipe;
 
mm/compaction.c +1 -1
index 13799fbb2a9a..4a75c56af0b0 100644
@@ -1925,7 +1925,7 @@ static void compaction_free(struct folio *dst, unsigned long data)
 	struct page *page = &dst->page;
 
 	if (folio_put_testzero(dst)) {
-		free_pages_prepare(page, order);
+		free_pages_prepare(page, order, NULL);
 		list_add(&dst->lru, &cc->freepages[order]);
 		cc->nr_freepages += 1 << order;
 	}
mm/internal.h +37 -5

66 lines not on this page · raw message

mm/memory.c +37 -2
index 100f54fc9e6c..12c9e87e489d 100644
@@ -3011,6 +3011,15 @@ static inline int pte_unmap_same(struct vm_fault *vmf)
 	return same;
 }
 
+static bool need_luf_flush(struct vm_fault *vmf)
+{
+	if ((vmf->flags & FAULT_FLAG_ORIG_PTE_VALID) &&
+	    pte_write(vmf->orig_pte))
+		return false;
+
+	return pte_write(ptep_get(vmf->pte));
+}
+
 /*
  * Return:
  *	0:		copied succeeded
@@ -3026,6 +3035,7 @@ static inline int __wp_page_copy_user(struct page *dst, struct page *src,
 	struct vm_area_struct *vma = vmf->vma;
 	struct mm_struct *mm = vma->vm_mm;
 	unsigned long addr = vmf->address;
+	bool luf = false;
 
 	if (likely(src)) {
 		if (copy_mc_user_highpage(dst, src, addr, vma)) {
@@ -3059,8 +3069,10 @@ static inline int __wp_page_copy_user(struct page *dst, struct page *src,
 			 * Other thread has already handled the fault
 			 * and update local tlb only
 			 */
-			if (vmf->pte)
+			if (vmf->pte) {
 				update_mmu_tlb(vma, addr, vmf->pte);
+				luf = need_luf_flush(vmf);
+			}
 			ret = -EAGAIN;
 			goto pte_unlock;
 		}
@@ -3084,8 +3096,10 @@ static inline int __wp_page_copy_user(struct page *dst, struct page *src,
 		vmf->pte = pte_offset_map_lock(mm, vmf->pmd, addr, &vmf->ptl);
 		if (unlikely(!vmf->pte || !pte_same(ptep_get(vmf->pte), vmf->orig_pte))) {
 			/* The PTE changed under us, update local tlb */
-			if (vmf->pte)
+			if (vmf->pte) {
 				update_mmu_tlb(vma, addr, vmf->pte);
+				luf = need_luf_flush(vmf);
+			}
 			ret = -EAGAIN;
 			goto pte_unlock;
 		}
@@ -3112,6 +3126,8 @@ static inline int __wp_page_copy_user(struct page *dst, struct page *src,
 		pte_unmap_unlock(vmf->pte, vmf->ptl);
 	pagefault_enable();
 	kunmap_local(kaddr);
+	if (luf)
+		luf_flush();
 	flush_dcache_page(dst);
 
 	return ret;
@@ -3446,6 +3462,8 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)
 	} else if (vmf->pte) {
 		update_mmu_tlb(vma, vmf->address, vmf->pte);
 		pte_unmap_unlock(vmf->pte, vmf->ptl);
+		if (need_luf_flush(vmf))
+			luf_flush();
 	}
 
 	mmu_notifier_invalidate_range_end(&range);
@@ -3501,6 +3519,8 @@ static vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf, struct folio *folio
 	if (!pte_same(ptep_get(vmf->pte), vmf->orig_pte)) {
 		update_mmu_tlb(vmf->vma, vmf->address, vmf->pte);
 		pte_unmap_unlock(vmf->pte, vmf->ptl);
+		if (need_luf_flush(vmf))
+			luf_flush();
 		return VM_FAULT_NOPAGE;
 	}
 	wp_page_reuse(vmf, folio);
@@ -4469,6 +4489,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
 	vm_fault_t ret = 0;
 	int nr_pages = 1;
 	pte_t entry;
+	bool luf = false;
 
 	/* File mapping without ->vm_ops ? */
 	if (vma->vm_flags & VM_SHARED)
@@ -4492,6 +4513,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
 			goto unlock;
 		if (vmf_pte_changed(vmf)) {
 			update_mmu_tlb(vma, vmf->address, vmf->pte);
+			luf = need_luf_flush(vmf);
 			goto unlock;
 		}
 		ret = check_stable_address_space(vma->vm_mm);
@@ -4536,9 +4558,11 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
 		goto release;
 	if (nr_pages == 1 && vmf_pte_changed(vmf)) {
 		update_mmu_tlb(vma, addr, vmf->pte);
+		luf = need_luf_flush(vmf);
 		goto release;
 	} else if (nr_pages > 1 && !pte_range_none(vmf->pte, nr_pages)) {
 		update_mmu_tlb_range(vma, addr, vmf->pte, nr_pages);
+		luf = need_luf_flush(vmf);
 		goto release;
 	}
 
@@ -4570,6 +4594,8 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
 unlock:
 	if (vmf->pte)
 		pte_unmap_unlock(vmf->pte, vmf->ptl);
+	if (luf)
+		luf_flush();
 	return ret;
 release:
 	folio_put(folio);
@@ -4796,6 +4822,7 @@ vm_fault_t finish_fault(struct vm_fault *vmf)
 	vm_fault_t ret;
 	bool is_cow = (vmf->flags & FAULT_FLAG_WRITE) &&
 		      !(vma->vm_flags & VM_SHARED);
+	bool luf = false;
 
 	/* Did we COW the page? */
 	if (is_cow)
@@ -4841,10 +4868,14 @@ vm_fault_t finish_fault(struct vm_fault *vmf)
 		ret = 0;
 	} else {
 		update_mmu_tlb(vma, vmf->address, vmf->pte);
+		luf = need_luf_flush(vmf);
 		ret = VM_FAULT_NOPAGE;
 	}
 
 	pte_unmap_unlock(vmf->pte, vmf->ptl);
+
+	if (luf)
+		luf_flush();
 	return ret;
 }
 
@@ -5397,6 +5428,7 @@ static vm_fault_t wp_huge_pud(struct vm_fault *vmf, pud_t orig_pud)
 static vm_fault_t handle_pte_fault(struct vm_fault *vmf)
 {
 	pte_t entry;
+	bool luf = false;
 
 	if (unlikely(pmd_none(*vmf->pmd))) {
 		/*
@@ -5440,6 +5472,7 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf)
 	entry = vmf->orig_pte;
 	if (unlikely(!pte_same(ptep_get(vmf->pte), entry))) {
 		update_mmu_tlb(vmf->vma, vmf->address, vmf->pte);
+		luf = need_luf_flush(vmf);
 		goto unlock;
 	}
 	if (vmf->flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) {
@@ -5469,6 +5502,8 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf)
 	}
 unlock:
 	pte_unmap_unlock(vmf->pte, vmf->ptl);
+	if (luf)
+		luf_flush();
 	return 0;
 }
 
mm/page_alloc.c +13 -4
index c9acb4da91e0..4007c9757c3f 100644
@@ -1048,7 +1048,7 @@ void kernel_init_pages(struct page *page, int numpages)
 }
 
 __always_inline bool free_pages_prepare(struct page *page,
-			unsigned int order)
+			unsigned int order, unsigned short int *ugen)
 {
 	int bad = 0;
 	bool skip_kasan_poison = should_skip_kasan_poison(page);
@@ -1062,6 +1062,15 @@ __always_inline bool free_pages_prepare(struct page *page,
 	 */
 	set_page_private(page, 0);
 
+	/*
+	 * The contents of the pages will be updated for some reasons.
+	 * So we should give up luf.
+	 */
+	if ((!skip_kasan_poison || init) && ugen && *ugen) {
+		check_luf_flush(*ugen);
+		*ugen = 0;
+	}
+
 	trace_mm_page_free(page, order);
 	kmsan_free_page(page, order);
 
@@ -1236,7 +1245,7 @@ static void __free_pages_ok(struct page *page, unsigned int order,
 	unsigned long pfn = page_to_pfn(page);
 	struct zone *zone = page_zone(page);
 
-	if (!free_pages_prepare(page, order))
+	if (!free_pages_prepare(page, order, NULL))
 		return;
 
 	free_one_page(zone, page, pfn, order, fpi_flags, 0);
@@ -2664,7 +2673,7 @@ void free_unref_page(struct page *page, unsigned int order,
 		return;
 	}
 
-	if (!free_pages_prepare(page, order))
+	if (!free_pages_prepare(page, order, &ugen))
 		return;
 
 	/*
@@ -2712,7 +2721,7 @@ void free_unref_folios(struct folio_batch *folios, unsigned short int ugen)
 		unsigned int order = folio_order(folio);
 
 		folio_undo_large_rmappable(folio);
-		if (!free_pages_prepare(&folio->page, order))
+		if (!free_pages_prepare(&folio->page, order, &ugen))
 			continue;
 		/*
 		 * Free orders not handled on the PCP directly to the
mm/rmap.c +309 -6

373 lines not on this page · raw message

Signature · 1 line
2.17.1
#10 lore raw

Re: [PATCH v11 09/12] mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped

On 5/31/24 02:19, Byungchul Park wrote:
...

Quoted 15 lines
include/linux/fs.h +6 -0
index 0283cf366c2a..03683bf66031 100644
@@ -2872,6 +2872,12 @@ static inline void file_end_write(struct file *file)
 	if (!S_ISREG(file_inode(file)->i_mode))
 		return;
 	sb_end_write(file_inode(file)->i_sb);
+
+	/*
+	 * XXX: If needed, can be optimized by avoiding luf_flush() if
+	 * the address space of the file has never been involved by luf.
+	 */
+	luf_flush();
 }

...

Quoted 4 lines
include/linux/fs.h +4 -0
+void luf_flush(void)
+{
+	unsigned long flags;
+	unsigned short int ugen;

include/linux/fs.h

Quoted 10 lines
include/linux/fs.h +10 -0
+
+	/*
+	 * Obtain the latest ugen number.
+	 */
+	spin_lock_irqsave(&luf_lock, flags);
+	ugen = luf_gen;
+	spin_unlock_irqrestore(&luf_lock, flags);
+
+	check_luf_flush(ugen);
+}

Am I reading this right? There's now an unconditional global spinlock acquired in the sys_write() path? How can this possibly scale?

So, yeah, I think an optimization is absolutely needed. But, on a more fundamental level, I just don't believe these patches are being tested. Even a simple microbenchmark should show a pretty nasty regression on any decently large system:

Quoted 1 line

https://github.com/antonblanchard/will-it-scale/blob/master/tests/write1.c

Second, I was just pointing out sys_write() as an example of how the page cache could change. Couldn't a separate, read/write mmap() of the file do the same thing and *not* go through sb_end_write()?

So:

	fd = open("foo");
	ptr1 = mmap(fd, PROT_READ);
	ptr2 = mmap(fd, PROT_READ|PROT_WRITE);

	foo = *ptr1; // populate the page cache
	... page cache page is reclaimed and LUF'd
	*ptr2 = bar; // new page cache page is allocated and written to

	printk("*ptr1: %d\n", *ptr1);

Doesn't the printk() see stale data?

I think tglx would call all of this "tinkering". The approach to this series is to "fix" narrow, specific cases that reviewers point out, make it compile, then send it out again, hoping someone will apply it.

So, for me, until the approach to this series changes: NAK, for x86. Andrew, please don't take this series. Or, if you do, please drop the patch enabling it on x86.

I also have the feeling our VFS friends won't take kindly to having random luf_foo() hooks in their hot paths, optimized or not. I don't see any of them on cc.

#11 lore raw

Re: [PATCH v11 09/12] mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped

Quoted 19 lines · Dave Hansen

On 5/31/24 02:19, Byungchul Park wrote:
..

Quoted 15 lines
include/linux/fs.h +6 -0
index 0283cf366c2a..03683bf66031 100644
@@ -2872,6 +2872,12 @@ static inline void file_end_write(struct file *file)
      if (!S_ISREG(file_inode(file)->i_mode))
              return;
      sb_end_write(file_inode(file)->i_sb);
+
+     /*
+      * XXX: If needed, can be optimized by avoiding luf_flush() if
+      * the address space of the file has never been involved by luf.
+      */
+     luf_flush();
 }

..

Quoted 16 lines
Quoted 4 lines
include/linux/fs.h +4 -0
+void luf_flush(void)
+{
+     unsigned long flags;
+     unsigned short int ugen;

include/linux/fs.h

Quoted 10 lines
include/linux/fs.h +10 -0
+
+     /*
+      * Obtain the latest ugen number.
+      */
+     spin_lock_irqsave(&luf_lock, flags);
+     ugen = luf_gen;
+     spin_unlock_irqrestore(&luf_lock, flags);
+
+     check_luf_flush(ugen);
+}

Am I reading this right? There's now an unconditional global spinlock

It looked *too much* to split the lock to several locks as rcu does until version 11. However, this code introduced in v11 looks problematic.

Quoted 1 line

acquired in the sys_write() path? How can this possibly scale?

I should find a better way.

Quoted 13 lines

So, yeah, I think an optimization is absolutely needed. But, on a more fundamental level, I just don't believe these patches are being tested. Even a simple microbenchmark should show a pretty nasty regression on any decently large system:

Quoted 1 line

https://github.com/antonblanchard/will-it-scale/blob/master/tests/write1.c

Second, I was just pointing out sys_write() as an example of how the page cache could change. Couldn't a separate, read/write mmap() of the file do the same thing and *not* go through sb_end_write()?

So:

Quoted 7 lines

fd = open("foo");
ptr1 = mmap(fd, PROT_READ);
ptr2 = mmap(fd, PROT_READ|PROT_WRITE);

foo = *ptr1; // populate the page cache
... page cache page is reclaimed and LUF'd
*ptr2 = bar; // new page cache page is allocated and written to

I think this part would work but I'm not convinced. I will check again.

Quoted 7 lines
        printk("*ptr1: %d\n", *ptr1);

Doesn't the printk() see stale data?

I think tglx would call all of this "tinkering". The approach to this series is to "fix" narrow, specific cases that reviewers point out, make it compile, then send it out again, hoping someone will apply it.

Sorry for not perfect work and bothering you but you know what? I can see what is happening in this community too. Of course, I bet you would post better quality mm patches from the 1st version than me but might not in other subsystems.

Quoted 1 line

So, for me, until the approach to this series changes: NAK, for x86.

I understand why you got mad and feel sorry but I couldn't expect the regression you mentioned above. And I admit the patches have had problems I couldn't find in advance until you, Hildenbrand and Ying. I will do better.

Quoted 2 lines

Andrew, please don't take this series. Or, if you do, please drop the patch enabling it on x86.

I don't want to ask to merge either, if there are still issues.

Quoted 1 line

I also have the feeling our VFS friends won't take kindly to having

That is also what I thought it was. What should I do then? I don't believe you do not agree with the concept itself. Thing is the current version is not good enough. I will do my best by doing what I can do.

Quoted 2 lines

random luf_foo() hooks in their hot paths, optimized or not. I don't see any of them on cc.

Yes. I should've cc'd them. I will.

Byungchul

#12 lore raw

Re: [PATCH v11 09/12] mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped

On 5/31/24 11:04, Byungchul Park wrote:
...

Quoted 3 lines

I don't believe you do not agree with the concept itself. Thing is the current version is not good enough. I will do my best by doing what I can do.

More performance is good. I agree with that.

But it has to be weighed against the risk and the complexity. The more I look at this approach, the more I think this is not a good trade off. There's a lot of risk and a lot of complexity and we haven't seen the full complexity picture. The gaps are being fixed by adding complexity in new subsystems (the VFS in this case).

There are going to be winners and losers, and this version for example makes file writes lose performance.

Just to be crystal clear: I disagree with the concept of leaving stale TLB entries in place in an attempt to gain performance.

#13 lore raw

Re: [PATCH v11 09/12] mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped

Quoted 14 lines · Dave Hansen

On 5/31/24 11:04, Byungchul Park wrote:
...

Quoted 3 lines

I don't believe you do not agree with the concept itself. Thing is the current version is not good enough. I will do my best by doing what I can do.

More performance is good. I agree with that.

But it has to be weighed against the risk and the complexity. The more I look at this approach, the more I think this is not a good trade off. There's a lot of risk and a lot of complexity and we haven't seen the full complexity picture. The gaps are being fixed by adding complexity in new subsystems (the VFS in this case).

Quoted 5 lines

There are going to be winners and losers, and this version for example makes file writes lose performance.

Just to be crystal clear: I disagree with the concept of leaving stale TLB entries in place in an attempt to gain performance.

FWIW, I agree with Dave. This feels insanely dangerous and I don't think you're paranoid enough about things that can go wrong.

#14 lore raw

Re: [PATCH v11 09/12] mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped

In reply to Dave Hansen
Quoted 7 lines · Dave Hansen

On 5/31/24 11:04, Byungchul Park wrote:
...

Quoted 3 lines

I don't believe you do not agree with the concept itself. Thing is the current version is not good enough. I will do my best by doing what I can do.

Quoted 5 lines

More performance is good. I agree with that.

But it has to be weighed against the risk and the complexity. The more I look at this approach, the more I think this is not a good trade off. There's a lot of risk and a lot of complexity and we haven't seen the

All the complexity comes from the fact that I can't use a new space in struct page - that can make the design even lockless.

I agree that keeping things simple is the best but I don't think all the existing fields in struct page are the result of trying to make things simple that you love. Some of them are more complicated.

I'd like to find a better way together instead of yelling "it's unworthy cuz it's too complicated and there's too little space in mm world to accommodate new things".

However, for the issues already discussed, I will think about it more before the next spin.

Byungchul

Quoted 9 lines

full complexity picture. The gaps are being fixed by adding complexity in new subsystems (the VFS in this case).

There are going to be winners and losers, and this version for example makes file writes lose performance.

Just to be crystal clear: I disagree with the concept of leaving stale TLB entries in place in an attempt to gain performance.

#15 lore raw

Re: [PATCH v11 09/12] mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped

In reply to Dave Hansen
Quoted 14 lines · Dave Hansen

On 5/31/24 11:04, Byungchul Park wrote:
...

Quoted 3 lines

I don't believe you do not agree with the concept itself. Thing is the current version is not good enough. I will do my best by doing what I can do.

More performance is good. I agree with that.

But it has to be weighed against the risk and the complexity. The more I look at this approach, the more I think this is not a good trade off. There's a lot of risk and a lot of complexity and we haven't seen the full complexity picture. The gaps are being fixed by adding complexity in new subsystems (the VFS in this case).

Quoted 5 lines

There are going to be winners and losers, and this version for example makes file writes lose performance.

Just to be crystal clear: I disagree with the concept of leaving stale TLB entries in place in an attempt to gain performance.

There is the inherent problem that a CPU reading from such (unmapped but not flushed yet) memory will not get a page fault, which I think is the most controversial part here (besides interaction with other deferred TLB flushing, and how this glues into the buddy).

What we used to do so far was limiting the timeframe where that could happen, under well-controlled circumstances. On the common unmap/zap path, we perform the batched TLB flush before any page faults / VMA changes would have be possible and munmap() would have returned with "succeess". Now that time frame could be significantly longer.

So in current code, at the point in time where we would process a page fault, mmap()/munmap()/... the TLB would have been flushed already.

To "mimic" the old behavior, we'd essentially have to force any page faults/mmap/whatsoever to perform the deferred flush such that the CPU will see the "reality" again. Not sure how that could be done in a *consistent* way (check whenever we take the mmap/vma lock etc ...) and if there would still be a performance win.

Signature · 3 lines
Cheers,

David / dhildenb
author reviewer
#16 lore raw

Re: [PATCH v11 09/12] mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped

Quoted 32 lines · David Hildenbrand
Quoted 14 lines · Dave Hansen

On 5/31/24 11:04, Byungchul Park wrote:
...

Quoted 3 lines

I don't believe you do not agree with the concept itself. Thing is the current version is not good enough. I will do my best by doing what I can do.

More performance is good. I agree with that.

But it has to be weighed against the risk and the complexity. The more I look at this approach, the more I think this is not a good trade off. There's a lot of risk and a lot of complexity and we haven't seen the full complexity picture. The gaps are being fixed by adding complexity in new subsystems (the VFS in this case).

Quoted 5 lines

There are going to be winners and losers, and this version for example makes file writes lose performance.

Just to be crystal clear: I disagree with the concept of leaving stale TLB entries in place in an attempt to gain performance.

There is the inherent problem that a CPU reading from such (unmapped but not flushed yet) memory will not get a page fault, which I think is the most controversial part here (besides interaction with other deferred TLB flushing, and how this glues into the buddy).

What we used to do so far was limiting the timeframe where that could happen, under well-controlled circumstances. On the common unmap/zap path, we perform the batched TLB flush before any page faults / VMA changes would have be possible and munmap() would have returned with "succeess". Now that time frame could be significantly longer.

Quoted 6 lines

So in current code, at the point in time where we would process a page fault, mmap()/munmap()/... the TLB would have been flushed already.

To "mimic" the old behavior, we'd essentially have to force any page faults/mmap/whatsoever to perform the deferred flush such that the CPU will see the "reality" again. Not sure how that could be done in a *consistent*

In luf's point of view, the points where the deferred flush should be performed are simply:

1. when changing the vma maps, that might be luf'ed.
2. when updating data of the pages, that might be luf'ed.

All we need to do is to indentify the points:

1. when changing the vma maps, that might be luf'ed.

a) mmap and munmap e.i. fault handler or unmap_region().
b) permission to writable e.i. mprotect or fault handler.
c) what I'm missing.

2. when updating data of the pages, that might be luf'ed.

a) updating files through vfs e.g. file_end_write().
b) updating files through writable maps e.i. 1-a) or 1-b).
c) what I'm missing.

Some of them are already performing necessary tlb flush and the others are not. luf has to handle the others, that I've been focusing on. Of course, there might be what I'm missing tho.

Worth noting again, luf is working only on *migration* and *reclaim* currently. Thing is when to stop the pending initiated from migration or reclaim by luf.

Byungchul

Quoted 7 lines

way (check whenever we take the mmap/vma lock etc ...) and if there would still be a performance win.

Signature · 3 lines
Cheers,

David / dhildenb
#17 lore raw

Re: [PATCH v11 09/12] mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped

On 6/3/24 02:35, Byungchul Park wrote:
...> In luf's point of view, the points where the deferred flush should be

Quoted 4 lines

performed are simply:

1. when changing the vma maps, that might be luf'ed.
2. when updating data of the pages, that might be luf'ed.

It's simple, but the devil is in the details as always.

Quoted 7 lines

All we need to do is to indentify the points:

1. when changing the vma maps, that might be luf'ed.

a) mmap and munmap e.i. fault handler or unmap_region().
b) permission to writable e.i. mprotect or fault handler.
c) what I'm missing.

I'd say it even more generally: anything that installs a PTE which is inconsistent with the original PTE. That, of course, includes writes. But it also includes crazy things that we do like uprobes. Take a look at __replace_page().

I think the page_vma_mapped_walk() checks plus the ptl keep LUF at bay there. But it needs some really thorough review.

But the bigger concern is that, if there was a problem, I can't think of a systematic way to find it.

Quoted 5 lines

2. when updating data of the pages, that might be luf'ed.

a) updating files through vfs e.g. file_end_write().
b) updating files through writable maps e.i. 1-a) or 1-b).
c) what I'm missing.

Filesystems or block devices that change content without a "write" from the local system. Network filesystems and block devices come to mind. I honestly don't know what all the rules are around these, but they could certainly be troublesome.

There appear to be some interactions for NFS between file locking and page cache flushing.

But, stepping back ...

I'd honestly be a lot more comfortable if there was even a debugging LUF mode that enforced a rule that said:

1. A LUF'd PTE can't be rewritten until after a luf_flush() occurs
2. A LUF'd page's position in the page cache can't be replaced until after a luf_flush()

or *some* other independent set of rules that can tell us when something goes wrong. That uprobes code, for instance, seems like it will work. But I can also imagine writing it ten other ways where it would break when combined with LUF.

#18 lore raw

Re: [PATCH v11 09/12] mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped

Quoted 22 lines · Dave Hansen

On 6/3/24 02:35, Byungchul Park wrote:
...> In luf's point of view, the points where the deferred flush should be

Quoted 4 lines

performed are simply:

1. when changing the vma maps, that might be luf'ed.
2. when updating data of the pages, that might be luf'ed.

It's simple, but the devil is in the details as always.

Quoted 7 lines

All we need to do is to indentify the points:

1. when changing the vma maps, that might be luf'ed.

a) mmap and munmap e.i. fault handler or unmap_region().
b) permission to writable e.i. mprotect or fault handler.
c) what I'm missing.

I'd say it even more generally: anything that installs a PTE which is inconsistent with the original PTE. That, of course, includes writes. But it also includes crazy things that we do like uprobes. Take a look at __replace_page().

Quoted 5 lines

I think the page_vma_mapped_walk() checks plus the ptl keep LUF at bay there. But it needs some really thorough review.

But the bigger concern is that, if there was a problem, I can't think of a systematic way to find it.

Fully agreed!

Quoted 15 lines
Quoted 6 lines

2. when updating data of the pages, that might be luf'ed.

a) updating files through vfs e.g. file_end_write().
b) updating files through writable maps e.i. 1-a) or 1-b).
c) what I'm missing.

Filesystems or block devices that change content without a "write" from the local system. Network filesystems and block devices come to mind. I honestly don't know what all the rules are around these, but they could certainly be troublesome.

There appear to be some interactions for NFS between file locking and page cache flushing.

Quoted 6 lines

But, stepping back ...

I'd honestly be a lot more comfortable if there was even a debugging LUF mode that enforced a rule that said:

1. A LUF'd PTE can't be rewritten until after a luf_flush() occurs

I was playing with the idea of using a PTE marker. Then it's clear for munmap/mremap/page faults that there is an outstanding flush required. the alternative might be a VMA flag, but that's harder to actually enforce an invariant.

Quoted 2 lines

2. A LUF'd page's position in the page cache can't be replaced until after a luf_flush()

That's the most tricky bit. I think these are the VFS concerns like

1) Page migration/reclaim ends up freeing the old page. TLB not flushed.
2) write() to the new page / write from other process to the new page
3) CPU reads stale content from old page

PTE markers can't handle that.

Signature · 3 lines
Cheers,

David / dhildenb
#19 lore raw

Re: [PATCH v11 09/12] mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped

On 6/3/24 09:05, David Hildenbrand wrote:
...

Quoted 10 lines
Quoted 2 lines

   2. A LUF'd page's position in the page cache can't be replaced until after a luf_flush()

That's the most tricky bit. I think these are the VFS concerns like

1) Page migration/reclaim ends up freeing the old page. TLB not flushed.
2) write() to the new page / write from other process to the new page
3) CPU reads stale content from old page

PTE markers can't handle that.

Yeah, we'd need some equivalent of a PTE marker, but for the page cache.
Presumably some xa_value() that means a reader has to go do a
luf_flush() before going any farther.

That would actually have a chance at fixing two issues: One where a new page cache insertion is attempted. The other where someone goes to look in the page cache and takes some action _because_ it is empty (I think NFS is doing some of this for file locks).

LUF is also pretty fundamentally built on the idea that files can't change without LUF being aware. That model seems to work decently for normal old filesystems on normal old local block devices. I'm worried about NFS, and I don't know how seriously folks take FUSE, but it obviously can't work well for FUSE.

#20 lore raw

Re: [PATCH v11 09/12] mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped

Quoted 3 lines · Dave Hansen

Yeah, we'd need some equivalent of a PTE marker, but for the page cache.
Presumably some xa_value() that means a reader has to go do a
luf_flush() before going any farther.

I can allocate one for that. We've got something like 1000 currently unused values which can't be mistaken for anything else.

Quoted 10 lines

That would actually have a chance at fixing two issues: One where a new page cache insertion is attempted. The other where someone goes to look in the page cache and takes some action _because_ it is empty (I think NFS is doing some of this for file locks).

LUF is also pretty fundamentally built on the idea that files can't change without LUF being aware. That model seems to work decently for normal old filesystems on normal old local block devices. I'm worried about NFS, and I don't know how seriously folks take FUSE, but it obviously can't work well for FUSE.

I'm more concerned with:

- page goes back to buddy
- page is allocated to slab
- application reads through stale TLB entry and sees kernel memory

Or did that scenario get resolved?

#21 lore raw

Re: [PATCH v11 09/12] mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped

Quoted 7 lines · Matthew Wilcox
Quoted 3 lines · Dave Hansen

Yeah, we'd need some equivalent of a PTE marker, but for the page cache.
Presumably some xa_value() that means a reader has to go do a
luf_flush() before going any farther.

I can allocate one for that. We've got something like 1000 currently unused values which can't be mistaken for anything else.

I'm curious when to set that, though.

While migrating/reclaiming, when unmapping the folio from the page tables, the folio is still valid in the page cache. So at the point in time of unmapping from one process, we cannot simply replace the folio in the page cache by some other value -- I think.

Maybe it's all easier than I think.

Signature · 3 lines
Cheers,

David / dhildenb
#22 lore raw

Re: [PATCH v11 09/12] mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped

Quoted 9 lines · David Hildenbrand
Quoted 7 lines · Matthew Wilcox
Quoted 3 lines · Dave Hansen

Yeah, we'd need some equivalent of a PTE marker, but for the page cache.
Presumably some xa_value() that means a reader has to go do a
luf_flush() before going any farther.

I can allocate one for that. We've got something like 1000 currently
unused values which can't be mistaken for anything else.

Quoted 8 lines

I'm curious when to set that, though.

While migrating/reclaiming, when unmapping the folio from the page tables, the folio is still valid in the page cache. So at the point in time of unmapping from one process, we cannot simply replace the folio in the page cache by some other value -- I think.

Maybe it's all easier than I think.

IIUC, we need to held folio lock before replacing the folio in the page cache. In page_cache_delete(), folio_test_locked() is checked. And, we will lock the folio before writing to it via write syscall. So, it's safe to defer TLB flushing until we unlock the folio.

--
Best Regards,
Huang, Ying

author reviewer
#23 lore raw

Re: [PATCH v11 09/12] mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped

In reply to Matthew Wilcox
Quoted 19 lines · Matthew Wilcox
Quoted 3 lines · Dave Hansen

Yeah, we'd need some equivalent of a PTE marker, but for the page cache.
Presumably some xa_value() that means a reader has to go do a
luf_flush() before going any farther.

I can allocate one for that. We've got something like 1000 currently unused values which can't be mistaken for anything else.

Quoted 10 lines

That would actually have a chance at fixing two issues: One where a new page cache insertion is attempted. The other where someone goes to look in the page cache and takes some action _because_ it is empty (I think NFS is doing some of this for file locks).

LUF is also pretty fundamentally built on the idea that files can't change without LUF being aware. That model seems to work decently for normal old filesystems on normal old local block devices. I'm worried about NFS, and I don't know how seriously folks take FUSE, but it obviously can't work well for FUSE.

Quoted 4 lines

I'm more concerned with:

- page goes back to buddy
- page is allocated to slab

At this point, tlb flush needed will be performed in prep_new_page().

Quoted 1 line

- application reads through stale TLB entry and sees kernel memory

No worry for this case.

Byungchul

Quoted 2 lines

Or did that scenario get resolved?

reviewer
#24 lore raw

Re: [PATCH v11 09/12] mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped

Quoted 20 lines · Byungchul Park
Quoted 19 lines · Matthew Wilcox
Quoted 3 lines · Dave Hansen

Yeah, we'd need some equivalent of a PTE marker, but for the page cache.
Presumably some xa_value() that means a reader has to go do a
luf_flush() before going any farther.

I can allocate one for that. We've got something like 1000 currently unused values which can't be mistaken for anything else.

Quoted 10 lines

That would actually have a chance at fixing two issues: One where a new page cache insertion is attempted. The other where someone goes to look in the page cache and takes some action _because_ it is empty (I think NFS is doing some of this for file locks).

LUF is also pretty fundamentally built on the idea that files can't change without LUF being aware. That model seems to work decently for normal old filesystems on normal old local block devices. I'm worried about NFS, and I don't know how seriously folks take FUSE, but it obviously can't work well for FUSE.

Quoted 6 lines
Quoted 4 lines

I'm more concerned with:

- page goes back to buddy
- page is allocated to slab

At this point, tlb flush needed will be performed in prep_new_page().

But that does mean that an unaware caller would get an additional overhead of the flushing, right? I think it would be just a matter of time before somebody can turn that into a side channel attack, not to mention unexpected latencies introduced.

Signature · 2 lines
Michal Hocko
SUSE Labs
author reviewer
#25 lore raw

Re: [PATCH v11 09/12] mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped

Quoted 23 lines · Michal Hocko
Quoted 22 lines · Byungchul Park
Quoted 21 lines · Matthew Wilcox
Quoted 3 lines · Dave Hansen

Yeah, we'd need some equivalent of a PTE marker, but for the page cache.
Presumably some xa_value() that means a reader has to go do a
luf_flush() before going any farther.

I can allocate one for that. We've got something like 1000 currently unused values which can't be mistaken for anything else.

Quoted 10 lines

That would actually have a chance at fixing two issues: One where a new page cache insertion is attempted. The other where someone goes to look in the page cache and takes some action _because_ it is empty (I think NFS is doing some of this for file locks).

LUF is also pretty fundamentally built on the idea that files can't change without LUF being aware. That model seems to work decently for normal old filesystems on normal old local block devices. I'm worried about NFS, and I don't know how seriously folks take FUSE, but it obviously can't work well for FUSE.

I'm more concerned with:

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

- page goes back to buddy
- page is allocated to slab

At this point, tlb flush needed will be performed in prep_new_page().

But that does mean that an unaware caller would get an additional overhead of the flushing, right? I think it would be just a matter of

pcp for locality is already a better source of side channel attack. FYI, tlb flush gets barely performed only if pending tlb flush exists.

Quoted 2 lines

time before somebody can turn that into a side channel attack, not to mention unexpected latencies introduced.

Nope. The pending tlb flush performed in prep_new_page() is the one that would've done already with the vanilla kernel. It's not additional tlb flushes but it's subset of all the skipped ones.

It's worth noting all the existing mm reclaim mechaisms have already introduced worse unexpected latencies.

Byungchul

Quoted 3 lines
Signature · 2 lines
Michal Hocko
SUSE Labs
reviewer
#26 lore raw

Re: [PATCH v11 09/12] mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped

Quoted 27 lines · Byungchul Park
Quoted 26 lines · Michal Hocko
Quoted 25 lines · Byungchul Park
Quoted 23 lines · Matthew Wilcox
Quoted 3 lines · Dave Hansen

Yeah, we'd need some equivalent of a PTE marker, but for the page cache.
Presumably some xa_value() that means a reader has to go do a
luf_flush() before going any farther.

I can allocate one for that. We've got something like 1000 currently unused values which can't be mistaken for anything else.

Quoted 10 lines

That would actually have a chance at fixing two issues: One where a new page cache insertion is attempted. The other where someone goes to look in the page cache and takes some action _because_ it is empty (I think NFS is doing some of this for file locks).

LUF is also pretty fundamentally built on the idea that files can't change without LUF being aware. That model seems to work decently for normal old filesystems on normal old local block devices. I'm worried about NFS, and I don't know how seriously folks take FUSE, but it obviously can't work well for FUSE.

I'm more concerned with:

- page goes back to buddy
- page is allocated to slab

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

At this point, tlb flush needed will be performed in prep_new_page().

But that does mean that an unaware caller would get an additional overhead of the flushing, right? I think it would be just a matter of

pcp for locality is already a better source of side channel attack. FYI, tlb flush gets barely performed only if pending tlb flush exists.

Right but rare and hard to predict latencies are much worse than consistent once.

Quoted 6 lines
Quoted 2 lines

time before somebody can turn that into a side channel attack, not to mention unexpected latencies introduced.

Nope. The pending tlb flush performed in prep_new_page() is the one that would've done already with the vanilla kernel. It's not additional tlb flushes but it's subset of all the skipped ones.

But those skipped once could have happened in a completely different context (e.g. a different process or even a diffrent security domain), right?

Quoted 2 lines

It's worth noting all the existing mm reclaim mechaisms have already introduced worse unexpected latencies.

Right, but a reclaim, especially direct reclaim, are expected to be slow. It is much different to see spike latencies on system with a lot of memory.

Signature · 2 lines
Michal Hocko
SUSE Labs
author reviewer
#27 lore raw

Re: [PATCH v11 09/12] mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped

Quoted 30 lines · Michal Hocko
Quoted 29 lines · Byungchul Park
Quoted 28 lines · Michal Hocko
Quoted 26 lines · Byungchul Park
Quoted 19 lines · Matthew Wilcox
Quoted 3 lines · Dave Hansen

Yeah, we'd need some equivalent of a PTE marker, but for the page cache.
Presumably some xa_value() that means a reader has to go do a
luf_flush() before going any farther.

I can allocate one for that. We've got something like 1000 currently unused values which can't be mistaken for anything else.

Quoted 10 lines

That would actually have a chance at fixing two issues: One where a new page cache insertion is attempted. The other where someone goes to look in the page cache and takes some action _because_ it is empty (I think NFS is doing some of this for file locks).

LUF is also pretty fundamentally built on the idea that files can't change without LUF being aware. That model seems to work decently for normal old filesystems on normal old local block devices. I'm worried about NFS, and I don't know how seriously folks take FUSE, but it obviously can't work well for FUSE.

Quoted 4 lines

I'm more concerned with:

- page goes back to buddy
- page is allocated to slab

At this point, tlb flush needed will be performed in prep_new_page().

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

But that does mean that an unaware caller would get an additional overhead of the flushing, right? I think it would be just a matter of

pcp for locality is already a better source of side channel attack. FYI, tlb flush gets barely performed only if pending tlb flush exists.

Right but rare and hard to predict latencies are much worse than consistent once.

No doubt it'd be the best if we keep things consistent as long as possible. How consistent *we require* it would be, matters. Lemme know criteria for that if any. I will check it.

Quoted 10 lines
Quoted 6 lines
Quoted 2 lines

time before somebody can turn that into a side channel attack, not to mention unexpected latencies introduced.

Nope. The pending tlb flush performed in prep_new_page() is the one that would've done already with the vanilla kernel. It's not additional tlb flushes but it's subset of all the skipped ones.

But those skipped once could have happened in a completely different context (e.g. a different process or even a diffrent security domain), right?

Right.

Quoted 6 lines
Quoted 2 lines

It's worth noting all the existing mm reclaim mechaisms have already introduced worse unexpected latencies.

Right, but a reclaim, especially direct reclaim, are expected to be slow. It is much different to see spike latencies on system with a lot of memory.

Talking about rt system? In rt system, the system should prevent its memory from being reclaimed, IMHO, since these will add unexpected latencies.

Reclaim and migrations alreay introduce unexpected latencies themselves. Why does only latencies by luf matter? I'm asking to understand what you mean, in order to fix luf if any.

   vanilla
   -------
   alloc_page() {
      ...
      preempted by kswapd or direct reclaim {
         ...
         reclaim
            unmap file pages
   	 tlb shootdown
         ...
         migration
            unmap pages
   	 tlb shootdown
         ...
      }
      ...
      interrupted by tlb shootdown from other CPUs {
         ...
      }
      ...
      prep_new_page() {
         ...
      }
   }
   
   with luf
   --------
   alloc_page() {
      ...
      preempted by kswapd or direct reclaim {
         ...
         reclaim
            unmap file pages
   	 (skip tlb shootdown)
         ...
         migration
            unmap pages
   	 (skip tlb shootdown)
         ...
      }
      ...
      interrupted by tlb shootdown from other CPUs {
         ...
      }
      ...
      prep_new_page() {
         ...
         /*
          * This can be tlb shootdown skipped in this context or others.
          */
         tlb shootdown with much smaller cpumask
         ...
      }
   }

I really want to understand why only latentcies introduced in luf matter? Why does not latencies already introduced in vanilla matter?

Byungchul

Quoted 3 lines
Signature · 2 lines
Michal Hocko
SUSE Labs
author reviewer
#28 lore raw

Re: [PATCH v11 09/12] mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped

In reply to Dave Hansen
Quoted 8 lines · Dave Hansen

On 6/3/24 02:35, Byungchul Park wrote:
...> In luf's point of view, the points where the deferred flush should be

Quoted 4 lines

performed are simply:

1. when changing the vma maps, that might be luf'ed.
2. when updating data of the pages, that might be luf'ed.

It's simple, but the devil is in the details as always.

Agree with that.

Quoted 19 lines
Quoted 7 lines

All we need to do is to indentify the points:

1. when changing the vma maps, that might be luf'ed.

a) mmap and munmap e.i. fault handler or unmap_region().
b) permission to writable e.i. mprotect or fault handler.
c) what I'm missing.

I'd say it even more generally: anything that installs a PTE which is inconsistent with the original PTE. That, of course, includes writes. But it also includes crazy things that we do like uprobes. Take a look at __replace_page().

I think the page_vma_mapped_walk() checks plus the ptl keep LUF at bay there. But it needs some really thorough review.

But the bigger concern is that, if there was a problem, I can't think of a systematic way to find it.

Quoted 8 lines
Quoted 5 lines

2. when updating data of the pages, that might be luf'ed.

a) updating files through vfs e.g. file_end_write().
b) updating files through writable maps e.i. 1-a) or 1-b).
c) what I'm missing.

Filesystems or block devices that change content without a "write" from the local system. Network filesystems and block devices come to mind.

AFAIK, every network filesystem eventully "updates" its connected local filesystem. It could be still handled at the point where updating the local file system.

Quoted 9 lines

I honestly don't know what all the rules are around these, but they could certainly be troublesome.

There appear to be some interactions for NFS between file locking and page cache flushing.

But, stepping back ...

I'd honestly be a lot more comfortable if there was even a debugging LUF

I'd better provide a method for better debugging. Lemme know whatever it is we need.

Quoted 1 line

mode that enforced a rule that said:

Why "debugging mode"? The following rules should be enforced always.

Quoted 1 line

1. A LUF'd PTE can't be rewritten until after a luf_flush() occurs

"luf_flush() should be followed when.." is more correct because "luf_flush() -> another luf -> the pte gets rewritten" can happen. So it should be "the pte gets rewritten -> another luf by any chance -> luf_flush()", that is still safe.

Quoted 2 lines

2. A LUF'd page's position in the page cache can't be replaced until after a luf_flush()

"luf_flush() should be followed when.." is more correct too.

These two rules are exactly same as what I described but more specific. I like your way to describe the rules.

Byungchul

Quoted 4 lines

or *some* other independent set of rules that can tell us when something goes wrong. That uprobes code, for instance, seems like it will work. But I can also imagine writing it ten other ways where it would break when combined with LUF.

author reviewer
#29 lore raw

Re: [PATCH v11 09/12] mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped

Quoted 50 lines · Byungchul Park
Quoted 8 lines · Dave Hansen

On 6/3/24 02:35, Byungchul Park wrote:
...> In luf's point of view, the points where the deferred flush should be

Quoted 4 lines

performed are simply:

1. when changing the vma maps, that might be luf'ed.
2. when updating data of the pages, that might be luf'ed.

It's simple, but the devil is in the details as always.

Agree with that.

Quoted 19 lines
Quoted 7 lines

All we need to do is to indentify the points:

1. when changing the vma maps, that might be luf'ed.

a) mmap and munmap e.i. fault handler or unmap_region().
b) permission to writable e.i. mprotect or fault handler.
c) what I'm missing.

I'd say it even more generally: anything that installs a PTE which is inconsistent with the original PTE. That, of course, includes writes. But it also includes crazy things that we do like uprobes. Take a look at __replace_page().

I think the page_vma_mapped_walk() checks plus the ptl keep LUF at bay there. But it needs some really thorough review.

But the bigger concern is that, if there was a problem, I can't think of a systematic way to find it.

Quoted 8 lines
Quoted 5 lines

2. when updating data of the pages, that might be luf'ed.

a) updating files through vfs e.g. file_end_write().
b) updating files through writable maps e.i. 1-a) or 1-b).
c) what I'm missing.

Filesystems or block devices that change content without a "write" from the local system. Network filesystems and block devices come to mind.

AFAIK, every network filesystem eventully "updates" its connected local filesystem. It could be still handled at the point where updating the local file system.

Quoted 6 lines

I honestly don't know what all the rules are around these, but they could certainly be troublesome.

There appear to be some interactions for NFS between file locking and page cache flushing.

Quoted 8 lines
Quoted 3 lines

But, stepping back ...

I'd honestly be a lot more comfortable if there was even a debugging LUF

I'd better provide a method for better debugging. Lemme know whatever it is we need.

Quoted 1 line

mode that enforced a rule that said:

Do you means a debugging mode that can WARN or inform the situation that we don't want? If yes, sure. Now that I get this, I will re-read all you guys' talk.

Byungchul

Quoted 23 lines

Why "debugging mode"? The following rules should be enforced always.

Quoted 1 line

1. A LUF'd PTE can't be rewritten until after a luf_flush() occurs

"luf_flush() should be followed when.." is more correct because "luf_flush() -> another luf -> the pte gets rewritten" can happen. So it should be "the pte gets rewritten -> another luf by any chance -> luf_flush()", that is still safe.

Quoted 2 lines

2. A LUF'd page's position in the page cache can't be replaced until after a luf_flush()

"luf_flush() should be followed when.." is more correct too.

These two rules are exactly same as what I described but more specific. I like your way to describe the rules.

Byungchul

Quoted 4 lines

or *some* other independent set of rules that can tell us when something goes wrong. That uprobes code, for instance, seems like it will work. But I can also imagine writing it ten other ways where it would break when combined with LUF.

#30 lore raw

Re: [PATCH v11 09/12] mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped

Quoted 55 lines · Byungchul Park
Quoted 54 lines · Byungchul Park
Quoted 8 lines · Dave Hansen

On 6/3/24 02:35, Byungchul Park wrote:
...> In luf's point of view, the points where the deferred flush should be

Quoted 4 lines

performed are simply:

1. when changing the vma maps, that might be luf'ed.
2. when updating data of the pages, that might be luf'ed.

It's simple, but the devil is in the details as always.

Agree with that.

Quoted 19 lines
Quoted 7 lines

All we need to do is to indentify the points:

1. when changing the vma maps, that might be luf'ed.

a) mmap and munmap e.i. fault handler or unmap_region().
b) permission to writable e.i. mprotect or fault handler.
c) what I'm missing.

I'd say it even more generally: anything that installs a PTE which is inconsistent with the original PTE. That, of course, includes writes. But it also includes crazy things that we do like uprobes. Take a look at __replace_page().

I think the page_vma_mapped_walk() checks plus the ptl keep LUF at bay there. But it needs some really thorough review.

But the bigger concern is that, if there was a problem, I can't think of a systematic way to find it.

Quoted 8 lines
Quoted 5 lines

2. when updating data of the pages, that might be luf'ed.

a) updating files through vfs e.g. file_end_write().
b) updating files through writable maps e.i. 1-a) or 1-b).
c) what I'm missing.

Filesystems or block devices that change content without a "write" from the local system. Network filesystems and block devices come to mind.

AFAIK, every network filesystem eventully "updates" its connected local filesystem. It could be still handled at the point where updating the local file system.

Quoted 9 lines

I honestly don't know what all the rules are around these, but they could certainly be troublesome.

There appear to be some interactions for NFS between file locking and page cache flushing.

But, stepping back ...

I'd honestly be a lot more comfortable if there was even a debugging LUF

Quoted 9 lines
Quoted 4 lines

I'd better provide a method for better debugging. Lemme know whatever it is we need.

Quoted 1 line

mode that enforced a rule that said:

Do you means a debugging mode that can WARN or inform the situation that we don't want? If yes, sure. Now that I get this, I will re-read all you guys' talk.

In my opinion either for debugging, or for actually enforcing it at runtime. Whatever the cost of that would be needs to be determined.

Signature · 3 lines
Cheers,

David / dhildenb
author reviewer
#31 lore raw

Re: [PATCH v11 09/12] mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped

In reply to Byungchul Park
Quoted 37 lines · Byungchul Park
Quoted 8 lines · Dave Hansen

On 6/3/24 02:35, Byungchul Park wrote:
...> In luf's point of view, the points where the deferred flush should be

Quoted 4 lines

performed are simply:

1. when changing the vma maps, that might be luf'ed.
2. when updating data of the pages, that might be luf'ed.

It's simple, but the devil is in the details as always.

Agree with that.

Quoted 25 lines
Quoted 7 lines

All we need to do is to indentify the points:

1. when changing the vma maps, that might be luf'ed.

a) mmap and munmap e.i. fault handler or unmap_region().
b) permission to writable e.i. mprotect or fault handler.
c) what I'm missing.

I'd say it even more generally: anything that installs a PTE which is inconsistent with the original PTE. That, of course, includes writes. But it also includes crazy things that we do like uprobes. Take a look at __replace_page().

I think the page_vma_mapped_walk() checks plus the ptl keep LUF at bay there. But it needs some really thorough review.

But the bigger concern is that, if there was a problem, I can't think of a systematic way to find it.

Quoted 5 lines

2. when updating data of the pages, that might be luf'ed.

a) updating files through vfs e.g. file_end_write().
b) updating files through writable maps e.i. 1-a) or 1-b).
c) what I'm missing.

Quoted 6 lines
Quoted 2 lines

Filesystems or block devices that change content without a "write" from the local system. Network filesystems and block devices come to mind.

AFAIK, every network filesystem eventully "updates" its connected local filesystem. It could be still handled at the point where updating the local file system.

To cover client of network file systems and any using page cache, struct address_space_operations's write_end() call sites seem to be the best place to handle that. At the same time, of course, I should limit the target of luf to 'folio_mapping(folio) != NULL' for file pages.

Byungchul

Quoted 38 lines
Quoted 9 lines

I honestly don't know what all the rules are around these, but they could certainly be troublesome.

There appear to be some interactions for NFS between file locking and page cache flushing.

But, stepping back ...

I'd honestly be a lot more comfortable if there was even a debugging LUF

I'd better provide a method for better debugging. Lemme know whatever it is we need.

Quoted 1 line

mode that enforced a rule that said:

Why "debugging mode"? The following rules should be enforced always.

Quoted 1 line

1. A LUF'd PTE can't be rewritten until after a luf_flush() occurs

"luf_flush() should be followed when.." is more correct because "luf_flush() -> another luf -> the pte gets rewritten" can happen. So it should be "the pte gets rewritten -> another luf by any chance -> luf_flush()", that is still safe.

Quoted 2 lines

2. A LUF'd page's position in the page cache can't be replaced until after a luf_flush()

"luf_flush() should be followed when.." is more correct too.

These two rules are exactly same as what I described but more specific. I like your way to describe the rules.

Byungchul

Quoted 4 lines

or *some* other independent set of rules that can tell us when something goes wrong. That uprobes code, for instance, seems like it will work. But I can also imagine writing it ten other ways where it would break when combined with LUF.

author reviewer
#32 lore raw

Re: [PATCH v11 09/12] mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped

In reply to Dave Hansen
Quoted 42 lines · Dave Hansen

On 6/3/24 02:35, Byungchul Park wrote:
...> In luf's point of view, the points where the deferred flush should be

Quoted 4 lines

performed are simply:

1. when changing the vma maps, that might be luf'ed.
2. when updating data of the pages, that might be luf'ed.

It's simple, but the devil is in the details as always.

Quoted 7 lines

All we need to do is to indentify the points:

1. when changing the vma maps, that might be luf'ed.

a) mmap and munmap e.i. fault handler or unmap_region().
b) permission to writable e.i. mprotect or fault handler.
c) what I'm missing.

I'd say it even more generally: anything that installs a PTE which is inconsistent with the original PTE. That, of course, includes writes. But it also includes crazy things that we do like uprobes. Take a look at __replace_page().

I think the page_vma_mapped_walk() checks plus the ptl keep LUF at bay there. But it needs some really thorough review.

But the bigger concern is that, if there was a problem, I can't think of a systematic way to find it.

Quoted 5 lines

2. when updating data of the pages, that might be luf'ed.

a) updating files through vfs e.g. file_end_write().
b) updating files through writable maps e.i. 1-a) or 1-b).
c) what I'm missing.

Filesystems or block devices that change content without a "write" from the local system. Network filesystems and block devices come to mind. I honestly don't know what all the rules are around these, but they could certainly be troublesome.

There appear to be some interactions for NFS between file locking and page cache flushing.

Quoted 8 lines

But, stepping back ...

I'd honestly be a lot more comfortable if there was even a debugging LUF mode that enforced a rule that said:

1. A LUF'd PTE can't be rewritten until after a luf_flush() occurs
2. A LUF'd page's position in the page cache can't be replaced until after a luf_flush()

I'm thinking a debug mode doing the following *pseudo* code - check the logic only since the grammer might be wrong:

0-a) Introduce new fields in page_ext:

	#ifdef LUF_DEBUG
	struct list_head __percpu luf_node;
	#endif

0-b) Introduce new fields in struct address_space:

	#ifdef LUF_DEBUG
	struct list_head __percpu luf_node;
	#endif

0-c) Introduce new fields in struct task_struct:

	#ifdef LUF_DEBUG
	cpumask_t luf_pending_cpus;
	#endif

0-d) Define percpu list_head to link luf'd folios:

	#ifdef LUF_DEBUG
	DEFINE_PER_CPU(struct list_head, luf_folios);
	DEFINE_PER_CPU(struct list_head, luf_address_spaces);
	#endif

1) When skipping tlb flush in reclaim or migration for a folio:

	#ifdef LUF_DEBUG
	ext = get_page_ext_for_luf_debug(folio);
	as = folio_mapping(folio);

	for_each_cpu(cpu, skip_cpus) {
		list_add(per_cpu_ptr(ext->luf_node, cpu),
			 per_cpu_ptr(luf_folios, cpu));
		if (as)
			list_add(per_cpu_ptr(as->luf_node, cpu),
				 per_cpu_ptr(luf_address_spaces, cpu));
	}
	put_page_ext(ext);
	#endif

2) When performing tlb flush in try_to_unmap_flush():

      Remind luf only works on unmapping during reclaim and migration.

	#ifdef LUF_DEBUG
	for_each_cpu(cpu, now_flushing_cpus) {
		for_each_node_safe(folio, per_cpu_ptr(luf_folios)) {
			ext = get_page_ext_for_luf_debug(folio);
			list_del_init(per_cpu_ptr(ext->luf_node, cpu))
			put_page_ext(ext);
		}

		for_each_node_safe(as, per_cpu_ptr(luf_address_spaces))
			list_del_init(per_cpu_ptr(as->luf_node, cpu))

		cpumask_clear_cpu(cpu, current->luf_pending_cpus);
	}
	#endif

3) In pte_mkwrite():

	#ifdef LUF_DEBUG
	ext = get_page_ext_for_luf_debug(folio);

	for_each_cpu(cpu, online_cpus)
		if (!list_empty(per_cpu_ptr(ext->luf_node, cpu)))
			cpumask_set_cpu(cpu, current->luf_pending_cpus);
	put_page_ext(ext);
	#endif

4) On returning to user:

	#ifdef LUF_DEBUG
	WARN_ON(!cpumask_empty(current->luf_pending_cpus));
	#endif

5) On right after every a_ops->write_end() call:

	#ifdef LUF_DEBUG
	as = get_address_space_to_write_to();
	for_each_cpu(cpu, online_cpus)
		if (!list_empty(per_cpu_ptr(as->luf_node, cpu)))
			cpumask_set_cpu(cpu, current->luf_pending_cpus);
	#endif

	luf_flush_or_its_optimized_version();

	#ifdef LUF_DEBUG
	WARN_ON(!cpumask_empty(current->luf_pending_cpus));
	#endif

I will implement the debug mode this way with all serialized. Do you think it works for what we want?

Byungchul

Quoted 4 lines

or *some* other independent set of rules that can tell us when something goes wrong. That uprobes code, for instance, seems like it will work. But I can also imagine writing it ten other ways where it would break when combined with LUF.

author reviewer
#33 lore raw

PATCH v11 10/12 mm: separate move/undo parts from migrate_pages_batch()

In reply to Byungchul Park

Functionally, no change. This is a preparation for luf mechanism that requires to use separated folio lists for its own handling during migration. Refactored migrate_pages_batch() so as to separate move/undo parts from migrate_pages_batch().

---
 mm/migrate.c | 134 +++++++++++++++++++++++++++++++--------------------
 1 file changed, 83 insertions(+), 51 deletions(-)
mm/migrate.c +83 -51
index e04b451c4289..6c22a1402923 100644
@@ -1584,6 +1584,81 @@ static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio,
 	return nr_failed;
 }
 
+static void migrate_folios_move(struct list_head *src_folios,
+		struct list_head *dst_folios,
+		free_folio_t put_new_folio, unsigned long private,
+		enum migrate_mode mode, int reason,
+		struct list_head *ret_folios,
+		struct migrate_pages_stats *stats,
+		int *retry, int *thp_retry, int *nr_failed,
+		int *nr_retry_pages)
+{
+	struct folio *folio, *folio2, *dst, *dst2;
+	bool is_thp;
+	int nr_pages;
+	int rc;
+
+	dst = list_first_entry(dst_folios, struct folio, lru);
+	dst2 = list_next_entry(dst, lru);
+	list_for_each_entry_safe(folio, folio2, src_folios, lru) {
+		is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio);
+		nr_pages = folio_nr_pages(folio);
+
+		cond_resched();
+
+		rc = migrate_folio_move(put_new_folio, private,
+				folio, dst, mode,
+				reason, ret_folios);
+		/*
+		 * The rules are:
+		 *	Success: folio will be freed
+		 *	-EAGAIN: stay on the unmap_folios list
+		 *	Other errno: put on ret_folios list
+		 */
+		switch(rc) {
+		case -EAGAIN:
+			*retry += 1;
+			*thp_retry += is_thp;
+			*nr_retry_pages += nr_pages;
+			break;
+		case MIGRATEPAGE_SUCCESS:
+			stats->nr_succeeded += nr_pages;
+			stats->nr_thp_succeeded += is_thp;
+			break;
+		default:
+			*nr_failed += 1;
+			stats->nr_thp_failed += is_thp;
+			stats->nr_failed_pages += nr_pages;
+			break;
+		}
+		dst = dst2;
+		dst2 = list_next_entry(dst, lru);
+	}
+}
+
+static void migrate_folios_undo(struct list_head *src_folios,
+		struct list_head *dst_folios,
+		free_folio_t put_new_folio, unsigned long private,
+		struct list_head *ret_folios)
+{
+	struct folio *folio, *folio2, *dst, *dst2;
+
+	dst = list_first_entry(dst_folios, struct folio, lru);
+	dst2 = list_next_entry(dst, lru);
+	list_for_each_entry_safe(folio, folio2, src_folios, lru) {
+		int old_page_state = 0;
+		struct anon_vma *anon_vma = NULL;
+
+		__migrate_folio_extract(dst, &old_page_state, &anon_vma);
+		migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED,
+				anon_vma, true, ret_folios);
+		list_del(&dst->lru);
+		migrate_folio_undo_dst(dst, true, put_new_folio, private);
+		dst = dst2;
+		dst2 = list_next_entry(dst, lru);
+	}
+}
+
 /*
  * migrate_pages_batch() first unmaps folios in the from list as many as
  * possible, then move the unmapped folios.
@@ -1606,7 +1681,7 @@ static int migrate_pages_batch(struct list_head *from,
 	int pass = 0;
 	bool is_thp = false;
 	bool is_large = false;
-	struct folio *folio, *folio2, *dst = NULL, *dst2;
+	struct folio *folio, *folio2, *dst = NULL;
 	int rc, rc_saved = 0, nr_pages;
 	LIST_HEAD(unmap_folios);
 	LIST_HEAD(dst_folios);
@@ -1765,42 +1840,11 @@ static int migrate_pages_batch(struct list_head *from,
 		thp_retry = 0;
 		nr_retry_pages = 0;
 
-		dst = list_first_entry(&dst_folios, struct folio, lru);
-		dst2 = list_next_entry(dst, lru);
-		list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) {
-			is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio);
-			nr_pages = folio_nr_pages(folio);
-
-			cond_resched();
-
-			rc = migrate_folio_move(put_new_folio, private,
-						folio, dst, mode,
-						reason, ret_folios);
-			/*
-			 * The rules are:
-			 *	Success: folio will be freed
-			 *	-EAGAIN: stay on the unmap_folios list
-			 *	Other errno: put on ret_folios list
-			 */
-			switch(rc) {
-			case -EAGAIN:
-				retry++;
-				thp_retry += is_thp;
-				nr_retry_pages += nr_pages;
-				break;
-			case MIGRATEPAGE_SUCCESS:
-				stats->nr_succeeded += nr_pages;
-				stats->nr_thp_succeeded += is_thp;
-				break;
-			default:
-				nr_failed++;
-				stats->nr_thp_failed += is_thp;
-				stats->nr_failed_pages += nr_pages;
-				break;
-			}
-			dst = dst2;
-			dst2 = list_next_entry(dst, lru);
-		}
+		/* Move the unmapped folios */
+		migrate_folios_move(&unmap_folios, &dst_folios,
+				put_new_folio, private, mode, reason,
+				ret_folios, stats, &retry, &thp_retry,
+				&nr_failed, &nr_retry_pages);
 	}
 	nr_failed += retry;
 	stats->nr_thp_failed += thp_retry;
@@ -1809,20 +1853,8 @@ static int migrate_pages_batch(struct list_head *from,
 	rc = rc_saved ? : nr_failed;
 out:
 	/* Cleanup remaining folios */
-	dst = list_first_entry(&dst_folios, struct folio, lru);
-	dst2 = list_next_entry(dst, lru);
-	list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) {
-		int old_page_state = 0;
-		struct anon_vma *anon_vma = NULL;
-
-		__migrate_folio_extract(dst, &old_page_state, &anon_vma);
-		migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED,
-				       anon_vma, true, ret_folios);
-		list_del(&dst->lru);
-		migrate_folio_undo_dst(dst, true, put_new_folio, private);
-		dst = dst2;
-		dst2 = list_next_entry(dst, lru);
-	}
+	migrate_folios_undo(&unmap_folios, &dst_folios,
+			put_new_folio, private, ret_folios);
 
 	return rc;
 }
Signature · 1 line
2.17.1
author reviewer
#34 lore raw

PATCH v11 11/12 mm, migrate: apply luf mechanism to unmapping during migration

In reply to Byungchul Park

A new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read only and were unmapped, since the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.

Applied the mechanism to unmapping during migration.

---
 include/linux/rmap.h |  2 +-
 mm/migrate.c         | 56 ++++++++++++++++++++++++++++++++------------
 mm/rmap.c            |  9 ++++---
 3 files changed, 48 insertions(+), 19 deletions(-)
include/linux/rmap.h +1 -1
index bb53e5920b88..6aca569e342b 100644
@@ -660,7 +660,7 @@ static inline int folio_try_share_anon_rmap_pmd(struct folio *folio,
 int folio_referenced(struct folio *, int is_locked,
 			struct mem_cgroup *memcg, unsigned long *vm_flags);
 
-void try_to_migrate(struct folio *folio, enum ttu_flags flags);
+bool try_to_migrate(struct folio *folio, enum ttu_flags flags);
 void try_to_unmap(struct folio *, enum ttu_flags flags);
 
 int make_device_exclusive_range(struct mm_struct *mm, unsigned long start,
mm/migrate.c +41 -15

155 lines not on this page · raw message

mm/rmap.c +6 -3
index 459d4d1631f0..b8b977278a1b 100644
@@ -2635,8 +2635,9 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,
  *
  * Tries to remove all the page table entries which are mapping this folio and
  * replace them with special swap entries. Caller must hold the folio lock.
+ * Return true if all the mappings are read-only, otherwise false.
  */
-void try_to_migrate(struct folio *folio, enum ttu_flags flags)
+bool try_to_migrate(struct folio *folio, enum ttu_flags flags)
 {
 	struct rmap_walk_control rwc = {
 		.rmap_one = try_to_migrate_one,
@@ -2655,11 +2656,11 @@ void try_to_migrate(struct folio *folio, enum ttu_flags flags)
 	 */
 	if (WARN_ON_ONCE(flags & ~(TTU_RMAP_LOCKED | TTU_SPLIT_HUGE_PMD |
 					TTU_SYNC | TTU_BATCH_FLUSH)))
-		return;
+		return false;
 
 	if (folio_is_zone_device(folio) &&
 	    (!folio_is_device_private(folio) && !folio_is_device_coherent(folio)))
-		return;
+		return false;
 
 	/*
 	 * During exec, a temporary VMA is setup and later moved.
@@ -2684,6 +2685,8 @@ void try_to_migrate(struct folio *folio, enum ttu_flags flags)
 		fold_ubc(tlb_ubc_luf, tlb_ubc_ro);
 	else
 		fold_ubc(tlb_ubc, tlb_ubc_ro);
+
+	return can_luf;
 }
 
 #ifdef CONFIG_DEVICE_PRIVATE
Signature · 1 line
2.17.1
author reviewer
#35 lore raw

PATCH v11 12/12 mm, vmscan: apply luf mechanism to unmapping during folio reclaim

In reply to Byungchul Park

A new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read only and were unmapped, since the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.

Applied the mechanism to unmapping during folio reclaim.

---
 include/linux/rmap.h |  5 +++--
 mm/rmap.c            |  5 ++++-
 mm/vmscan.c          | 21 ++++++++++++++++++++-
 3 files changed, 27 insertions(+), 4 deletions(-)
include/linux/rmap.h +3 -2
index 6aca569e342b..9f3e66239f0a 100644
@@ -661,7 +661,7 @@ int folio_referenced(struct folio *, int is_locked,
 			struct mem_cgroup *memcg, unsigned long *vm_flags);
 
 bool try_to_migrate(struct folio *folio, enum ttu_flags flags);
-void try_to_unmap(struct folio *, enum ttu_flags flags);
+bool try_to_unmap(struct folio *, enum ttu_flags flags);
 
 int make_device_exclusive_range(struct mm_struct *mm, unsigned long start,
 				unsigned long end, struct page **pages,
@@ -770,8 +770,9 @@ static inline int folio_referenced(struct folio *folio, int is_locked,
 	return 0;
 }
 
-static inline void try_to_unmap(struct folio *folio, enum ttu_flags flags)
+static inline bool try_to_unmap(struct folio *folio, enum ttu_flags flags)
 {
+	return false;
 }
 
 static inline int folio_mkclean(struct folio *folio)
mm/rmap.c +4 -1
index b8b977278a1b..6f90c2adc4ae 100644
@@ -2272,10 +2272,11 @@ static int folio_not_mapped(struct folio *folio)
  * Tries to remove all the page table entries which are mapping this
  * folio.  It is the caller's responsibility to check if the folio is
  * still mapped if needed (use TTU_SYNC to prevent accounting races).
+ * Return true if all the mappings are read-only, otherwise false.
  *
  * Context: Caller must hold the folio lock.
  */
-void try_to_unmap(struct folio *folio, enum ttu_flags flags)
+bool try_to_unmap(struct folio *folio, enum ttu_flags flags)
 {
 	struct rmap_walk_control rwc = {
 		.rmap_one = try_to_unmap_one,
@@ -2300,6 +2301,8 @@ void try_to_unmap(struct folio *folio, enum ttu_flags flags)
 		fold_ubc(tlb_ubc_luf, tlb_ubc_ro);
 	else
 		fold_ubc(tlb_ubc, tlb_ubc_ro);
+
+	return can_luf;
 }
 
 /*
mm/vmscan.c +20 -1
index 15efe6f0edce..d52a6e605183 100644
@@ -1034,14 +1034,17 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 		struct reclaim_stat *stat, bool ignore_references)
 {
 	struct folio_batch free_folios;
+	struct folio_batch free_folios_luf;
 	LIST_HEAD(ret_folios);
 	LIST_HEAD(demote_folios);
 	unsigned int nr_reclaimed = 0;
 	unsigned int pgactivate = 0;
 	bool do_demote_pass;
 	struct swap_iocb *plug = NULL;
+	unsigned short int ugen;
 
 	folio_batch_init(&free_folios);
+	folio_batch_init(&free_folios_luf);
 	memset(stat, 0, sizeof(*stat));
 	cond_resched();
 	do_demote_pass = can_demote(pgdat->node_id, sc);
@@ -1053,6 +1056,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 		enum folio_references references = FOLIOREF_RECLAIM;
 		bool dirty, writeback;
 		unsigned int nr_pages;
+		bool can_luf = false;
 
 		cond_resched();
 
@@ -1295,7 +1299,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 			if (folio_test_large(folio) && list_empty(&folio->_deferred_list))
 				flags |= TTU_SYNC;
 
-			try_to_unmap(folio, flags);
+			can_luf = try_to_unmap(folio, flags);
 			if (folio_mapped(folio)) {
 				stat->nr_unmap_fail += nr_pages;
 				if (!was_swapbacked &&
@@ -1458,6 +1462,18 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 		nr_reclaimed += nr_pages;
 
 		folio_undo_large_rmappable(folio);
+
+		if (can_luf) {
+			if (folio_batch_add(&free_folios_luf, folio) == 0) {
+				mem_cgroup_uncharge_folios(&free_folios_luf);
+				ugen = try_to_unmap_luf();
+				if (!ugen)
+					try_to_unmap_flush();
+				free_unref_folios(&free_folios_luf, ugen);
+			}
+			continue;
+		}
+
 		if (folio_batch_add(&free_folios, folio) == 0) {
 			mem_cgroup_uncharge_folios(&free_folios);
 			try_to_unmap_flush();
@@ -1527,8 +1543,11 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 	pgactivate = stat->nr_activate[0] + stat->nr_activate[1];
 
 	mem_cgroup_uncharge_folios(&free_folios);
+	mem_cgroup_uncharge_folios(&free_folios_luf);
+	ugen = try_to_unmap_luf();
 	try_to_unmap_flush();
 	free_unref_folios(&free_folios, 0);
+	free_unref_folios(&free_folios_luf, ugen);
 
 	list_splice(&ret_folios, folio_list);
 	count_vm_events(PGACTIVATE, pgactivate);
Signature · 1 line
2.17.1