LUF(Lazy Unmap Flush) reducing tlb numbers over 90%
Subsystems (27)
Byungchul Park
reviewerv12 cover LUF(Lazy Unmap Flush) reducing tlb numbers over 90%
To check luf's stability, I ran a heavy LLM inference workload consuming 210GiB over 7 days on a machine with 140GiB memory, and decided it's stable enough.
I'm posting the latest version so that anyone can try luf mechanism if wanted by any chance. However, I tagged RFC again because there are still issues that should be resolved to merge to mainline:
1. Even though system wide total cpu time for TLB shootdown is reduced over 95%, page allocation paths should take additional cpu time shifted from page reclaim to perform TLB shootdown.
2. We need luf debug feature to detect when luf goes wrong by any chance. I implemented just a draft version that checks the sanity on mkwrite(), kmap(), and so on. I need to gather better ideas to improve the debug feature.
---
Hi everyone,
While I'm working with a tiered memory system e.g. CXL memory, I have been facing migration overhead esp. tlb shootdown on promotion or demotion between different tiers. Yeah.. most tlb shootdowns on migration through hinting fault can be avoided thanks to Huang Ying's work, commit 4d4b6d66db ("mm,unmap: avoid flushing tlb in batch if PTE is inaccessible").
However, it's only for migration through hinting fault. I thought it'd be much better if we have a general mechanism to reduce all the tlb numbers that we can apply to any unmap code, that we normally believe tlb flush should be followed.
I'm suggesting a new mechanism, LUF(Lazy Unmap Flush), that defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read-only and were unmapped, as long as the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.
tlb flush can be defered when folios get unmapped as long as it guarantees to perform tlb flush needed, before the folios actually become used, of course, only if all the corresponding ptes don't have write permission. Otherwise, the system will get messed up.
To achieve that, for the folios that map only to non-writable tlb entries, prevent tlb flush during unmapping but perform it just before the folios actually become used, out of buddy or pcp.
However, we should cancel the pending by LUF and perform the deferred TLB flush right away when:
1. a writable pte is newly set through fault handler
2. a file is updated
3. kasan needs poisoning on free
4. the kernel wants to init pages on free
No matter what type of workload is used for performance evaluation, the result would be positive thanks to the unconditional reduction of tlb flushes, tlb misses and interrupts. For the test, I picked up one of the most popular and heavy workload, llama.cpp that is a LLM(Large Language Model) inference engine.
The result would depend on memory latency and how often reclaim runs, which implies tlb miss overhead and how many times unmapping happens. In my system, the result shows:
1. tlb shootdown interrupts are reduced about 97%.
2. The test program runtime is reduced about 4.5%.
The test environment and the result is like:
Machine: bare metal, x86_64, Intel(R) Xeon(R) Gold 6430
CPU: 1 socket 64 core with hyper thread on Numa: 2 nodes (64 CPUs DRAM 42GB, no CPUs CXL expander 98GB) Config: swap off, numa balancing tiering on, demotion enabled
The test set:
llama.cpp/main -m $(70G_model1) -p "who are you?" -s 1 -t 15 -n 20 & llama.cpp/main -m $(70G_model2) -p "who are you?" -s 1 -t 15 -n 20 & llama.cpp/main -m $(70G_model3) -p "who are you?" -s 1 -t 15 -n 20 & wait
where -t: nr of threads, -s: seed used to make the runtime stable, -n: nr of tokens that determines the runtime, -p: prompt to ask, -m: LLM model to use.
Run the test set 5 times successively with caches dropped every run via 'echo 3 > /proc/sys/vm/drop_caches'. Each inference prints its runtime at the end of each.
1. Runtime from the output of llama.cpp:
BEFORE
------
llama_print_timings: total time = 883450.54 ms / 24 tokens llama_print_timings: total time = 861665.91 ms / 24 tokens llama_print_timings: total time = 898079.02 ms / 24 tokens llama_print_timings: total time = 879897.69 ms / 24 tokens llama_print_timings: total time = 892360.75 ms / 24 tokens llama_print_timings: total time = 884587.85 ms / 24 tokens llama_print_timings: total time = 861023.19 ms / 24 tokens llama_print_timings: total time = 900022.18 ms / 24 tokens llama_print_timings: total time = 878771.88 ms / 24 tokens llama_print_timings: total time = 889027.98 ms / 24 tokens llama_print_timings: total time = 880783.90 ms / 24 tokens llama_print_timings: total time = 856475.29 ms / 24 tokens llama_print_timings: total time = 896842.21 ms / 24 tokens llama_print_timings: total time = 878883.53 ms / 24 tokens llama_print_timings: total time = 890122.10 ms / 24 tokens
AFTER
-----
llama_print_timings: total time = 871060.86 ms / 24 tokens llama_print_timings: total time = 825609.53 ms / 24 tokens llama_print_timings: total time = 836854.81 ms / 24 tokens llama_print_timings: total time = 843147.99 ms / 24 tokens llama_print_timings: total time = 831426.65 ms / 24 tokens llama_print_timings: total time = 873939.23 ms / 24 tokens llama_print_timings: total time = 826127.69 ms / 24 tokens llama_print_timings: total time = 835489.26 ms / 24 tokens llama_print_timings: total time = 842589.62 ms / 24 tokens llama_print_timings: total time = 833700.66 ms / 24 tokens llama_print_timings: total time = 875996.19 ms / 24 tokens llama_print_timings: total time = 826401.73 ms / 24 tokens llama_print_timings: total time = 839341.28 ms / 24 tokens llama_print_timings: total time = 841075.10 ms / 24 tokens llama_print_timings: total time = 835136.41 ms / 24 tokens
2. tlb shootdowns from 'cat /proc/interrupts':
BEFORE
------
TLB:
80911532 93691786 100296251 111062810 109769109 109862429 108968588 119175230 115779676 118377498 119325266 120300143 124514185 116697222 121068466 118031913 122660681 117494403 121819907 116960596 120936335 117217061 118630217 122322724 119595577 111693298 119232201 120030377 115334687 113179982 118808254 116353592 140987367 137095516 131724276 139742240 136501150 130428761 127585535 132483981 133430250 133756207 131786710 126365824 129812539 133850040 131742690 125142213 128572830 132234350 131945922 128417707 133355434 129972846 126331823 134050849 133991626 121129038 124637283 132830916 126875507 122322440 125776487 124340278 TLB shootdowns
AFTER
-----
TLB:
2121206 2615108 2983494 2911950 3055086 3092672
3204894 3346082 3286744 3307310 3357296 3315940
3428034 3112596 3143325 3185551 3186493 3322314
3330523 3339663 3156064 3272070 3296309 3198962
3332662 3315870 3234467 3353240 3281234 3300666
3345452 3173097 4009196 3932215 3898735 3726531
3717982 3671726 3728788 3724613 3799147 3691764
3620630 3684655 3666688 3393974 3448651 3487593
3446357 3618418 3671920 3712949 3575264 3715385
3641513 3630897 3691047 3630690 3504933 3662647
3629926 3443044 3832970 3548813 TLB shootdowns---
Changes from v11:
1. Rebase on akpm/mm.git mm-unstable(5a7056135b) as of Nov 22, 2024.
2. Very big changes have been applied including refactoring for better reuse and readability, and other features for stability and performance optimization. It's hard to list up all changes but lemme list up main changes.
3. Meta data for pending TLB shootdown is kept in struct zone instead of struct page so as to perform pending TLB shootdown when it's needed in batched manner.
4. Introduce pend_list in strcut free_area so as to keep luf pages separatedly, that need to perform pending TLB shootdown on returning from buddy.
5. Control the amount of non-luf pages in buddy, that do not need pending TLB shootdown on returning from buddy, so that any atomic contexts can allocate pages without consideration of TLB shootdown. Try the best to keep non-luf pages as much as WMARK_MIN for now.
6. Optimzation to reduce lock contention by locks introduced by luf mechanism.
7. Implement luf debug feature for detecting when luf goes wrong. It's just a draft version that need to improve much.
8. Avoid pending TLB shootdown as much as possible e.g. use non-luf pages in buddy within preempt disabled context if the number of non-luf pages in buddy is fair enough.
Changes from v10:
1. Rebase on akpm/mm.git mm-unstable as of May 28, 2024.
2. Cancel LUF on file_end_write() when updating a file. (feedbacked by Dave Hansen)
3. Cancel LUF after every update_mmu_tlb*() in fault handler if it's going to set the pte to writable.
4. Cancel LUF on freeing pages if kasan needs poisoning. (feedbacked by David Hildenbrand)
5. Cancel LUF on freeing pages if want_init_on_free(). (feedbacked by David Hildenbrand)
6. Change test iteration from 10 times to 5 times.
7. Not include perf result. (I will add it if needed.)
8. Trivial optimization.
Changes from v9:
1. Expand the candidate to apply this mechanism:
BEFORE - The souce folios at any type of migration. AFTER - Any folios that have been unmapped and freed.
2. Change the workload for test:
BEFORE - XSBench
AFTER - llama.cpp (one of the most popluar real workload)
3. Change the test environment:
BEFORE - qemu machine, too small DRAM(1GB), large remote mem AFTER - bare metal, real CXL memory, practical memory size
4. Rename the mechanism from MIGRC(Migration Read Copy) to LUF(Lazy Unmap Flush) to reflect the current version of the mechanism can be applied not only to unmap during migration but any unmap code e.g. unmap in shrink_folio_list().
5. Fix build error for riscv. (feedbacked by kernel test bot)
6. Supplement commit messages to describe what this mechanism is for, especially in the patches for arch code. (feedbacked by Thomas Gleixner)
7. Clean up some trivial things.
Changes from v8:
1. Rebase on akpm/mm.git mm-unstable as of April 18, 2024.
2. Supplement comments and commit message.
3. Change the candidate to apply migrc mechanism:
BEFORE - The source folios at demotion and promotion. AFTER - The souce folios at any type of migration.
4. Change how migrc mechanism works:
BEFORE - Reduce tlb flushes by deferring folio_free() for
source folios during demotion and promotion.
AFTER - Reduce tlb flushes by deferring tlb flush until they
actually become used, out of pcp or buddy. The current version of migrc does *not* defer calling folio_free() but let it go as it is as the same as vanilla kernel, with the folios marked kind of 'need to tlb flush'. And then handle the flush when the page exits from pcp or buddy so as to prevent changing vm stats e.g. free pages.
Changes from v7:
1. Rewrite cover letter to explain what 'migrc' mechasism is. (feedbacked by Andrew Morton)
2. Supplement the commit message of a patch 'mm: Add APIs to free a folio directly to the buddy bypassing pcp'. (feedbacked by Andrew Morton)
Changes from v6:
1. Fix build errors in case of CONFIG_ARCH_WANT_BATCHED_UNMAP_tlb_FLUSH disabled by moving migrc_flush_{start,end}() calls from arch code to try_to_unmap_flush() in mm/rmap.c.
Changes from v5:
1. Fix build errors in case of CONFIG_MIGRATION disabled or CONFIG_HWPOISON_INJECT moduled. (feedbacked by kernel test bot and Raymond Jay Golo)
2. Organize migrc code with two kconfigs, CONFIG_MIGRATION and CONFIG_ARCH_WANT_BATCHED_UNMAP_tlb_FLUSH.
Changes from v4:
1. Rebase on v6.7.
2. Fix build errors in arm64 that is doing nothing for tlb flush but has CONFIG_ARCH_WANT_BATCHED_UNMAP_tlb_FLUSH. (reported by kernel test robot)
3. Don't use any page flag. So the system would give up migrc mechanism more often but it's okay. The final improvement is good enough.
4. Instead, optimize full tlb flush(arch_tlbbatch_flush()) by avoiding redundant CPUs from tlb flush.
Changes from v3:
1. Don't use the kconfig, CONFIG_MIGRC, and remove sysctl knob, migrc_enable. (feedbacked by Nadav)
2. Remove the optimization skipping CPUs that have already performed tlb flushes needed by any reason when performing tlb flushes by migrc because I can't tell the performance difference between w/ the optimization and w/o that. (feedbacked by Nadav)
3. Minimize arch-specific code. While at it, move all the migrc declarations and inline functions from include/linux/mm.h to mm/internal.h (feedbacked by Dave Hansen, Nadav)
4. Separate a part making migrc paused when the system is in high memory pressure to another patch. (feedbacked by Nadav)
5. Rename:
a. arch_tlbbatch_clean() to arch_tlbbatch_clear(), b. tlb_ubc_nowr to tlb_ubc_ro,
c. migrc_try_flush_free_folios() to migrc_flush_free_folios(), d. migrc_stop to migrc_pause.
(feedbacked by Nadav)
6. Use ->lru list_head instead of introducing a new llist_head. (feedbacked by Nadav)
7. Use non-atomic operations of page-flag when it's safe. (feedbacked by Nadav)
8. Use stack instead of keeping a pointer of 'struct migrc_req' in struct task, which is for manipulating it locally. (feedbacked by Nadav)
9. Replace a lot of simple functions to inline functions placed in a header, mm/internal.h. (feedbacked by Nadav)
10. Add additional sufficient comments. (feedbacked by Nadav)
11. Remove a lot of wrapper functions. (feedbacked by Nadav)
Changes from RFC v2:
1. Remove additional occupation in struct page. To do that, unioned with lru field for migrc's list and added a page flag. I know page flag is a thing that we don't like to add but no choice because migrc should distinguish folios under migrc's control from others. Instead, I force migrc to be used only on 64 bit system to mitigate you guys from getting angry.
2. Remove meaningless internal object allocator that I introduced to minimize impact onto the system. However, a ton of tests showed there was no difference.
3. Stop migrc from working when the system is in high memory pressure like about to perform direct reclaim. At the condition where the swap mechanism is heavily used, I found the system suffered from regression without this control.
4. Exclude folios that pte_dirty() == true from migrc's interest so that migrc can work simpler.
5. Combine several patches that work tightly coupled to one.
6. Add sufficient comments for better review.
7. Manage migrc's request in per-node manner (from globally).
8. Add tlb miss improvement in commit message.
9. Test with more CPUs(4 -> 16) to see bigger improvement.
Changes from RFC:
1. Fix a bug triggered when a destination folio at the previous migration becomes a source folio at the next migration, before the folio gets handled properly so that the folio can play with another migration. There was inconsistency in the folio's state. Fixed it.
2. Split the patch set into more pieces so that the folks can review better. (Feedbacked by Nadav Amit)
3. Fix a wrong usage of barrier e.g. smp_mb__after_atomic(). (Feedbacked by Nadav Amit)
4. Tried to add sufficient comments to explain the patch set better. (Feedbacked by Nadav Amit)
Byungchul Park (26):
x86/tlb: add APIs manipulating tlb batch's arch data arm64/tlbflush: add APIs manipulating tlb batch's arch data riscv/tlb: add APIs manipulating tlb batch's arch data x86/tlb, riscv/tlb, mm/rmap: separate arch_tlbbatch_clear() out of
arch_tlbbatch_flush()
mm/buddy: make room for a new variable, luf_key, in struct page
mm: move should_skip_kasan_poison() to mm/internal.h
mm: introduce luf_ugen to be used as a global timestamp
mm: introduce luf_batch to be used as hash table to store luf meta
data
mm: introduce API to perform tlb shootdown on exit from page allocator
mm: introduce APIs to check if the page allocation is tlb
shootdownable
mm: deliver luf_key to pcp or buddy on free after unmapping
mm: delimit critical sections to take off pages from pcp or buddy
alloctor
mm: introduce pend_list in struct free_area to track luf'd pages mm/rmap: recognize read-only tlb entries during batched tlb flush fs, filemap: refactor to gather the scattered ->write_{begin,end}()
calls
mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get
unmapped
x86/tlb, riscv/tlb, arm64/tlbflush, mm: remove cpus from tlb shootdown
that already have been done
mm/page_alloc: retry 3 times to take pcp pages on luf check failure
mm: skip luf tlb flush for luf'd mm that already has been done mm, fs: skip tlb flushes for luf'd filemap that already has been done
mm: perform luf tlb shootdown per zone in batched manner mm/page_alloc: not allow to tlb shootdown if !preemptable() &&
non_luf_pages_ok()
mm: separate move/undo parts from migrate_pages_batch() mm/migrate: apply luf mechanism to unmapping during migration mm/vmscan: apply luf mechanism to unmapping during folio reclaim mm/luf: implement luf debug feature
arch/arm64/include/asm/tlbflush.h | 53 ++ arch/riscv/include/asm/tlbflush.h | 35 + arch/riscv/mm/tlbflush.c | 142 +++- arch/x86/include/asm/pgtable.h | 10 + arch/x86/include/asm/tlbflush.h | 31 + arch/x86/mm/pgtable.c | 10 + arch/x86/mm/tlb.c | 143 +++- drivers/gpu/drm/i915/gem/i915_gem_shmem.c | 11 +- fs/affs/file.c | 4 +- fs/buffer.c | 14 +- fs/exfat/file.c | 5 +- fs/ext4/verity.c | 5 +- fs/f2fs/super.c | 5 +- fs/f2fs/verity.c | 5 +- fs/inode.c | 1 + fs/namei.c | 5 +- include/asm-generic/tlb.h | 5 + include/linux/fs.h | 30 + include/linux/highmem-internal.h | 5 + include/linux/mm.h | 94 ++- include/linux/mm_types.h | 70 +- include/linux/mm_types_task.h | 16 + include/linux/mmzone.h | 12 + include/linux/rmap.h | 7 +- include/linux/sched.h | 24 + kernel/fork.c | 1 + kernel/power/snapshot.c | 14 + kernel/sched/core.c | 1 + kernel/vmcore_info.c | 2 + mm/compaction.c | 61 +- mm/filemap.c | 5 +- mm/highmem.c | 1 + mm/internal.h | 196 +++++- mm/memory.c | 43 ++ mm/migrate.c | 193 ++++-- mm/mm_init.c | 7 + mm/page_alloc.c | 798 +++++++++++++++++++--- mm/page_ext.c | 3 + mm/page_isolation.c | 10 +- mm/page_reporting.c | 45 +- mm/pgtable-generic.c | 2 + mm/rmap.c | 684 ++++++++++++++++++- mm/swap.c | 6 +- mm/truncate.c | 55 +- mm/vmscan.c | 55 +- mm/vmstat.c | 15 + 46 files changed, 2680 insertions(+), 259 deletions(-)
base-commit: 5a7056135bb69da2ce0a42eb8c07968c1331777b
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 1/26 x86/tlb: add APIs manipulating tlb batch's arch data
A new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read-only and were unmapped, since the contents of the folios wouldn't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.
This is a preparation for the mechanism that needs to recognize read-only tlb entries by separating tlb batch arch data into two, one is for read-only entries and the other is for writable ones, and merging those two when needed.
It also optimizes tlb shootdown by skipping CPUs that have already performed tlb flush needed since. To support it, added APIs manipulating arch data for x86.
--- arch/x86/include/asm/tlbflush.h | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
arch/x86/include/asm/tlbflush.h +24 -0
@@ -5,6 +5,7 @@#include <linux/mm_types.h> #include <linux/mmu_notifier.h> #include <linux/sched.h> +#include <linux/cpumask.h> #include <asm/processor.h> #include <asm/cpufeature.h>
@@ -293,6 +294,29 @@ static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm)extern void arch_tlbbatch_flush (struct arch_tlbflush_unmap_batch *batch); +static inline void arch_tlbbatch_clear (struct arch_tlbflush_unmap_batch *batch) +{ +cpumask_clear (&batch->cpumask); +} + +static inline void arch_tlbbatch_fold (struct arch_tlbflush_unmap_batch *bdst, +struct arch_tlbflush_unmap_batch *bsrc) +{ +cpumask_or (&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + +static inline bool arch_tlbbatch_need_fold (struct arch_tlbflush_unmap_batch *batch, +struct mm_struct *mm) +{ +return !cpumask_subset (mm_cpumask (mm), &batch->cpumask); +} + +static inline bool arch_tlbbatch_done (struct arch_tlbflush_unmap_batch *bdst, +struct arch_tlbflush_unmap_batch *bsrc) +{ +return !cpumask_andnot (&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} +static inline bool pte_flags_need_flush (unsigned long oldflags,unsigned long newflags,bool ignore_access)
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 2/26 arm64/tlbflush: add APIs manipulating tlb batch's arch data
In reply to Byungchul ParkA new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read only and were unmapped, since the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.
This is a preparation for the mechanism that requires to manipulate tlb batch's arch data. Even though arm64 does nothing for tlb things, arch with CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH should provide the APIs.
--- arch/arm64/include/asm/tlbflush.h | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
arch/arm64/include/asm/tlbflush.h +27 -0
@@ -354,6 +354,33 @@ static inline void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)dsb (ish); } +static inline void arch_tlbbatch_clear (struct arch_tlbflush_unmap_batch *batch) +{ +/* nothing to do */ +} + +static inline void arch_tlbbatch_fold (struct arch_tlbflush_unmap_batch *bdst, +struct arch_tlbflush_unmap_batch *bsrc) +{ +/* nothing to do */ +} + +static inline bool arch_tlbbatch_need_fold (struct arch_tlbflush_unmap_batch *batch, +struct mm_struct *mm) +{ +/* +* Nothing is needed in this architecture. +*/ +return false; +} + +static inline bool arch_tlbbatch_done (struct arch_tlbflush_unmap_batch *bdst, +struct arch_tlbflush_unmap_batch *bsrc) +{ +/* Kernel can consider tlb batch always has been done. */ +return true; +} +/* * This is meant to avoid soft lock-ups on large TLB flushing ranges and not * necessarily a performance improvement.
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 3/26 riscv/tlb: add APIs manipulating tlb batch's arch data
In reply to Byungchul ParkA new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read only and were unmapped, since the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.
This is a preparation for the mechanism that needs to recognize read-only tlb entries by separating tlb batch arch data into two, one is for read-only entries and the other is for writable ones, and merging those two when needed.
It also optimizes tlb shootdown by skipping CPUs that have already performed tlb flush needed since. To support it, added APIs manipulating arch data for riscv.
--- arch/riscv/include/asm/tlbflush.h | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)
arch/riscv/include/asm/tlbflush.h +28 -0
@@ -8,6 +8,7 @@#define _ASM_RISCV_TLBFLUSH_H#include <linux/mm_types.h> +#include <linux/cpumask.h> #include <asm/smp.h> #include <asm/errata_list.h>
@@ -65,6 +66,33 @@ void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch,void arch_flush_tlb_batched_pending (struct mm_struct *mm);void arch_tlbbatch_flush (struct arch_tlbflush_unmap_batch *batch); +static inline void arch_tlbbatch_clear (struct arch_tlbflush_unmap_batch *batch) +{ +cpumask_clear (&batch->cpumask); + +} + +static inline void arch_tlbbatch_fold (struct arch_tlbflush_unmap_batch *bdst, +struct arch_tlbflush_unmap_batch *bsrc) +{ +cpumask_or (&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); + +} + +static inline bool arch_tlbbatch_need_fold (struct arch_tlbflush_unmap_batch *batch, +struct mm_struct *mm) +{ +return !cpumask_subset (mm_cpumask (mm), &batch->cpumask); + +} + +static inline bool arch_tlbbatch_done (struct arch_tlbflush_unmap_batch *bdst, +struct arch_tlbflush_unmap_batch *bsrc) +{ +return !cpumask_andnot (&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); + +} +extern unsigned long tlb_flush_all_threshold;#else /* CONFIG_MMU */ #define local_flush_tlb_all () do { } while (0)
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 4/26 x86/tlb, riscv/tlb, mm/rmap: separate arch_tlbbatch_clear() out of arch_tlbbatch_flush()
In reply to Byungchul ParkA new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read only and were unmapped, since the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.
This is a preparation for the mechanism that requires to avoid redundant tlb flush by manipulating tlb batch's arch data. To achieve that, we need to separate the part clearing the tlb batch's arch data out of arch_tlbbatch_flush().
--- arch/riscv/mm/tlbflush.c | 1 - arch/x86/mm/tlb.c | 2 -- mm/rmap.c | 1 + 3 files changed, 1 insertion(+), 3 deletions(-)
arch/riscv/mm/tlbflush.c +0 -1
@@ -201,5 +201,4 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) {__flush_tlb_range (&batch->cpumask,FLUSH_TLB_NO_ASID ,0 ,FLUSH_TLB_MAX_SIZE ,PAGE_SIZE ); -cpumask_clear (&batch->cpumask); }
arch/x86/mm/tlb.c +0 -2
@@ -1262,8 +1262,6 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)local_irq_enable (); } -cpumask_clear (&batch->cpumask); -put_flush_tlb_info ();put_cpu (); }
mm/rmap.c +1 -0
@@ -648,6 +648,7 @@ void try_to_unmap_flush(void)return ;arch_tlbbatch_flush (&tlb_ubc->arch); +arch_tlbbatch_clear (&tlb_ubc->arch); tlb_ubc->flush_required = false; tlb_ubc->writable = false; }
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 5/26 mm/buddy: make room for a new variable, luf_key, in struct page
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that tracks need of tlb flush for each page residing in buddy.
Since the private field in struct page is used only to store page order in buddy, ranging from 0 to MAX_PAGE_ORDER, that can be covered with unsigned short. So splitted it into two smaller ones, order and luf_key, so that the both can be used in buddy at the same time.
--- include/linux/mm_types.h | 42 +++++++++++++++++++++++++++++++++------- mm/internal.h | 4 ++-- mm/page_alloc.c | 2 +- 3 files changed, 38 insertions(+), 10 deletions(-)
include/linux/mm_types.h +35 -7
@@ -106,13 +106,27 @@ struct page {pgoff_t index;/* Our offset within mapping. */ unsigned long share;/* share count for fsdax */ }; -/** -* @private: Mapping-private opaque data. -* Usually used for buffer_heads if PagePrivate. -* Used for swp_entry_t if swapcache flag set. -* Indicates order in the buddy system if PageBuddy. -*/ -unsigned long private; +union { +/** +* @private: Mapping-private opaque data. +* Usually used for buffer_heads if PagePrivate. +* Used for swp_entry_t if swapcache flag set. +* Indicates order in the buddy system if PageBuddy. +*/ +unsigned long private; +struct { +/* +* Indicates order in the buddy system if PageBuddy. +*/ +unsigned short order; + +/* +* For tracking need of tlb flush, +* by luf(lazy unmap flush). +*/ +unsigned short luf_key; + }; + }; };struct {/* page_pool used by netstack */ /**
@@ -537,6 +551,20 @@ static inline void set_page_private(struct page *page, unsigned long private) page->private = private; } +#define page_buddy_order (page) ((page)->order) + +static inline void set_page_buddy_order (struct page *page,unsigned int order) +{ + page->order = (unsigned short )order; +} + +#define page_luf_key (page) ((page)->luf_key) + +static inline void set_page_luf_key (struct page *page,unsigned short luf_key) +{ + page->luf_key = luf_key; +} +static inline void *folio_get_private (struct folio *folio) {return folio->private;
mm/internal.h +2 -2
@@ -541,7 +541,7 @@ struct alloc_context {static inline unsigned int buddy_order (struct page *page) {/* PageBuddy() must be checked by the caller */ -return page_private (page); +return page_buddy_order (page); }/*
@@ -555,7 +555,7 @@ static inline unsigned int buddy_order(struct page *page)* times, potentially observing different values in the tests and the actual * use of the result. */ -#define buddy_order_unsafe (page) READ_ONCE(page_private(page)) +#define buddy_order_unsafe (page) READ_ONCE(page_buddy_order(page))/* * This function checks whether a page is free && is the buddy
mm/page_alloc.c +1 -1
@@ -576,7 +576,7 @@ void prep_compound_page(struct page *page, unsigned int order)static inline void set_buddy_order (struct page *page,unsigned int order) { -set_page_private (page, order); +set_page_buddy_order (page, order);__SetPageBuddy (page); }
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 6/26 mm: move should_skip_kasan_poison() to mm/internal.h
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that needs to use should_skip_kasan_poison() function in mm/internal.h.
--- mm/internal.h | 47 +++++++++++++++++++++++++++++++++++++++++++++++ mm/page_alloc.c | 47 ----------------------------------------------- 2 files changed, 47 insertions(+), 47 deletions(-)
mm/internal.h +47 -0
@@ -1038,8 +1038,55 @@ static inline void vunmap_range_noflush(unsigned long start, unsigned long end)DECLARE_STATIC_KEY_TRUE (deferred_pages);bool __init deferred_grow_zone (struct zone *zone,unsigned int order); + +static inline bool deferred_pages_enabled (void ) +{ +return static_branch_unlikely (&deferred_pages); +} +#else +static inline bool deferred_pages_enabled (void ) +{ +return false; +}#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ +/* +* Skip KASAN memory poisoning when either: +* +* 1. For generic KASAN: deferred memory initialization has not yet completed. +* Tag-based KASAN modes skip pages freed via deferred memory initialization +* using page tags instead (see below). +* 2. For tag-based KASAN modes: the page has a match-all KASAN tag, indicating +* that error detection is disabled for accesses via the page address. +* +* Pages will have match-all tags in the following circumstances: +* +* 1. Pages are being initialized for the first time, including during deferred +* memory init; see the call to page_kasan_tag_reset in __init_single_page. +* 2. The allocation was not unpoisoned due to __GFP_SKIP_KASAN, with the +* exception of pages unpoisoned by kasan_unpoison_vmalloc. +* 3. The allocation was excluded from being checked due to sampling, +* see the call to kasan_unpoison_pages. +* +* Poisoning pages during deferred memory init will greatly lengthen the +* process and cause problem in large memory systems as the deferred pages +* initialization is done with interrupt disabled. +* +* Assuming that there will be no reference to those newly initialized +* pages before they are ever allocated, this should have no effect on +* KASAN memory tracking as the poison will be properly inserted at page +* allocation time. The only corner case is when pages are allocated by +* on-demand allocation and then freed again before the deferred pages +* initialization is done, but this is not likely to happen. +*/ +static inline bool should_skip_kasan_poison (struct page *page) +{ +if (IS_ENABLED (CONFIG_KASAN_GENERIC )) +return deferred_pages_enabled (); + +return page_kasan_tag (page) ==KASAN_TAG_KERNEL ; +} +enum mminit_level {MMINIT_WARNING ,MMINIT_VERIFY ,
mm/page_alloc.c +0 -47
65 lines not on this page · raw message
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 7/26 mm: introduce luf_ugen to be used as a global timestamp
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that needs to evaluate the temporal sequence of events to determine whether tlb flush required has been done on each CPU.
To achieve that, this patch introduced a generation number, luf_ugen, and a few APIs manipulating the number. It's worth noting the number is designed to wraparound so care must be taken when using it.
--- include/linux/mm.h | 34 ++++++++++++++++++++++++++++++++++ mm/rmap.c | 22 ++++++++++++++++++++++ 2 files changed, 56 insertions(+)
include/linux/mm.h +34 -0
@@ -4161,4 +4161,38 @@ static inline int do_mseal(unsigned long start, size_t len_in, unsigned long fla }#endif +#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH ) +/* +* luf_ugen will start with 2 so that 1 can be regarded as a passed one. +*/ +#define LUF_UGEN_INIT 2 + +static inline bool ugen_before (unsigned long a,unsigned long b) +{ +/* +* Consider wraparound. +*/ +return (long )(a - b) <0 ; +} + +static inline unsigned long next_ugen (unsigned long ugen) +{ +if (ugen +1 ) +return ugen +1 ; +/* +* Avoid invalid ugen, zero. +*/ +return ugen +2 ; +} + +static inline unsigned long prev_ugen (unsigned long ugen) +{ +if (ugen -1 ) +return ugen -1 ; +/* +* Avoid invalid ugen, zero. +*/ +return ugen -2 ; +} +#endif #endif /* _LINUX_MM_H */
mm/rmap.c +22 -0
@@ -634,6 +634,28 @@ struct anon_vma *folio_lock_anon_vma_read(const struct folio *folio, }#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH + +/* +* This generation number is primarily used as a global timestamp to +* determine whether tlb flush required has been done on each CPU. The +* function, ugen_before(), should be used to evaluate the temporal +* sequence of events because the number is designed to wraparound. +*/ +static atomic_long_t __maybe_unused luf_ugen =ATOMIC_LONG_INIT (LUF_UGEN_INIT ); + +/* +* Don't return invalid luf_ugen, zero. +*/ +static unsigned long __maybe_unused new_luf_ugen (void ) +{ +unsigned long ugen =atomic_long_inc_return (&luf_ugen); + +if (!ugen) + ugen =atomic_long_inc_return (&luf_ugen); + +return ugen; +} +/* * Flush TLB entries for recently unmapped pages from remote CPUs. It is * important if a PTE was dirty when it was unmapped that it's flushed
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 8/26 mm: introduce luf_batch to be used as hash table to store luf meta data
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that needs to keep luf meta data per page while staying in pcp or buddy allocator. The meta data includes cpumask for tlb shootdown and luf's request generation number.
Since struct page doesn't have enough room to store luf meta data, this patch introduces a hash table to store them and makes each page keep its hash key instead.
Since all the pages in pcp or buddy share the hash table, confliction is inevitable so care must be taken when reading or updating its entry.
--- include/linux/mm_types.h | 10 ++++ mm/internal.h | 8 +++ mm/rmap.c | 122 +++++++++++++++++++++++++++++++++++++-- 3 files changed, 136 insertions(+), 4 deletions(-)
include/linux/mm_types.h +10 -0
@@ -32,6 +32,16 @@struct address_space ;struct mem_cgroup ; +#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH +struct luf_batch { +struct tlbflush_unmap_batch batch; +unsigned long ugen; +rwlock_t lock; +}; +#else +struct luf_batch {}; +#endif +/* * Each physical page in the system has a struct page associated with * it to keep track of whatever it is we are using the page for at the
mm/internal.h +8 -0
@@ -1240,6 +1240,8 @@ extern struct workqueue_struct *mm_percpu_wq;void try_to_unmap_flush (void );void try_to_unmap_flush_dirty (void );void flush_tlb_batched_pending (struct mm_struct *mm); +void fold_batch (struct tlbflush_unmap_batch *dst,struct tlbflush_unmap_batch *src,bool reset); +void fold_luf_batch (struct luf_batch *dst,struct luf_batch *src);#else static inline void try_to_unmap_flush (void ) {
@@ -1250,6 +1252,12 @@ static inline void try_to_unmap_flush_dirty(void)static inline void flush_tlb_batched_pending (struct mm_struct *mm) { } +static inline void fold_batch (struct tlbflush_unmap_batch *dst,struct tlbflush_unmap_batch *src,bool reset) +{ +} +static inline void fold_luf_batch (struct luf_batch *dst,struct luf_batch *src) +{ +}#endif /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ extern const struct trace_print_flags pageflag_names[];
mm/rmap.c +118 -4
140 lines not on this page · raw message
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 9/26 mm: introduce API to perform tlb shootdown on exit from page allocator
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that performs tlb shootdown required on exit from page allocator.
This patch introduced a new API rather than making use of existing try_to_unmap_flush() to avoid repeated and redundant tlb shootdown due to frequent page allocations during a session of batched unmap flush.
--- include/linux/sched.h | 1 + mm/internal.h | 4 ++++ mm/rmap.c | 20 ++++++++++++++++++++ 3 files changed, 25 insertions(+)
include/linux/sched.h +1 -0
@@ -1375,6 +1375,7 @@ struct task_struct {#endif struct tlbflush_unmap_batch tlb_ubc; +struct tlbflush_unmap_batch tlb_ubc_takeoff;/* Cache last used pipe for splice(): */ struct pipe_inode_info *splice_pipe;
mm/internal.h +4 -0
@@ -1239,6 +1239,7 @@ extern struct workqueue_struct *mm_percpu_wq;#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH void try_to_unmap_flush (void );void try_to_unmap_flush_dirty (void ); +void try_to_unmap_flush_takeoff (void );void flush_tlb_batched_pending (struct mm_struct *mm);void fold_batch (struct tlbflush_unmap_batch *dst,struct tlbflush_unmap_batch *src,bool reset);void fold_luf_batch (struct luf_batch *dst,struct luf_batch *src);
@@ -1249,6 +1250,9 @@ static inline void try_to_unmap_flush(void)static inline void try_to_unmap_flush_dirty (void ) { } +static inline void try_to_unmap_flush_takeoff (void ) +{ +}static inline void flush_tlb_batched_pending (struct mm_struct *mm) { }
mm/rmap.c +20 -0
@@ -772,6 +772,26 @@ void fold_luf_batch(struct luf_batch *dst, struct luf_batch *src)read_unlock_irqrestore (&src->lock, flags); } +void try_to_unmap_flush_takeoff (void ) +{ +struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; +struct tlbflush_unmap_batch *tlb_ubc_takeoff = ¤t->tlb_ubc_takeoff; + +if (!tlb_ubc_takeoff->flush_required) +return ; + +arch_tlbbatch_flush (&tlb_ubc_takeoff->arch); + +/* +* Now that tlb shootdown of tlb_ubc_takeoff has been performed, +* it's good chance to shrink tlb_ubc if possible. +*/ +if (arch_tlbbatch_done (&tlb_ubc->arch, &tlb_ubc_takeoff->arch)) +reset_batch (tlb_ubc); + +reset_batch (tlb_ubc_takeoff); +} +/* * Flush TLB entries for recently unmapped pages from remote CPUs. It is * important if a PTE was dirty when it was unmapped that it's flushed
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 10/26 mm: introduce APIs to check if the page allocation is tlb shootdownable
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that should indentify if tlb shootdown can be performed on page allocation.
In a context with irq disabled or non-task, tlb shootdown cannot be performed because of deadlock issue. Thus, page allocator should work being aware of whether tlb shootdown can be performed on returning page.
This patch introduced APIs that pcp or buddy page allocator can use to delimit the critical sections taking off pages and indentify whether tlb shootdown can be performed.
--- include/linux/sched.h | 5 ++ mm/internal.h | 14 ++++ mm/page_alloc.c | 159 ++++++++++++++++++++++++++++++++++++++++++ mm/rmap.c | 2 +- 4 files changed, 179 insertions(+), 1 deletion(-)
include/linux/sched.h +5 -0
@@ -1374,6 +1374,11 @@ struct task_struct {struct callback_head cid_work;#endif +#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH ) +int luf_no_shootdown; +int luf_takeoff_started; +#endif +struct tlbflush_unmap_batch tlb_ubc;struct tlbflush_unmap_batch tlb_ubc_takeoff;
mm/internal.h +14 -0
@@ -1583,6 +1583,20 @@ static inline void accept_page(struct page *page) { }#endif /* CONFIG_UNACCEPTED_MEMORY */ +#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH ) +extern struct luf_batch luf_batch[]; +bool luf_takeoff_start (void ); +void luf_takeoff_end (void ); +bool luf_takeoff_no_shootdown (void ); +bool luf_takeoff_check (struct page *page); +bool luf_takeoff_check_and_fold (struct page *page); +#else +static inline bool luf_takeoff_start (void ) {return false; } +static inline void luf_takeoff_end (void ) {} +static inline bool luf_takeoff_no_shootdown (void ) {return true; } +static inline bool luf_takeoff_check (struct page *page) {return true; } +static inline bool luf_takeoff_check_and_fold (struct page *page) {return true; } +#endif /* pagewalk.c */ int walk_page_range_mm(struct mm_struct *mm, unsigned long start,
mm/page_alloc.c +159 -0
165 lines not on this page · raw message
mm/rmap.c +1 -1
@@ -693,7 +693,7 @@ void fold_batch(struct tlbflush_unmap_batch *dst,/* * Use 0th entry as accumulated batch. */ -static struct luf_batch luf_batch[NR_LUF_BATCH ]; +struct luf_batch luf_batch[NR_LUF_BATCH ];static void luf_batch_init (struct luf_batch *lb) {
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 11/26 mm: deliver luf_key to pcp or buddy on free after unmapping
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that needs to pass luf_key to pcp or buddy allocator on free after unmapping e.g. during page reclaim or page migration.
The luf_key will be used to track need of tlb shootdown and which cpus need to perform tlb flush, per page residing in pcp or buddy, and should be handed over properly when pages travel between pcp and buddy.
--- mm/internal.h | 4 +- mm/page_alloc.c | 120 ++++++++++++++++++++++++++++++++------------ mm/page_isolation.c | 6 +++ mm/page_reporting.c | 6 +++ mm/swap.c | 4 +- mm/vmscan.c | 8 +-- 6 files changed, 109 insertions(+), 39 deletions(-)
mm/internal.h +2 -2
@@ -741,8 +741,8 @@ extern bool free_pages_prepare(struct page *page, unsigned int order);extern int user_min_free_kbytes; -void free_unref_page (struct page *page,unsigned int order); -void free_unref_folios (struct folio_batch *fbatch); +void free_unref_page (struct page *page,unsigned int order,unsigned short luf_key); +void free_unref_folios (struct folio_batch *fbatch,unsigned short luf_key);extern void zone_pcp_reset (struct zone *zone);extern void zone_pcp_disable (struct zone *zone);
mm/page_alloc.c +89 -31
335 lines not on this page · raw message
mm/page_isolation.c +6 -0
@@ -258,6 +258,12 @@ static void unset_migratetype_isolate(struct page *page, int migratetype)WARN_ON_ONCE (!move_freepages_block_isolate (zone, page, migratetype)); }else {set_pageblock_migratetype (page, migratetype); + +/* +* Do not clear the page's private to keep its luf_key +* unchanged. +*/ +__putback_isolated_page (page, order, migratetype); } zone->nr_isolate_pageblock--;
mm/page_reporting.c +6 -0
@@ -116,6 +116,12 @@ page_reporting_drain(struct page_reporting_dev_info *prdev,int mt =get_pageblock_migratetype (page);unsigned int order =get_order (sg->length); +/* +* Ensure private is zero before putting into the +* allocator. +*/ +set_page_private (page,0 ); +__putback_isolated_page (page, order, mt);/* If the pages were not reported due to error skip flagging */
mm/swap.c +2 -2
@@ -109,7 +109,7 @@ void __folio_put(struct folio *folio)page_cache_release (folio);folio_unqueue_deferred_split (folio);mem_cgroup_uncharge (folio); -free_unref_page (&folio->page,folio_order (folio)); +free_unref_page (&folio->page,folio_order (folio),0 ); }EXPORT_SYMBOL (__folio_put);
@@ -959,7 +959,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs) folios->nr = j;mem_cgroup_uncharge_folios (folios); -free_unref_folios (folios); +free_unref_folios (folios,0 ); }EXPORT_SYMBOL (folios_put_refs);
mm/vmscan.c +4 -4
@@ -1480,7 +1480,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (folio_batch_add (&free_folios, folio) ==0 ) {mem_cgroup_uncharge_folios (&free_folios);try_to_unmap_flush (); -free_unref_folios (&free_folios); +free_unref_folios (&free_folios,0 ); }continue ;
@@ -1548,7 +1548,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,mem_cgroup_uncharge_folios (&free_folios);try_to_unmap_flush (); -free_unref_folios (&free_folios); +free_unref_folios (&free_folios,0 );list_splice (&ret_folios, folio_list);count_vm_events (PGACTIVATE , pgactivate);
@@ -1868,7 +1868,7 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec,if (folio_batch_add (&free_folios, folio) ==0 ) {spin_unlock_irq (&lruvec->lru_lock);mem_cgroup_uncharge_folios (&free_folios); -free_unref_folios (&free_folios); +free_unref_folios (&free_folios,0 );spin_lock_irq (&lruvec->lru_lock); }
@@ -1890,7 +1890,7 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec,if (free_folios.nr) {spin_unlock_irq (&lruvec->lru_lock);mem_cgroup_uncharge_folios (&free_folios); -free_unref_folios (&free_folios); +free_unref_folios (&free_folios,0 );spin_lock_irq (&lruvec->lru_lock); }
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 12/26 mm: delimit critical sections to take off pages from pcp or buddy alloctor
In reply to Byungchul ParkNow that luf mechanism has been introduced, tlb shootdown might be necessary when luf'd pages exit from pcp or buddy allocator. Check if it's okay to take off pages and can perform for luf'd pages before use.
--- mm/compaction.c | 32 ++++++++++++++++-- mm/internal.h | 2 +- mm/page_alloc.c | 79 +++++++++++++++++++++++++++++++++++++++++++-- mm/page_isolation.c | 4 ++- mm/page_reporting.c | 20 +++++++++++- 5 files changed, 129 insertions(+), 8 deletions(-)
mm/compaction.c +29 -3
@@ -605,6 +605,7 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, page =pfn_to_page (blockpfn); +luf_takeoff_start ();/* Isolate free pages. */ for (; blockpfn < end_pfn; blockpfn += stride, page += stride) {int isolated;
@@ -652,9 +653,12 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, goto isolate_fail; } +if (!luf_takeoff_check (page)) + goto isolate_fail; +/* Found a free page, will break it into order-0 pages */ order =buddy_order (page); - isolated =__isolate_free_page (page, order); + isolated =__isolate_free_page (page, order, false);if (!isolated)break ;set_page_private (page, order);
@@ -682,6 +686,11 @@ static unsigned long isolate_freepages_block(struct compact_control *cc,if (locked)spin_unlock_irqrestore (&cc->zone->lock, flags); +/* +* Check and flush before using the pages taken off. +*/ +luf_takeoff_end (); +/* * Be careful to not go outside of the pageblock. */
@@ -1589,6 +1598,7 @@ static void fast_isolate_freepages(struct compact_control *cc)if (!area->nr_free)continue ; +luf_takeoff_start ();spin_lock_irqsave (&cc->zone->lock, flags); freelist = &area->free_list[MIGRATE_MOVABLE ];list_for_each_entry_reverse (freepage, freelist, buddy_list) {
@@ -1596,6 +1606,10 @@ static void fast_isolate_freepages(struct compact_control *cc) order_scanned++; nr_scanned++; + +if (!luf_takeoff_check (freepage)) + goto scan_next; + pfn =page_to_pfn (freepage);if (pfn >= highest)
@@ -1615,7 +1629,7 @@ static void fast_isolate_freepages(struct compact_control *cc)/* Shorten the scan if a candidate is found */ limit >>=1 ; } - +scan_next:if (order_scanned >= limit)break ; }
@@ -1633,7 +1647,7 @@ static void fast_isolate_freepages(struct compact_control *cc)/* Isolate the page if available */ if (page) { -if (__isolate_free_page (page, order)) { +if (__isolate_free_page (page, order, false)) {set_page_private (page, order); nr_isolated =1 << order; nr_scanned += nr_isolated -1 ;
@@ -1650,6 +1664,11 @@ static void fast_isolate_freepages(struct compact_control *cc)spin_unlock_irqrestore (&cc->zone->lock, flags); +/* +* Check and flush before using the pages taken off. +*/ +luf_takeoff_end (); +/* Skip fast search if enough freepages isolated */ if (cc->nr_freepages >= cc->nr_migratepages)break ;
@@ -2369,7 +2388,14 @@ static enum compact_result compact_finished(struct compact_control *cc) {int ret; +/* +* luf_takeoff_{start,end}() is required to identify whether +* this compaction context is tlb shootdownable for luf'd pages. +*/ +luf_takeoff_start (); ret =__compact_finished (cc); +luf_takeoff_end (); +trace_mm_compaction_finished (cc->zone, cc->order, ret);if (ret ==COMPACT_NO_SUITABLE_PAGE ) ret =COMPACT_CONTINUE ;
mm/internal.h +1 -1
@@ -662,7 +662,7 @@ static inline void clear_zone_contiguous(struct zone *zone) zone->contiguous = false; } -extern int __isolate_free_page (struct page *page,unsigned int order); +extern int __isolate_free_page (struct page *page,unsigned int order,bool willputback);extern void __putback_isolated_page (struct page *page,unsigned int order,int mt); extern void memblock_free_pages(struct page *page, unsigned long pfn,
mm/page_alloc.c +77 -2
246 lines not on this page · raw message
mm/page_isolation.c +3 -1
@@ -211,6 +211,7 @@ static void unset_migratetype_isolate(struct page *page, int migratetype)struct page *buddy; zone =page_zone (page); +luf_takeoff_start ();spin_lock_irqsave (&zone->lock, flags);if (!is_migrate_isolate_page (page)) goto out;
@@ -229,7 +230,7 @@ static void unset_migratetype_isolate(struct page *page, int migratetype) buddy =find_buddy_page_pfn (page,page_to_pfn (page), order,NULL );if (buddy && !is_migrate_isolate_page (buddy)) { - isolated_page = !!__isolate_free_page (page, order); + isolated_page = !!__isolate_free_page (page, order, true);/* * Isolating a free page in an isolated pageblock * is expected to always work as watermarks don't
@@ -269,6 +270,7 @@ static void unset_migratetype_isolate(struct page *page, int migratetype) zone->nr_isolate_pageblock--; out:spin_unlock_irqrestore (&zone->lock, flags); +luf_takeoff_end (zone); } static inline struct page *
mm/page_reporting.c +19 -1
@@ -167,6 +167,7 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,if (list_empty (list))return err; +luf_takeoff_start ();spin_lock_irq (&zone->lock);/*
@@ -191,6 +192,11 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,if (PageReported (page))continue ; +if (!luf_takeoff_check (page)) { +VM_WARN_ON (1 ); +continue ; + } +/* * If we fully consumed our budget then update our * state to indicate that we are requesting additional
@@ -204,7 +210,7 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,/* Attempt to pull page from list and place in scatterlist */ if (*offset) { -if (!__isolate_free_page (page, order)) { +if (!__isolate_free_page (page, order, false)) { next = page;break ; }
@@ -227,6 +233,11 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,/* release lock before waiting on report processing */ spin_unlock_irq (&zone->lock); +/* +* Check and flush before using the pages taken off. +*/ +luf_takeoff_end (); +/* begin processing pages in local list */ err = prdev->report (prdev, sgl,PAGE_REPORTING_CAPACITY );
@@ -236,6 +247,8 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,/* update budget to reflect call to report function */ budget--; +luf_takeoff_start (); +/* reacquire zone lock and resume processing */ spin_lock_irq (&zone->lock);
@@ -259,6 +272,11 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,spin_unlock_irq (&zone->lock); +/* +* Check and flush before using the pages taken off. +*/ +luf_takeoff_end (); +return err; }
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 13/26 mm: introduce pend_list in struct free_area to track luf'd pages
In reply to Byungchul Parkluf'd pages requires tlb shootdown on exiting from page allocator. For some page allocation request, it's okay to return luf'd page followed by tlb shootdown but it's not okay for e.g. irq context.
This patch splitted the list in free_area into two, 'free_list' for non-luf'd pages and 'pend_list' for luf'd pages so that the buddy allocator can work better with various conditions of context.
--- include/linux/mmzone.h | 3 ++ kernel/power/snapshot.c | 14 ++++++ kernel/vmcore_info.c | 2 + mm/compaction.c | 33 ++++++++++--- mm/internal.h | 17 ++++++- mm/mm_init.c | 2 + mm/page_alloc.c | 105 ++++++++++++++++++++++++++++++++++------ mm/page_reporting.c | 22 ++++++--- mm/vmstat.c | 15 ++++++ 9 files changed, 184 insertions(+), 29 deletions(-)
include/linux/mmzone.h +3 -0
@@ -116,6 +116,7 @@ extern int page_group_by_mobility_disabled; MIGRATETYPE_MASK)struct free_area {struct list_head free_list[MIGRATE_TYPES ]; +struct list_head pend_list[MIGRATE_TYPES ];unsigned long nr_free; };
@@ -995,6 +996,8 @@ struct zone {/* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS ];atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS ]; +/* Count pages that need tlb shootdown on allocation */ +atomic_long_t nr_luf_pages; }____cacheline_internodealigned_in_smp ;enum pgdat_flags {
kernel/power/snapshot.c +14 -0
@@ -1288,6 +1288,20 @@ static void mark_free_pages(struct zone *zone)swsusp_set_page_free (pfn_to_page (pfn + i)); } } + +list_for_each_entry (page, + &zone->free_area[order].pend_list[t], buddy_list) { +unsigned long i; + + pfn =page_to_pfn (page); +for (i =0 ; i < (1UL << order); i++) { +if (!--page_count) { +touch_nmi_watchdog (); + page_count =WD_PAGE_COUNT ; + } +swsusp_set_page_free (pfn_to_page (pfn + i)); + } + } }spin_unlock_irqrestore (&zone->lock, flags); }
kernel/vmcore_info.c +2 -0
@@ -188,11 +188,13 @@ static int __init crash_save_vmcoreinfo_init(void)VMCOREINFO_OFFSET (zone, vm_stat);VMCOREINFO_OFFSET (zone, spanned_pages);VMCOREINFO_OFFSET (free_area, free_list); +VMCOREINFO_OFFSET (free_area, pend_list);VMCOREINFO_OFFSET (list_head, next);VMCOREINFO_OFFSET (list_head, prev);VMCOREINFO_LENGTH (zone.free_area,NR_PAGE_ORDERS );log_buf_vmcoreinfo_setup ();VMCOREINFO_LENGTH (free_area.free_list,MIGRATE_TYPES ); +VMCOREINFO_LENGTH (free_area.pend_list,MIGRATE_TYPES );VMCOREINFO_NUMBER (NR_FREE_PAGES );VMCOREINFO_NUMBER (PG_lru);VMCOREINFO_NUMBER (PG_private);
mm/compaction.c +27 -6
@@ -1590,24 +1590,28 @@ static void fast_isolate_freepages(struct compact_control *cc) order = next_search_order(cc, order)) {struct free_area *area = &cc->zone->free_area[order];struct list_head *freelist; +struct list_head *high_pfn_list;struct page *freepage;unsigned long flags;unsigned int order_scanned =0 ;unsigned long high_pfn =0 ; +bool consider_pend = false; +bool can_shootdown;if (!area->nr_free)continue ; -luf_takeoff_start (); + can_shootdown =luf_takeoff_start ();spin_lock_irqsave (&cc->zone->lock, flags); freelist = &area->free_list[MIGRATE_MOVABLE ]; +retry:list_for_each_entry_reverse (freepage, freelist, buddy_list) {unsigned long pfn; order_scanned++; nr_scanned++; -if (!luf_takeoff_check (freepage)) +if (unlikely (consider_pend && !luf_takeoff_check (freepage))) goto scan_next; pfn =page_to_pfn (freepage);
@@ -1620,26 +1624,34 @@ static void fast_isolate_freepages(struct compact_control *cc) cc->fast_search_fail =0 ; cc->search_order = order; page = freepage; -break ; + goto done; }if (pfn >= min_pfn && pfn > high_pfn) { high_pfn = pfn; + high_pfn_list = freelist;/* Shorten the scan if a candidate is found */ limit >>=1 ; } scan_next:if (order_scanned >= limit) -break ; + goto done; } +if (!consider_pend && can_shootdown) { + consider_pend = true; + freelist = &area->pend_list[MIGRATE_MOVABLE ]; + goto retry; + } +done:/* Use a maximum candidate pfn if a preferred one was not found */ if (!page && high_pfn) { page =pfn_to_page (high_pfn);/* Update freepage for the list reorder below */ freepage = page; + freelist = high_pfn_list; }/* Reorder to so a future search skips recent pages */
@@ -2036,18 +2048,20 @@ static unsigned long fast_find_migrateblock(struct compact_control *cc)struct list_head *freelist;unsigned long flags;struct page *freepage; +bool consider_pend = false;if (!area->nr_free)continue ;spin_lock_irqsave (&cc->zone->lock, flags); freelist = &area->free_list[MIGRATE_MOVABLE ]; +retry:list_for_each_entry (freepage, freelist, buddy_list) {unsigned long free_pfn;if (nr_scanned++ >= limit) {move_freelist_tail (freelist, freepage); -break ; + goto done; } free_pfn =page_to_pfn (freepage);
@@ -2070,9 +2084,16 @@ static unsigned long fast_find_migrateblock(struct compact_control *cc) pfn = cc->zone->zone_start_pfn; cc->fast_search_fail =0 ; found_block = true; -break ; + goto done; } } + +if (!consider_pend) { + consider_pend = true; + freelist = &area->pend_list[MIGRATE_MOVABLE ]; + goto retry; + } +done:spin_unlock_irqrestore (&cc->zone->lock, flags); }
mm/internal.h +16 -1
@@ -836,11 +836,16 @@ void init_cma_reserved_pageblock(struct page *page);int find_suitable_fallback (struct free_area *area,unsigned int order,int migratetype,bool only_stealable,bool *can_steal); -static inline bool free_area_empty (struct free_area *area,int migratetype) +static inline bool free_list_empty (struct free_area *area,int migratetype) {return list_empty (&area->free_list[migratetype]); } +static inline bool free_area_empty (struct free_area *area,int migratetype) +{ +return list_empty (&area->free_list[migratetype]) && +list_empty (&area->pend_list[migratetype]); +}/* mm/util.c */ struct anon_vma *folio_anon_vma (const struct folio *folio);
@@ -1590,12 +1595,22 @@ void luf_takeoff_end(void);bool luf_takeoff_no_shootdown (void );bool luf_takeoff_check (struct page *page);bool luf_takeoff_check_and_fold (struct page *page); + +static inline bool non_luf_pages_ok (struct zone *zone) +{ +unsigned long nr_free =zone_page_state (zone,NR_FREE_PAGES ); +unsigned long min_wm =min_wmark_pages (zone); +unsigned long nr_luf_pages =atomic_long_read (&zone->nr_luf_pages); + +return nr_free - nr_luf_pages > min_wm; +}#else static inline bool luf_takeoff_start (void ) {return false; }static inline void luf_takeoff_end (void ) {}static inline bool luf_takeoff_no_shootdown (void ) {return true; }static inline bool luf_takeoff_check (struct page *page) {return true; }static inline bool luf_takeoff_check_and_fold (struct page *page) {return true; } +static inline bool non_luf_pages_ok (struct zone *zone) {return true; }#endif /* pagewalk.c */
mm/mm_init.c +2 -0
@@ -1396,12 +1396,14 @@ static void __meminit zone_init_free_lists(struct zone *zone)unsigned int order, t;for_each_migratetype_order (order, t) {INIT_LIST_HEAD (&zone->free_area[order].free_list[t]); +INIT_LIST_HEAD (&zone->free_area[order].pend_list[t]); zone->free_area[order].nr_free =0 ; }#ifdef CONFIG_UNACCEPTED_MEMORY INIT_LIST_HEAD (&zone->unaccepted_pages);#endif +atomic_long_set (&zone->nr_luf_pages,0 ); } void __meminit init_currently_empty_zone(struct zone *zone,
mm/page_alloc.c +90 -15
193 lines not on this page · raw message
mm/page_reporting.c +15 -7
@@ -159,15 +159,17 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,struct page *page, *next;long budget;int err =0 ; +bool consider_pend = false; +bool can_shootdown;/* * Perform early check, if free area is empty there is * nothing to process so we can skip this free_list. */ -if (list_empty (list)) +if (free_area_empty (area, mt))return err; -luf_takeoff_start (); + can_shootdown =luf_takeoff_start ();spin_lock_irq (&zone->lock);/*
@@ -185,14 +187,14 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,* should always be a power of 2. */ budget =DIV_ROUND_UP (area->nr_free,PAGE_REPORTING_CAPACITY *16 ); - +retry:/* loop through free list adding unreported pages to sg list */ list_for_each_entry_safe (page, next, list, lru) {/* We are going to skip over the reported pages. */ if (PageReported (page))continue ; -if (!luf_takeoff_check (page)) { +if (unlikely (consider_pend && !luf_takeoff_check (page))) {VM_WARN_ON (1 );continue ; }
@@ -205,14 +207,14 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,if (budget <0 ) {atomic_set (&prdev->state,PAGE_REPORTING_REQUESTED ); next = page; -break ; + goto done; }/* Attempt to pull page from list and place in scatterlist */ if (*offset) {if (!__isolate_free_page (page, order, false)) { next = page; -break ; + goto done; }/* Add page to scatter list */
@@ -263,9 +265,15 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,/* exit on error */ if (err) -break ; + goto done; } +if (!consider_pend && can_shootdown) { + consider_pend = true; + list = &area->pend_list[mt]; + goto retry; + } +done:/* Rotate any leftover pages to the head of the freelist */ if (!list_entry_is_head (next, list, lru) && !list_is_first (&next->lru, list))list_rotate_to_front (&next->lru, list);
mm/vmstat.c +15 -0
@@ -1581,6 +1581,21 @@ static void pagetypeinfo_showfree_print(struct seq_file *m,break ; } } +list_for_each (curr, &area->pend_list[mtype]) { +/* +* Cap the pend_list iteration because it might +* be really large and we are under a spinlock +* so a long time spent here could trigger a +* hard lockup detector. Anyway this is a +* debugging tool so knowing there is a handful +* of pages of this order should be more than +* sufficient. +*/ +if (++freecount >=100000 ) { + overflow = true; +break ; + } + }seq_printf (m,"%s%6lu " , overflow ?">" :"" , freecount);spin_unlock_irq (&zone->lock);cond_resched ();
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 14/26 mm/rmap: recognize read-only tlb entries during batched tlb flush
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that requires to recognize read-only tlb entries and handle them in a different way. The newly introduced API in this patch, fold_ubc(), will be used by luf mechanism.
--- include/linux/sched.h | 1 + mm/rmap.c | 16 ++++++++++++++-- 2 files changed, 15 insertions(+), 2 deletions(-)
include/linux/sched.h +1 -0
@@ -1381,6 +1381,7 @@ struct task_struct {struct tlbflush_unmap_batch tlb_ubc;struct tlbflush_unmap_batch tlb_ubc_takeoff; +struct tlbflush_unmap_batch tlb_ubc_ro;/* Cache last used pipe for splice(): */ struct pipe_inode_info *splice_pipe;
mm/rmap.c +14 -2
@@ -775,6 +775,7 @@ void fold_luf_batch(struct luf_batch *dst, struct luf_batch *src)void try_to_unmap_flush_takeoff (void ) {struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; +struct tlbflush_unmap_batch *tlb_ubc_ro = ¤t->tlb_ubc_ro;struct tlbflush_unmap_batch *tlb_ubc_takeoff = ¤t->tlb_ubc_takeoff;if (!tlb_ubc_takeoff->flush_required)
@@ -789,6 +790,9 @@ void try_to_unmap_flush_takeoff(void)if (arch_tlbbatch_done (&tlb_ubc->arch, &tlb_ubc_takeoff->arch))reset_batch (tlb_ubc); +if (arch_tlbbatch_done (&tlb_ubc_ro->arch, &tlb_ubc_takeoff->arch)) +reset_batch (tlb_ubc_ro); +reset_batch (tlb_ubc_takeoff); }
@@ -801,7 +805,9 @@ void try_to_unmap_flush_takeoff(void)void try_to_unmap_flush (void ) {struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; +struct tlbflush_unmap_batch *tlb_ubc_ro = ¤t->tlb_ubc_ro; +fold_batch (tlb_ubc, tlb_ubc_ro, true);if (!tlb_ubc->flush_required)return ;
@@ -813,8 +819,9 @@ void try_to_unmap_flush(void)void try_to_unmap_flush_dirty (void ) {struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; +struct tlbflush_unmap_batch *tlb_ubc_ro = ¤t->tlb_ubc_ro; -if (tlb_ubc->writable) +if (tlb_ubc->writable || tlb_ubc_ro->writable)try_to_unmap_flush (); }
@@ -831,13 +838,18 @@ void try_to_unmap_flush_dirty(void)static void set_tlb_ubc_flush_pending (struct mm_struct *mm,pte_t pteval,unsigned long uaddr) { -struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; +struct tlbflush_unmap_batch *tlb_ubc;int batch;bool writable =pte_dirty (pteval);if (!pte_accessible (mm, pteval))return ; +if (pte_write (pteval)) + tlb_ubc = ¤t->tlb_ubc; +else + tlb_ubc = ¤t->tlb_ubc_ro; +arch_tlbbatch_add_pending (&tlb_ubc->arch, mm, uaddr); tlb_ubc->flush_required = true;
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 15/26 fs, filemap: refactor to gather the scattered ->write_{begin,end}() calls
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that requires to hook when updating page cache that might have pages that have been mapped on any tasks so that tlb flush needed can be performed.
--- drivers/gpu/drm/i915/gem/i915_gem_shmem.c | 11 ++++------- fs/affs/file.c | 4 ++-- fs/buffer.c | 14 ++++++-------- fs/exfat/file.c | 5 ++--- fs/ext4/verity.c | 5 ++--- fs/f2fs/super.c | 5 ++--- fs/f2fs/verity.c | 5 ++--- fs/namei.c | 5 ++--- include/linux/fs.h | 18 ++++++++++++++++++ mm/filemap.c | 5 ++--- 10 files changed, 42 insertions(+), 35 deletions(-)
drivers/gpu/drm/i915/gem/i915_gem_shmem.c +4 -7
@@ -422,7 +422,6 @@ shmem_pwrite(struct drm_i915_gem_object *obj, const struct drm_i915_gem_pwrite *arg) {struct address_space *mapping = obj->base.filp->f_mapping; -const struct address_space_operations *aops = mapping->a_ops;char __user *user_data =u64_to_user_ptr (arg->data_ptr);u64 remain;loff_t pos;
@@ -481,7 +480,7 @@ shmem_pwrite(struct drm_i915_gem_object *obj,if (err)return err; - err = aops->write_begin (obj->base.filp, mapping, pos, len, + err =mapping_write_begin (obj->base.filp, mapping, pos, len, &folio, &data);if (err <0 )return err;
@@ -492,7 +491,7 @@ shmem_pwrite(struct drm_i915_gem_object *obj,pagefault_enable ();kunmap_local (vaddr); - err = aops->write_end (obj->base.filp, mapping, pos, len, + err =mapping_write_end (obj->base.filp, mapping, pos, len, len - unwritten, folio, data);if (err <0 )return err;
@@ -658,7 +657,6 @@ i915_gem_object_create_shmem_from_data(struct drm_i915_private *i915, {struct drm_i915_gem_object *obj;struct file *file; -const struct address_space_operations *aops;loff_t pos;int err;
@@ -670,21 +668,20 @@ i915_gem_object_create_shmem_from_data(struct drm_i915_private *i915,GEM_BUG_ON (obj->write_domain !=I915_GEM_DOMAIN_CPU ); file = obj->base.filp; - aops = file->f_mapping->a_ops; pos =0 ; do { unsigned int len = min_t(typeof(size), size, PAGE_SIZE); struct folio *folio; void *fsdata; - err = aops->write_begin(file, file->f_mapping, pos, len, + err = mapping_write_begin(file, file->f_mapping, pos, len, &folio, &fsdata); if (err < 0) goto fail; memcpy_to_folio(folio, offset_in_folio(folio, pos), data, len); - err = aops->write_end(file, file->f_mapping, pos, len, len, + err = mapping_write_end(file, file->f_mapping, pos, len, len, folio, fsdata); if (err < 0) goto fail;
fs/affs/file.c +2 -2
@@ -885,9 +885,9 @@ affs_truncate(struct inode *inode)loff_t isize = inode->i_size;int res; - res = mapping->a_ops->write_begin (NULL , mapping, isize,0 , &folio, &fsdata); + res =mapping_write_begin (NULL , mapping, isize,0 , &folio, &fsdata);if (!res) - res = mapping->a_ops->write_end (NULL , mapping, isize,0 ,0 , folio, fsdata); + res =mapping_write_end (NULL , mapping, isize,0 ,0 , folio, fsdata);else inode->i_size =AFFS_I (inode)->mmu_private;mark_inode_dirty (inode);
fs/buffer.c +6 -8
@@ -2456,7 +2456,6 @@ EXPORT_SYMBOL(block_read_full_folio);int generic_cont_expand_simple (struct inode *inode,loff_t size) {struct address_space *mapping = inode->i_mapping; -const struct address_space_operations *aops = mapping->a_ops;struct folio *folio;void *fsdata =NULL ;int err;
@@ -2465,11 +2464,11 @@ int generic_cont_expand_simple(struct inode *inode, loff_t size)if (err) goto out; - err = aops->write_begin (NULL , mapping, size,0 , &folio, &fsdata); + err =mapping_write_begin (NULL , mapping, size,0 , &folio, &fsdata);if (err) goto out; - err = aops->write_end (NULL , mapping, size,0 ,0 , folio, fsdata); + err =mapping_write_end (NULL , mapping, size,0 ,0 , folio, fsdata);BUG_ON (err >0 ); out:
@@ -2481,7 +2480,6 @@ static int cont_expand_zero(struct file *file, struct address_space *mapping, loff_t pos, loff_t *bytes) { struct inode *inode = mapping->host; -const struct address_space_operations *aops = mapping->a_ops;unsigned int blocksize =i_blocksize (inode);struct folio *folio;void *fsdata =NULL ;
@@ -2501,12 +2499,12 @@ static int cont_expand_zero(struct file *file, struct address_space *mapping, } len =PAGE_SIZE - zerofrom; - err = aops->write_begin (file, mapping, curpos, len, + err =mapping_write_begin (file, mapping, curpos, len, &folio, &fsdata);if (err) goto out;folio_zero_range (folio,offset_in_folio (folio, curpos), len); - err = aops->write_end (file, mapping, curpos, len, len, + err =mapping_write_end (file, mapping, curpos, len, len, folio, fsdata);if (err <0 ) goto out;
@@ -2534,12 +2532,12 @@ static int cont_expand_zero(struct file *file, struct address_space *mapping, } len = offset - zerofrom; - err = aops->write_begin (file, mapping, curpos, len, + err =mapping_write_begin (file, mapping, curpos, len, &folio, &fsdata);if (err) goto out;folio_zero_range (folio,offset_in_folio (folio, curpos), len); - err = aops->write_end (file, mapping, curpos, len, len, + err =mapping_write_end (file, mapping, curpos, len, len, folio, fsdata);if (err <0 ) goto out;
fs/exfat/file.c +2 -3
@@ -539,7 +539,6 @@ static int exfat_extend_valid_size(struct file *file, loff_t new_valid_size)struct inode *inode =file_inode (file);struct exfat_inode_info *ei =EXFAT_I (inode);struct address_space *mapping = inode->i_mapping; -const struct address_space_operations *ops = mapping->a_ops; pos = ei->valid_size;while (pos < new_valid_size) {
@@ -550,11 +549,11 @@ static int exfat_extend_valid_size(struct file *file, loff_t new_valid_size)if (pos + len > new_valid_size) len = new_valid_size - pos; - err = ops->write_begin (file, mapping, pos, len, &folio,NULL ); + err =mapping_write_begin (file, mapping, pos, len, &folio,NULL );if (err) goto out; - err = ops->write_end (file, mapping, pos, len, len, folio,NULL ); + err =mapping_write_end (file, mapping, pos, len, len, folio,NULL );if (err <0 ) goto out; pos += len;
fs/ext4/verity.c +2 -3
@@ -68,7 +68,6 @@ static int pagecache_write(struct inode *inode, const void *buf, size_t count, loff_t pos) {struct address_space *mapping = inode->i_mapping; -const struct address_space_operations *aops = mapping->a_ops;if (pos + count > inode->i_sb->s_maxbytes)return -EFBIG ;
@@ -80,13 +79,13 @@ static int pagecache_write(struct inode *inode, const void *buf, size_t count,void *fsdata =NULL ;int res; - res = aops->write_begin (NULL , mapping, pos, n, &folio, &fsdata); + res =mapping_write_begin (NULL , mapping, pos, n, &folio, &fsdata);if (res)return res;memcpy_to_folio (folio,offset_in_folio (folio, pos), buf, n); - res = aops->write_end (NULL , mapping, pos, n, n, folio, fsdata); + res =mapping_write_end (NULL , mapping, pos, n, n, folio, fsdata);if (res <0 )return res;if (res != n)
fs/f2fs/super.c +2 -3
@@ -2678,7 +2678,6 @@ static ssize_t f2fs_quota_write(struct super_block *sb, int type, {struct inode *inode =sb_dqopt (sb)->files[type];struct address_space *mapping = inode->i_mapping; -const struct address_space_operations *a_ops = mapping->a_ops;int offset = off & (sb->s_blocksize -1 );size_t towrite = len;struct folio *folio;
@@ -2690,7 +2689,7 @@ static ssize_t f2fs_quota_write(struct super_block *sb, int type, tocopy = min_t(unsigned long, sb->s_blocksize - offset, towrite); retry: - err = a_ops->write_begin (NULL , mapping, off, tocopy, + err =mapping_write_begin (NULL , mapping, off, tocopy, &folio, &fsdata);if (unlikely (err)) {if (err == -ENOMEM ) {
@@ -2703,7 +2702,7 @@ static ssize_t f2fs_quota_write(struct super_block *sb, int type,memcpy_to_folio (folio,offset_in_folio (folio, off), data, tocopy); - a_ops->write_end (NULL , mapping, off, tocopy, tocopy, +mapping_write_end (NULL , mapping, off, tocopy, tocopy, folio, fsdata); offset =0 ; towrite -= tocopy;
fs/f2fs/verity.c +2 -3
@@ -72,7 +72,6 @@ static int pagecache_write(struct inode *inode, const void *buf, size_t count, loff_t pos) {struct address_space *mapping = inode->i_mapping; -const struct address_space_operations *aops = mapping->a_ops;if (pos + count >F2FS_BLK_TO_BYTES (max_file_blocks (inode)))return -EFBIG ;
@@ -84,13 +83,13 @@ static int pagecache_write(struct inode *inode, const void *buf, size_t count,void *fsdata =NULL ;int res; - res = aops->write_begin (NULL , mapping, pos, n, &folio, &fsdata); + res =mapping_write_begin (NULL , mapping, pos, n, &folio, &fsdata);if (res)return res;memcpy_to_folio (folio,offset_in_folio (folio, pos), buf, n); - res = aops->write_end (NULL , mapping, pos, n, n, folio, fsdata); + res =mapping_write_end (NULL , mapping, pos, n, n, folio, fsdata);if (res <0 )return res;if (res != n)
fs/namei.c +2 -3
@@ -5349,7 +5349,6 @@ EXPORT_SYMBOL(page_readlink);int page_symlink (struct inode *inode,const char *symname,int len) {struct address_space *mapping = inode->i_mapping; -const struct address_space_operations *aops = mapping->a_ops;bool nofs = !mapping_gfp_constraint (mapping, __GFP_FS);struct folio *folio;void *fsdata =NULL ;
@@ -5359,7 +5358,7 @@ int page_symlink(struct inode *inode, const char *symname, int len) retry:if (nofs) flags =memalloc_nofs_save (); - err = aops->write_begin (NULL , mapping,0 , len-1 , &folio, &fsdata); + err =mapping_write_begin (NULL , mapping,0 , len-1 , &folio, &fsdata);if (nofs)memalloc_nofs_restore (flags);if (err)
@@ -5367,7 +5366,7 @@ int page_symlink(struct inode *inode, const char *symname, int len)memcpy (folio_address (folio), symname, len -1 ); - err = aops->write_end (NULL , mapping,0 , len -1 , len -1 , + err =mapping_write_end (NULL , mapping,0 , len -1 , len -1 , folio, fsdata);if (err <0 ) goto fail;
include/linux/fs.h +18 -0
@@ -494,6 +494,24 @@ struct address_space {#define PAGECACHE_TAG_WRITEBACK XA_MARK_1#define PAGECACHE_TAG_TOWRITE XA_MARK_2 +static inline int mapping_write_begin (struct file *file, +struct address_space *mapping, +loff_t pos,unsigned len, +struct folio **foliop,void **fsdata) +{ +return mapping->a_ops->write_begin (file, mapping, pos, len, foliop, + fsdata); +} + +static inline int mapping_write_end (struct file *file, +struct address_space *mapping, +loff_t pos,unsigned len,unsigned copied, +struct folio *folio,void *fsdata) +{ +return mapping->a_ops->write_end (file, mapping, pos, len, copied, + folio, fsdata); +} +/* * Returns true if any of the pages in the mapping are marked with the tag. */
mm/filemap.c +2 -3
@@ -4016,7 +4016,6 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i)struct file *file = iocb->ki_filp;loff_t pos = iocb->ki_pos;struct address_space *mapping = file->f_mapping; -const struct address_space_operations *a_ops = mapping->a_ops;size_t chunk =mapping_max_folio_size (mapping);long status =0 ;ssize_t written =0 ;
@@ -4050,7 +4049,7 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i)break ; } - status = a_ops->write_begin (file, mapping, pos, bytes, + status =mapping_write_begin (file, mapping, pos, bytes, &folio, &fsdata);if (unlikely (status <0 ))break ;
@@ -4065,7 +4064,7 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i) copied =copy_folio_from_iter_atomic (folio, offset, bytes, i);flush_dcache_folio (folio); - status = a_ops->write_end (file, mapping, pos, bytes, copied, + status =mapping_write_end (file, mapping, pos, bytes, copied, folio, fsdata);if (unlikely (status != copied)) {iov_iter_revert (i, copied -max (status,0L ));
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 16/26 mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped
In reply to Byungchul ParkA new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read-only and were unmapped, as long as the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.
tlb flush can be defered when folios get unmapped as long as it guarantees to perform tlb flush needed, before the folios actually become used, of course, only if all the corresponding ptes don't have write permission. Otherwise, the system will get messed up.
To achieve that, for the folios that map only to non-writable tlb entries, prevent tlb flush during unmapping but perform it just before the folios actually become used, out of buddy or pcp.
However, we should cancel the pending by LUF and perform the deferred TLB flush right away when:
1. a writable pte is newly set through fault handler
2. a file is updated
3. kasan needs poisoning on free
4. the kernel wants to init pages on free
No matter what type of workload is used for performance evaluation, the result would be positive thanks to the unconditional reduction of tlb flushes, tlb misses and interrupts. For the test, I picked up one of the most popular and heavy workload, llama.cpp that is a LLM(Large Language Model) inference engine.
The result would depend on memory latency and how often reclaim runs, which implies tlb miss overhead and how many times unmapping happens. In my system, the result shows:
1. tlb shootdown interrupts are reduced about 97%.
2. The test program runtime is reduced about 4.5%.
The test environment and the test set are like:
Machine: bare metal, x86_64, Intel(R) Xeon(R) Gold 6430
CPU: 1 socket 64 core with hyper thread on Numa: 2 nodes (64 CPUs DRAM 42GB, no CPUs CXL expander 98GB) Config: swap off, numa balancing tiering on, demotion enabled
llama.cpp/main -m $(70G_model1) -p "who are you?" -s 1 -t 15 -n 20 & llama.cpp/main -m $(70G_model2) -p "who are you?" -s 1 -t 15 -n 20 & llama.cpp/main -m $(70G_model3) -p "who are you?" -s 1 -t 15 -n 20 & wait
where,
-t: nr of threads, -s: seed used to make the runtime stable, -n: nr of tokens that determines the runtime, -p: prompt to ask, -m: LLM model to use.
Run the test set 5 times successively with caches dropped every run via 'echo 3 > /proc/sys/vm/drop_caches'. Each inference prints its runtime at the end of each. The results are like:
1. Runtime from the output of llama.cpp
BEFORE
------
llama_print_timings: total time = 883450.54 ms / 24 tokens llama_print_timings: total time = 861665.91 ms / 24 tokens llama_print_timings: total time = 898079.02 ms / 24 tokens llama_print_timings: total time = 879897.69 ms / 24 tokens llama_print_timings: total time = 892360.75 ms / 24 tokens llama_print_timings: total time = 884587.85 ms / 24 tokens llama_print_timings: total time = 861023.19 ms / 24 tokens llama_print_timings: total time = 900022.18 ms / 24 tokens llama_print_timings: total time = 878771.88 ms / 24 tokens llama_print_timings: total time = 889027.98 ms / 24 tokens llama_print_timings: total time = 880783.90 ms / 24 tokens llama_print_timings: total time = 856475.29 ms / 24 tokens llama_print_timings: total time = 896842.21 ms / 24 tokens llama_print_timings: total time = 878883.53 ms / 24 tokens llama_print_timings: total time = 890122.10 ms / 24 tokens
AFTER
-----
llama_print_timings: total time = 871060.86 ms / 24 tokens llama_print_timings: total time = 825609.53 ms / 24 tokens llama_print_timings: total time = 836854.81 ms / 24 tokens llama_print_timings: total time = 843147.99 ms / 24 tokens llama_print_timings: total time = 831426.65 ms / 24 tokens llama_print_timings: total time = 873939.23 ms / 24 tokens llama_print_timings: total time = 826127.69 ms / 24 tokens llama_print_timings: total time = 835489.26 ms / 24 tokens llama_print_timings: total time = 842589.62 ms / 24 tokens llama_print_timings: total time = 833700.66 ms / 24 tokens llama_print_timings: total time = 875996.19 ms / 24 tokens llama_print_timings: total time = 826401.73 ms / 24 tokens llama_print_timings: total time = 839341.28 ms / 24 tokens llama_print_timings: total time = 841075.10 ms / 24 tokens llama_print_timings: total time = 835136.41 ms / 24 tokens
2. tlb shootdowns from 'cat /proc/interrupts'
BEFORE
------
TLB:
80911532 93691786 100296251 111062810 109769109 109862429 108968588 119175230 115779676 118377498 119325266 120300143 124514185 116697222 121068466 118031913 122660681 117494403 121819907 116960596 120936335 117217061 118630217 122322724 119595577 111693298 119232201 120030377 115334687 113179982 118808254 116353592 140987367 137095516 131724276 139742240 136501150 130428761 127585535 132483981 133430250 133756207 131786710 126365824 129812539 133850040 131742690 125142213 128572830 132234350 131945922 128417707 133355434 129972846 126331823 134050849 133991626 121129038 124637283 132830916 126875507 122322440 125776487 124340278 TLB shootdowns
AFTER
-----
TLB:
2121206 2615108 2983494 2911950 3055086 3092672
3204894 3346082 3286744 3307310 3357296 3315940
3428034 3112596 3143325 3185551 3186493 3322314
3330523 3339663 3156064 3272070 3296309 3198962
3332662 3315870 3234467 3353240 3281234 3300666
3345452 3173097 4009196 3932215 3898735 3726531
3717982 3671726 3728788 3724613 3799147 3691764
3620630 3684655 3666688 3393974 3448651 3487593
3446357 3618418 3671920 3712949 3575264 3715385
3641513 3630897 3691047 3630690 3504933 3662647
3629926 3443044 3832970 3548813 TLB shootdowns--- include/asm-generic/tlb.h | 5 ++ include/linux/fs.h | 12 +++- include/linux/mm_types.h | 6 ++ include/linux/sched.h | 9 +++ kernel/sched/core.c | 1 + mm/internal.h | 94 ++++++++++++++++++++++++- mm/memory.c | 15 ++++ mm/pgtable-generic.c | 2 + mm/rmap.c | 141 +++++++++++++++++++++++++++++++++++--- mm/truncate.c | 55 +++++++++++++-- mm/vmscan.c | 12 +++- 11 files changed, 333 insertions(+), 19 deletions(-)
include/asm-generic/tlb.h +5 -0
@@ -549,6 +549,11 @@ static inline void tlb_start_vma(struct mmu_gather *tlb, struct vm_area_struct *static inline void tlb_end_vma (struct mmu_gather *tlb,struct vm_area_struct *vma) { +/* +* Don't leave stale tlb entries for this vma. +*/ +luf_flush (0 ); +if (tlb->fullmm)return ;
include/linux/fs.h +11 -1
@@ -499,8 +499,18 @@ static inline int mapping_write_begin(struct file *file, loff_t pos, unsigned len, struct folio **foliop, void **fsdata) { - return mapping->a_ops->write_begin(file, mapping, pos, len, foliop, + int ret; + + ret = mapping->a_ops->write_begin (file, mapping, pos, len, foliop, fsdata); + +/* +* Ensure to clean stale tlb entries for this mapping. +*/ +if (!ret) +luf_flush (0 ); + +return ret; } static inline int mapping_write_end(struct file *file,
include/linux/mm_types.h +6 -0
@@ -1270,6 +1270,12 @@ extern void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm);extern void tlb_gather_mmu_fullmm (struct mmu_gather *tlb,struct mm_struct *mm);extern void tlb_finish_mmu (struct mmu_gather *tlb); +#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH ) +void luf_flush (unsigned short luf_key); +#else +static inline void luf_flush (unsigned short luf_key) {} +#endif +struct vm_fault ;/**
include/linux/sched.h +9 -0
@@ -1382,6 +1382,15 @@ struct task_struct {struct tlbflush_unmap_batch tlb_ubc;struct tlbflush_unmap_batch tlb_ubc_takeoff;struct tlbflush_unmap_batch tlb_ubc_ro; +struct tlbflush_unmap_batch tlb_ubc_luf; + +#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH ) +/* +* whether all the mappings of a folio during unmap are read-only +* so that luf can work on the folio +*/ +bool can_luf; +#endif /* Cache last used pipe for splice(): */ struct pipe_inode_info *splice_pipe;
kernel/sched/core.c +1 -0
@@ -5225,6 +5225,7 @@ static struct rq *finish_task_switch(struct task_struct *prev)if (mm) {membarrier_mm_sync_core_before_usermode (mm);mmdrop_lazy_tlb_sched (mm); +luf_flush (0 ); }if (unlikely (prev_state ==TASK_DEAD )) {
mm/internal.h +93 -1
106 lines not on this page · raw message
mm/memory.c +15 -0
@@ -6081,6 +6081,7 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,struct mm_struct *mm = vma->vm_mm;vm_fault_t ret;bool is_droppable; +bool flush = false;__set_current_state (TASK_RUNNING );
@@ -6106,6 +6107,14 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,lru_gen_enter_fault (vma); +/* +* Any potential cases that make pte writable even forcely +* should be considered. +*/ +if (vma->vm_flags & (VM_WRITE |VM_MAYWRITE ) || + flags &FAULT_FLAG_WRITE ) + flush = true; +if (unlikely (is_vm_hugetlb_page (vma))) ret =hugetlb_fault (vma->vm_mm, vma, address, flags);else
@@ -6137,6 +6146,12 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, out:mm_account_fault (mm, regs, address, flags, ret); +/* +* Ensure to clean stale tlb entries for this vma. +*/ +if (flush) +luf_flush (0 ); +return ret; }EXPORT_SYMBOL_GPL (handle_mm_fault);
mm/pgtable-generic.c +2 -0
@@ -99,6 +99,8 @@ pte_t ptep_clear_flush(struct vm_area_struct *vma, unsigned long address, pte =ptep_get_and_clear (mm, address, ptep);if (pte_accessible (mm, pte))flush_tlb_page (vma, address); +else +luf_flush (0 ); return pte; }#endif
mm/rmap.c +133 -8
276 lines not on this page · raw message
mm/truncate.c +48 -7
117 lines not on this page · raw message
mm/vmscan.c +10 -2
@@ -821,6 +821,8 @@ static int __remove_mapping(struct address_space *mapping, struct folio *folio,*/ long remove_mapping (struct address_space *mapping,struct folio *folio) { +long ret =0 ; +if (__remove_mapping (mapping, folio, false,NULL )) {/* * Unfreezing the refcount with 1 effectively
@@ -828,9 +830,15 @@ long remove_mapping(struct address_space *mapping, struct folio *folio)* atomic operation. */ folio_ref_unfreeze (folio,1 ); -return folio_nr_pages (folio); + ret =folio_nr_pages (folio); } -return 0 ; + +/* +* Ensure to clean stale tlb entries for this mapping. +*/ +luf_flush (0 ); + +return ret; }/**
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 17/26 x86/tlb, riscv/tlb, arm64/tlbflush, mm: remove cpus from tlb shootdown that already have been done
In reply to Byungchul Parkluf mechanism performs tlb shootdown for mappings that have been unmapped in lazy manner. However, it doesn't have to perform tlb shootdown to cpus that already have been done by others since the tlb shootdown was desired.
Since luf already introduced its own generation number used as a global timestamp, luf_ugen, it's possible to selectively pick cpus that have been done tlb flush required.
This patch introduced APIs that use the generation number to select and remove those cpus so that it can perform tlb shootdown with a smaller cpumask, for all the CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH archs, x86, riscv, and arm64.
--- arch/arm64/include/asm/tlbflush.h | 26 +++++++ arch/riscv/include/asm/tlbflush.h | 4 ++ arch/riscv/mm/tlbflush.c | 108 ++++++++++++++++++++++++++++++ arch/x86/include/asm/tlbflush.h | 4 ++ arch/x86/mm/tlb.c | 108 ++++++++++++++++++++++++++++++ include/linux/sched.h | 1 + mm/internal.h | 4 ++ mm/page_alloc.c | 32 +++++++-- mm/rmap.c | 46 ++++++++++++- 9 files changed, 327 insertions(+), 6 deletions(-)
arch/arm64/include/asm/tlbflush.h +26 -0
@@ -354,6 +354,32 @@ static inline void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)dsb (ish); } +static inline bool arch_tlbbatch_check_done (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen) +{ +/* +* Nothing is needed in this architecture. +*/ +return true; +} + +static inline bool arch_tlbbatch_diet (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen) +{ +/* +* Nothing is needed in this architecture. +*/ +return true; +} + +static inline void arch_tlbbatch_mark_ugen (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen) +{ +/* nothing to do */ +} + +static inline void arch_mm_mark_ugen (struct mm_struct *mm,unsigned long ugen) +{ +/* nothing to do */ +} +static inline void arch_tlbbatch_clear (struct arch_tlbflush_unmap_batch *batch) {/* nothing to do */
arch/riscv/include/asm/tlbflush.h +4 -0
@@ -65,6 +65,10 @@ void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch, unsigned long uaddr);void arch_flush_tlb_batched_pending (struct mm_struct *mm);void arch_tlbbatch_flush (struct arch_tlbflush_unmap_batch *batch); +bool arch_tlbbatch_check_done (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen); +bool arch_tlbbatch_diet (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen); +void arch_tlbbatch_mark_ugen (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen); +void arch_mm_mark_ugen (struct mm_struct *mm,unsigned long ugen);static inline void arch_tlbbatch_clear (struct arch_tlbflush_unmap_batch *batch) {
arch/riscv/mm/tlbflush.c +108 -0
111 lines not on this page · raw message
arch/x86/include/asm/tlbflush.h +4 -0
@@ -293,6 +293,10 @@ static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm) }extern void arch_tlbbatch_flush (struct arch_tlbflush_unmap_batch *batch); +extern bool arch_tlbbatch_check_done (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen); +extern bool arch_tlbbatch_diet (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen); +extern void arch_tlbbatch_mark_ugen (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen); +extern void arch_mm_mark_ugen (struct mm_struct *mm,unsigned long ugen);static inline void arch_tlbbatch_clear (struct arch_tlbflush_unmap_batch *batch) {
arch/x86/mm/tlb.c +108 -0
114 lines not on this page · raw message
include/linux/sched.h +1 -0
@@ -1377,6 +1377,7 @@ struct task_struct {#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH )int luf_no_shootdown;int luf_takeoff_started; +unsigned long luf_ugen;#endif struct tlbflush_unmap_batch tlb_ubc;
mm/internal.h +4 -0
@@ -1246,6 +1246,7 @@ void try_to_unmap_flush(void);void try_to_unmap_flush_dirty (void );void try_to_unmap_flush_takeoff (void );void flush_tlb_batched_pending (struct mm_struct *mm); +void reset_batch (struct tlbflush_unmap_batch *batch);void fold_batch (struct tlbflush_unmap_batch *dst,struct tlbflush_unmap_batch *src,bool reset);void fold_luf_batch (struct luf_batch *dst,struct luf_batch *src);#else
@@ -1261,6 +1262,9 @@ static inline void try_to_unmap_flush_takeoff(void)static inline void flush_tlb_batched_pending (struct mm_struct *mm) { } +static inline void reset_batch (struct tlbflush_unmap_batch *batch) +{ +}static inline void fold_batch (struct tlbflush_unmap_batch *dst,struct tlbflush_unmap_batch *src,bool reset) { }
mm/page_alloc.c +28 -4
@@ -668,9 +668,11 @@ bool luf_takeoff_start(void)*/ void luf_takeoff_end (void ) { +struct tlbflush_unmap_batch *tlb_ubc_takeoff = ¤t->tlb_ubc_takeoff;unsigned long flags;bool no_shootdown;bool outmost = false; +unsigned long cur_luf_ugen;local_irq_save (flags);VM_WARN_ON (!current->luf_takeoff_started);
@@ -697,10 +699,19 @@ void luf_takeoff_end(void)if (no_shootdown) goto out; + cur_luf_ugen = current->luf_ugen; + + current->luf_ugen =0 ; + +if (cur_luf_ugen &&arch_tlbbatch_diet (&tlb_ubc_takeoff->arch, cur_luf_ugen)) +reset_batch (tlb_ubc_takeoff); +try_to_unmap_flush_takeoff (); out: -if (outmost) +if (outmost) {VM_WARN_ON (current->luf_no_shootdown); +VM_WARN_ON (current->luf_ugen); + } }/*
@@ -757,6 +768,7 @@ bool luf_takeoff_check_and_fold(struct page *page)struct tlbflush_unmap_batch *tlb_ubc_takeoff = ¤t->tlb_ubc_takeoff;unsigned short luf_key =page_luf_key (page);struct luf_batch *lb; +unsigned long lb_ugen;unsigned long flags;/*
@@ -770,13 +782,25 @@ bool luf_takeoff_check_and_fold(struct page *page)if (!luf_key)return true; -if (current->luf_no_shootdown) -return false; - lb = &luf_batch[luf_key];read_lock_irqsave (&lb->lock, flags); + lb_ugen = lb->ugen; + +if (arch_tlbbatch_check_done (&lb->batch.arch, lb_ugen)) { +read_unlock_irqrestore (&lb->lock, flags); +return true; + } + +if (current->luf_no_shootdown) { +read_unlock_irqrestore (&lb->lock, flags); +return false; + } +fold_batch (tlb_ubc_takeoff, &lb->batch, false);read_unlock_irqrestore (&lb->lock, flags); + +if (!current->luf_ugen ||ugen_before (current->luf_ugen, lb_ugen)) + current->luf_ugen = lb_ugen; return true; }#endif
mm/rmap.c +44 -2
96 lines not on this page · raw message
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 18/26 mm/page_alloc: retry 3 times to take pcp pages on luf check failure
In reply to Byungchul Park--- mm/page_alloc.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-)
mm/page_alloc.c +20 -4
@@ -3339,6 +3339,12 @@ struct page *__rmqueue_pcplist(struct zone *zone, unsigned int order, {struct page *page; +/* +* give up taking page from pcp if it fails to take pcp page +* 3 times due to the tlb shootdownable issue. +*/ +int try_luf_pages =3 ; + do { if (list_empty(list)) { int batch = nr_pcp_alloc(pcp, zone, order);
@@ -3353,11 +3359,21 @@ struct page *__rmqueue_pcplist(struct zone *zone, unsigned int order,return NULL ; } - page = list_first_entry(list, struct page, pcp_list); -if (!luf_takeoff_check_and_fold (page)) +list_for_each_entry (page, list, pcp_list) { +if (luf_takeoff_check_and_fold (page)) { +list_del (&page->pcp_list); + pcp->count -=1 << order; +break ; + } +if (!--try_luf_pages) +return NULL ; + } + +/* +* If all the pages in the list fails... +*/ +if (list_entry_is_head (page, list, pcp_list))return NULL ; -list_del (&page->pcp_list); - pcp->count -=1 << order; }while (check_new_pages (page, order));return page;
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 19/26 mm: skip luf tlb flush for luf'd mm that already has been done
In reply to Byungchul ParkFault hander performs tlb flush pended by luf when a new pte becomes to have write permission, no matter whether tlb flush required has been performed or not.
By storing luf generation number, luf_ugen, in struct mm_struct, we can skip unnecessary tlb flush.
--- include/asm-generic/tlb.h | 2 +- include/linux/mm_types.h | 9 +++++ kernel/fork.c | 1 + kernel/sched/core.c | 2 +- mm/memory.c | 22 ++++++++++-- mm/pgtable-generic.c | 2 +- mm/rmap.c | 74 +++++++++++++++++++++++++++++++++++++-- 7 files changed, 104 insertions(+), 8 deletions(-)
include/asm-generic/tlb.h +1 -1
@@ -552,7 +552,7 @@ static inline void tlb_end_vma(struct mmu_gather *tlb, struct vm_area_struct *vm/* * Don't leave stale tlb entries for this vma. */ -luf_flush (0 ); +luf_flush_vma (vma);if (tlb->fullmm)return ;
include/linux/mm_types.h +9 -0
@@ -38,8 +38,10 @@ struct luf_batch {unsigned long ugen; rwlock_t lock; }; +void luf_batch_init (struct luf_batch *lb);#else struct luf_batch {}; +static inline void luf_batch_init (struct luf_batch *lb) {}#endif /*
@@ -1022,6 +1024,9 @@ struct mm_struct {* moving a PROT_NONE mapped page. */ atomic_t tlb_flush_pending; + +/* luf batch for this mm */ +struct luf_batch luf_batch;#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH /* See flush_tlb_batched_pending() */ atomic_t tlb_flush_batched;
@@ -1272,8 +1277,12 @@ extern void tlb_finish_mmu(struct mmu_gather *tlb);#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH )void luf_flush (unsigned short luf_key); +void luf_flush_mm (struct mm_struct *mm); +void luf_flush_vma (struct vm_area_struct *vma);#else static inline void luf_flush (unsigned short luf_key) {} +static inline void luf_flush_mm (struct mm_struct *mm) {} +static inline void luf_flush_vma (struct vm_area_struct *vma) {}#endif struct vm_fault ;
kernel/fork.c +1 -0
@@ -1268,6 +1268,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,memset (&mm->rss_stat,0 ,sizeof (mm->rss_stat));spin_lock_init (&mm->page_table_lock);spin_lock_init (&mm->arg_lock); +luf_batch_init (&mm->luf_batch);mm_init_cpumask (mm);mm_init_aio (mm);mm_init_owner (mm, p);
kernel/sched/core.c +1 -1
@@ -5225,7 +5225,7 @@ static struct rq *finish_task_switch(struct task_struct *prev)if (mm) {membarrier_mm_sync_core_before_usermode (mm);mmdrop_lazy_tlb_sched (mm); -luf_flush (0 ); +luf_flush_mm (mm); }if (unlikely (prev_state ==TASK_DEAD )) {
mm/memory.c +19 -3
@@ -6081,6 +6081,7 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,struct mm_struct *mm = vma->vm_mm;vm_fault_t ret;bool is_droppable; +struct address_space *mapping =NULL ;bool flush = false;__set_current_state (TASK_RUNNING );
@@ -6112,9 +6113,17 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,* should be considered. */ if (vma->vm_flags & (VM_WRITE |VM_MAYWRITE ) || - flags &FAULT_FLAG_WRITE ) + flags &FAULT_FLAG_WRITE ) { flush = true; +/* +* Doesn't care the !VM_SHARED cases because it won't +* update the pages that might be shared with others. +*/ +if (vma->vm_flags &VM_SHARED && vma->vm_file) + mapping = vma->vm_file->f_mapping; + } +if (unlikely (is_vm_hugetlb_page (vma))) ret =hugetlb_fault (vma->vm_mm, vma, address, flags);else
@@ -6149,8 +6158,15 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,/* * Ensure to clean stale tlb entries for this vma. */ -if (flush) -luf_flush (0 ); +if (flush) { +/* +* If it has a VM_SHARED mapping, all the mms involved +* should be luf_flush'ed. +*/ +if (mapping) +luf_flush (0 ); +luf_flush_mm (mm); + }return ret; }
mm/pgtable-generic.c +1 -1
@@ -100,7 +100,7 @@ pte_t ptep_clear_flush(struct vm_area_struct *vma, unsigned long address,if (pte_accessible (mm, pte))flush_tlb_page (vma, address);else -luf_flush (0 ); +luf_flush_vma (vma); return pte; }#endif
mm/rmap.c +72 -2
99 lines not on this page · raw message
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 20/26 mm, fs: skip tlb flushes for luf'd filemap that already has been done
In reply to Byungchul ParkFor luf'd filemap, tlb shootdown is performed when updating page cache, no matter whether tlb flushes required already has been done or not.
By storing luf meta data in struct address_space and updating the luf meta data properly, we can skip unnecessary tlb flush.
--- fs/inode.c | 1 + include/linux/fs.h | 4 ++- include/linux/mm_types.h | 2 ++ mm/memory.c | 4 +-- mm/rmap.c | 59 +++++++++++++++++++++++++--------------- mm/truncate.c | 14 +++++----- mm/vmscan.c | 2 +- 7 files changed, 53 insertions(+), 33 deletions(-)
fs/inode.c +1 -0
@@ -404,6 +404,7 @@ static void __address_space_init_once(struct address_space *mapping)init_rwsem (&mapping->i_mmap_rwsem);INIT_LIST_HEAD (&mapping->i_private_list);spin_lock_init (&mapping->i_private_lock); +luf_batch_init (&mapping->luf_batch); mapping->i_mmap =RB_ROOT_CACHED ; }
include/linux/fs.h +3 -1
@@ -461,6 +461,7 @@ extern const struct address_space_operations empty_aops;* @i_private_lock: For use by the owner of the address_space. * @i_private_list: For use by the owner of the address_space. * @i_private_data: For use by the owner of the address_space. +* @luf_batch: Data to track need of tlb flush by luf. */ struct address_space {struct inode *host;
@@ -482,6 +483,7 @@ struct address_space {struct list_head i_private_list;struct rw_semaphore i_mmap_rwsem;void * i_private_data; +struct luf_batch luf_batch; } __attribute__((aligned(sizeof(long)))) __randomize_layout;/* * On most architectures that alignment is already the case; but
@@ -508,7 +510,7 @@ static inline int mapping_write_begin(struct file *file,* Ensure to clean stale tlb entries for this mapping. */ if (!ret) -luf_flush (0 ); +luf_flush_mapping (mapping);return ret; }
include/linux/mm_types.h +2 -0
@@ -1279,10 +1279,12 @@ extern void tlb_finish_mmu(struct mmu_gather *tlb);void luf_flush (unsigned short luf_key);void luf_flush_mm (struct mm_struct *mm);void luf_flush_vma (struct vm_area_struct *vma); +void luf_flush_mapping (struct address_space *mapping);#else static inline void luf_flush (unsigned short luf_key) {}static inline void luf_flush_mm (struct mm_struct *mm) {}static inline void luf_flush_vma (struct vm_area_struct *vma) {} +static inline void luf_flush_mapping (struct address_space *mapping) {}#endif struct vm_fault ;
mm/memory.c +2 -2
@@ -6161,10 +6161,10 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,if (flush) {/* * If it has a VM_SHARED mapping, all the mms involved -* should be luf_flush'ed. +* in the struct address_space should be luf_flush'ed. */ if (mapping) -luf_flush (0 ); +luf_flush_mapping (mapping);luf_flush_mm (mm); }
mm/rmap.c +37 -22
131 lines not on this page · raw message
mm/truncate.c +7 -7
@@ -128,7 +128,7 @@ void folio_invalidate(struct folio *folio, size_t offset, size_t length)/* * Ensure to clean stale tlb entries for this mapping. */ -luf_flush (0 ); +luf_flush_mapping (folio->mapping); }EXPORT_SYMBOL_GPL (folio_invalidate);
@@ -170,7 +170,7 @@ int truncate_inode_folio(struct address_space *mapping, struct folio *folio)/* * Ensure to clean stale tlb entries for this mapping. */ -luf_flush (0 ); +luf_flush_mapping (mapping);return 0 ; }
@@ -220,7 +220,7 @@ bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end)/* * Ensure to clean stale tlb entries for this mapping. */ -luf_flush (0 ); +luf_flush_mapping (folio->mapping);if (!folio_test_large (folio))return true;
@@ -282,7 +282,7 @@ long mapping_evict_folio(struct address_space *mapping, struct folio *folio)/* * Ensure to clean stale tlb entries for this mapping. */ -luf_flush (0 ); +luf_flush_mapping (mapping);return ret; }
@@ -417,7 +417,7 @@ void truncate_inode_pages_range(struct address_space *mapping,/* * Ensure to clean stale tlb entries for this mapping. */ -luf_flush (0 ); +luf_flush_mapping (mapping); }EXPORT_SYMBOL (truncate_inode_pages_range);
@@ -537,7 +537,7 @@ unsigned long mapping_try_invalidate(struct address_space *mapping,/* * Ensure to clean stale tlb entries for this mapping. */ -luf_flush (0 ); +luf_flush_mapping (mapping);return count; }
@@ -704,7 +704,7 @@ int invalidate_inode_pages2_range(struct address_space *mapping,/* * Ensure to clean stale tlb entries for this mapping. */ -luf_flush (0 ); +luf_flush_mapping (mapping);return ret; }EXPORT_SYMBOL_GPL (invalidate_inode_pages2_range);
mm/vmscan.c +1 -1
@@ -836,7 +836,7 @@ long remove_mapping(struct address_space *mapping, struct folio *folio)/* * Ensure to clean stale tlb entries for this mapping. */ -luf_flush (0 ); +luf_flush_mapping (mapping);return ret; }
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 21/26 mm: perform luf tlb shootdown per zone in batched manner
In reply to Byungchul ParkEach luf page in buddy has its pending tlb shootdown information and performs the corresponding tlb shootdown on exit from buddy. However, every exit from buddy causes small but frequent IPIs. Even though total IPIs get reduced, unnecessary waits on conflict CPUs in IPI handler have been observed via perf profiling.
Thus, made it perfrom luf tlb shootdown per zone in batched manner when pages exit from buddy so as to avoid frequent IPIs.
--- include/linux/mm.h | 44 ++++- include/linux/mm_types.h | 19 +- include/linux/mmzone.h | 9 + include/linux/sched.h | 2 + mm/compaction.c | 10 +- mm/internal.h | 13 +- mm/mm_init.c | 5 + mm/page_alloc.c | 363 +++++++++++++++++++++++++++++++-------- mm/page_reporting.c | 9 +- mm/rmap.c | 6 +- 10 files changed, 383 insertions(+), 97 deletions(-)
include/linux/mm.h +41 -3
@@ -4161,12 +4161,16 @@ static inline int do_mseal(unsigned long start, size_t len_in, unsigned long fla }#endif -#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH )/* * luf_ugen will start with 2 so that 1 can be regarded as a passed one. */ #define LUF_UGEN_INIT 2 +/* +* zone_ugen will start with 2 so that 1 can be regarded as done. +*/ +#define ZONE_UGEN_INIT 2 +#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH )static inline bool ugen_before (unsigned long a,unsigned long b) {/*
@@ -4177,7 +4181,11 @@ static inline bool ugen_before(unsigned long a, unsigned long b)static inline unsigned long next_ugen (unsigned long ugen) { -if (ugen +1 ) +/* +* Avoid zero even in unsigned short range so as to treat +* '(unsigned short)ugen == 0' as invalid. +*/ +if ((unsigned short )(ugen +1 ))return ugen +1 ;/* * Avoid invalid ugen, zero.
@@ -4187,7 +4195,11 @@ static inline unsigned long next_ugen(unsigned long ugen)static inline unsigned long prev_ugen (unsigned long ugen) { -if (ugen -1 ) +/* +* Avoid zero even in unsigned short range so as to treat +* '(unsigned short)ugen == 0' as invalid. +*/ +if ((unsigned short )(ugen -1 ))return ugen -1 ;/* * Avoid invalid ugen, zero.
@@ -4195,4 +4207,30 @@ static inline unsigned long prev_ugen(unsigned long ugen)return ugen -2 ; }#endif + +/* +* return the biggest ugen but it should be before the real zone_ugen. +*/ +static inline unsigned long page_zone_ugen (struct zone *zone,struct page *page) +{ +unsigned long zone_ugen = zone->zone_ugen; +unsigned short short_zone_ugen = page->zone_ugen; +unsigned long cand1, cand2; + +if (!short_zone_ugen) +return 0 ; + + cand1 = (zone_ugen & ~(unsigned long )USHRT_MAX ) | short_zone_ugen; + cand2 = cand1 -USHRT_MAX -1 ; + +if (!ugen_before (zone_ugen, cand1)) +return cand1; + +return cand2; +} + +static inline void set_page_zone_ugen (struct page *page,unsigned short zone_ugen) +{ + page->zone_ugen = zone_ugen; +}#endif /* _LINUX_MM_H */
include/linux/mm_types.h +14 -5
@@ -132,11 +132,20 @@ struct page {*/ unsigned short order; -/* -* For tracking need of tlb flush, -* by luf(lazy unmap flush). -*/ -unsigned short luf_key; +union { +/* +* For tracking need of +* tlb flush, by +* luf(lazy unmap flush). +*/ +unsigned short luf_key; + +/* +* Casted zone_ugen with +* unsigned short. +*/ +unsigned short zone_ugen; + }; }; }; };
include/linux/mmzone.h +9 -0
@@ -117,6 +117,7 @@ extern int page_group_by_mobility_disabled;struct free_area {struct list_head free_list[MIGRATE_TYPES ];struct list_head pend_list[MIGRATE_TYPES ]; +unsigned long pend_zone_ugen[MIGRATE_TYPES ];unsigned long nr_free; };
@@ -998,6 +999,14 @@ struct zone {atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS ];/* Count pages that need tlb shootdown on allocation */ atomic_long_t nr_luf_pages; +/* Generation number for that tlb shootdown has been done */ +unsigned long zone_ugen_done; +/* Generation number to control zone batched tlb shootdown */ +unsigned long zone_ugen; +/* Approximate latest luf_ugen that have ever entered */ +unsigned long luf_ugen; +/* Accumulated tlb batch for this zone */ +struct tlbflush_unmap_batch zone_batch; }____cacheline_internodealigned_in_smp ;enum pgdat_flags {
include/linux/sched.h +2 -0
@@ -1378,6 +1378,8 @@ struct task_struct {int luf_no_shootdown;int luf_takeoff_started;unsigned long luf_ugen; +unsigned long zone_ugen; +unsigned long wait_zone_ugen;#endif struct tlbflush_unmap_batch tlb_ubc;
mm/compaction.c +5 -5
@@ -653,7 +653,7 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, goto isolate_fail; } -if (!luf_takeoff_check (page)) +if (!luf_takeoff_check (cc->zone, page)) goto isolate_fail;/* Found a free page, will break it into order-0 pages */
@@ -689,7 +689,7 @@ static unsigned long isolate_freepages_block(struct compact_control *cc,/* * Check and flush before using the pages taken off. */ -luf_takeoff_end (); +luf_takeoff_end (cc->zone);/* * Be careful to not go outside of the pageblock.
@@ -1611,7 +1611,7 @@ static void fast_isolate_freepages(struct compact_control *cc) order_scanned++; nr_scanned++; -if (unlikely (consider_pend && !luf_takeoff_check (freepage))) +if (unlikely (consider_pend && !luf_takeoff_check (cc->zone, freepage))) goto scan_next; pfn =page_to_pfn (freepage);
@@ -1679,7 +1679,7 @@ static void fast_isolate_freepages(struct compact_control *cc)/* * Check and flush before using the pages taken off. */ -luf_takeoff_end (); +luf_takeoff_end (cc->zone);/* Skip fast search if enough freepages isolated */ if (cc->nr_freepages >= cc->nr_migratepages)
@@ -2415,7 +2415,7 @@ static enum compact_result compact_finished(struct compact_control *cc)*/ luf_takeoff_start (); ret =__compact_finished (cc); -luf_takeoff_end (); +luf_takeoff_end (cc->zone);trace_mm_compaction_finished (cc->zone, cc->order, ret);if (ret ==COMPACT_NO_SUITABLE_PAGE )
mm/internal.h +6 -7
@@ -1595,10 +1595,10 @@ static inline void accept_page(struct page *page)#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH )extern struct luf_batch luf_batch[];bool luf_takeoff_start (void ); -void luf_takeoff_end (void ); +void luf_takeoff_end (struct zone *zone);bool luf_takeoff_no_shootdown (void ); -bool luf_takeoff_check (struct page *page); -bool luf_takeoff_check_and_fold (struct page *page); +bool luf_takeoff_check (struct zone *zone,struct page *page); +bool luf_takeoff_check_and_fold (struct zone *zone,struct page *page);static inline bool non_luf_pages_ok (struct zone *zone) {
@@ -1608,7 +1608,6 @@ static inline bool non_luf_pages_ok(struct zone *zone)return nr_free - nr_luf_pages > min_wm; } -unsigned short fold_unmap_luf (void );/*
@@ -1696,10 +1695,10 @@ static inline bool can_luf_vma(struct vm_area_struct *vma) }#else /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ static inline bool luf_takeoff_start (void ) {return false; } -static inline void luf_takeoff_end (void ) {} +static inline void luf_takeoff_end (struct zone *zone) {}static inline bool luf_takeoff_no_shootdown (void ) {return true; } -static inline bool luf_takeoff_check (struct page *page) {return true; } -static inline bool luf_takeoff_check_and_fold (struct page *page) {return true; } +static inline bool luf_takeoff_check (struct zone *zone,struct page *page) {return true; } +static inline bool luf_takeoff_check_and_fold (struct zone *zone,struct page *page) {return true; }static inline bool non_luf_pages_ok (struct zone *zone) {return true; }static inline unsigned short fold_unmap_luf (void ) {return 0 ; }
mm/mm_init.c +5 -0
@@ -1397,6 +1397,7 @@ static void __meminit zone_init_free_lists(struct zone *zone)for_each_migratetype_order (order, t) {INIT_LIST_HEAD (&zone->free_area[order].free_list[t]);INIT_LIST_HEAD (&zone->free_area[order].pend_list[t]); + zone->free_area[order].pend_zone_ugen[t] =ZONE_UGEN_INIT ; zone->free_area[order].nr_free =0 ; }
@@ -1404,6 +1405,10 @@ static void __meminit zone_init_free_lists(struct zone *zone)INIT_LIST_HEAD (&zone->unaccepted_pages);#endif atomic_long_set (&zone->nr_luf_pages,0 ); + zone->zone_ugen_done =ZONE_UGEN_INIT -1 ; + zone->zone_ugen =ZONE_UGEN_INIT ; + zone->luf_ugen =LUF_UGEN_INIT -1 ; +reset_batch (&zone->zone_batch); } void __meminit init_currently_empty_zone(struct zone *zone,
mm/page_alloc.c +291 -72
669 lines not on this page · raw message
mm/page_reporting.c +5 -4
@@ -118,7 +118,8 @@ page_reporting_drain(struct page_reporting_dev_info *prdev,/* * Ensure private is zero before putting into the -* allocator. +* allocator. tlb shootdown has already been performed +* at isolation. */ set_page_private (page,0 );
@@ -194,7 +195,7 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,if (PageReported (page))continue ; -if (unlikely (consider_pend && !luf_takeoff_check (page))) { +if (unlikely (consider_pend && !luf_takeoff_check (zone, page))) {VM_WARN_ON (1 );continue ; }
@@ -238,7 +239,7 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,/* * Check and flush before using the pages taken off. */ -luf_takeoff_end (); +luf_takeoff_end (zone);/* begin processing pages in local list */ err = prdev->report (prdev, sgl,PAGE_REPORTING_CAPACITY );
@@ -283,7 +284,7 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,/* * Check and flush before using the pages taken off. */ -luf_takeoff_end (); +luf_takeoff_end (zone);return err; }
mm/rmap.c +5 -1
@@ -650,7 +650,11 @@ static unsigned long new_luf_ugen(void) {unsigned long ugen =atomic_long_inc_return (&luf_ugen); -if (!ugen) +/* +* Avoid zero even in unsigned short range so as to treat +* '(unsigned short)ugen == 0' as invalid. +*/ +if (!(unsigned short )ugen) ugen =atomic_long_inc_return (&luf_ugen);return ugen;
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 22/26 mm/page_alloc: not allow to tlb shootdown if !preemptable() && non_luf_pages_ok()
In reply to Byungchul ParkDo not perform tlb shootdown if the context is in preempt disable and there are already enough non luf pages, not to hurt preemptibility.
--- mm/compaction.c | 6 +++--- mm/internal.h | 5 +++-- mm/page_alloc.c | 27 +++++++++++++++------------ mm/page_isolation.c | 2 +- mm/page_reporting.c | 4 ++-- 5 files changed, 24 insertions(+), 20 deletions(-)
mm/compaction.c +3 -3
@@ -605,7 +605,7 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, page =pfn_to_page (blockpfn); -luf_takeoff_start (); +luf_takeoff_start (cc->zone);/* Isolate free pages. */ for (; blockpfn < end_pfn; blockpfn += stride, page += stride) {int isolated;
@@ -1601,7 +1601,7 @@ static void fast_isolate_freepages(struct compact_control *cc)if (!area->nr_free)continue ; - can_shootdown =luf_takeoff_start (); + can_shootdown =luf_takeoff_start (cc->zone);spin_lock_irqsave (&cc->zone->lock, flags); freelist = &area->free_list[MIGRATE_MOVABLE ]; retry:
@@ -2413,7 +2413,7 @@ static enum compact_result compact_finished(struct compact_control *cc)* luf_takeoff_{start,end}() is required to identify whether * this compaction context is tlb shootdownable for luf'd pages. */ -luf_takeoff_start (); +luf_takeoff_start (cc->zone); ret =__compact_finished (cc);luf_takeoff_end (cc->zone);
mm/internal.h +3 -2
@@ -1594,7 +1594,7 @@ static inline void accept_page(struct page *page)#endif /* CONFIG_UNACCEPTED_MEMORY */ #if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH )extern struct luf_batch luf_batch[]; -bool luf_takeoff_start (void ); +bool luf_takeoff_start (struct zone *zone);void luf_takeoff_end (struct zone *zone);bool luf_takeoff_no_shootdown (void );bool luf_takeoff_check (struct zone *zone,struct page *page);
@@ -1608,6 +1608,7 @@ static inline bool non_luf_pages_ok(struct zone *zone)return nr_free - nr_luf_pages > min_wm; } +unsigned short fold_unmap_luf (void );/*
@@ -1694,7 +1695,7 @@ static inline bool can_luf_vma(struct vm_area_struct *vma) return true; }#else /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ -static inline bool luf_takeoff_start (void ) {return false; } +static inline bool luf_takeoff_start (struct zone *zone) {return false; }static inline void luf_takeoff_end (struct zone *zone) {}static inline bool luf_takeoff_no_shootdown (void ) {return true; }static inline bool luf_takeoff_check (struct zone *zone,struct page *page) {return true; }
mm/page_alloc.c +15 -12
@@ -623,22 +623,25 @@ compaction_capture(struct capture_control *capc, struct page *page,#endif /* CONFIG_COMPACTION */ #if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH ) -static bool no_shootdown_context (void ) +static bool no_shootdown_context (struct zone *zone) {/* -* If it performs with irq disabled, that might cause a deadlock. -* Avoid tlb shootdown in this case. +* Tries to avoid tlb shootdown if !preemptible(). However, it +* should be allowed under heavy memory pressure. */ +if (zone &&non_luf_pages_ok (zone)) +return !(preemptible () &&in_task ()); +return !(!irqs_disabled () &&in_task ()); }/* * Can be called with zone lock released and irq enabled. */ -bool luf_takeoff_start (void ) +bool luf_takeoff_start (struct zone *zone) {unsigned long flags; -bool no_shootdown =no_shootdown_context (); +bool no_shootdown =no_shootdown_context (zone);local_irq_save (flags);
@@ -2588,7 +2591,7 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac,* luf_takeoff_{start,end}() is required for * get_page_from_free_area() to use luf_takeoff_check(). */ -luf_takeoff_start (); +luf_takeoff_start (zone);spin_lock_irqsave (&zone->lock, flags);for (order =0 ; order <NR_PAGE_ORDERS ; order++) {struct free_area *area = &(zone->free_area[order]);
@@ -2829,7 +2832,7 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order,unsigned long flags;int i; -luf_takeoff_start (); +luf_takeoff_start (zone);spin_lock_irqsave (&zone->lock, flags);for (i =0 ; i < count; ++i) { struct page *page = __rmqueue(zone, order, migratetype,
@@ -3455,7 +3458,7 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, do { page = NULL; - luf_takeoff_start(); + luf_takeoff_start(zone); spin_lock_irqsave(&zone->lock, flags); if (alloc_flags & ALLOC_HIGHATOMIC) page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);
@@ -3600,7 +3603,7 @@ static struct page *rmqueue_pcplist(struct zone *preferred_zone,struct page *page;unsigned long __maybe_unused UP_flags; -luf_takeoff_start (); +luf_takeoff_start (NULL );/* spin_trylock may fail due to a parallel drain or IRQ reentrancy. */ pcp_trylock_prepare (UP_flags); pcp =pcp_spin_trylock (zone->per_cpu_pageset);
@@ -5229,7 +5232,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,if (unlikely (!zone)) goto failed; -luf_takeoff_start (); +luf_takeoff_start (NULL );/* spin_trylock may fail due to a parallel drain or IRQ reentrancy. */ pcp_trylock_prepare (UP_flags); pcp =pcp_spin_trylock (zone->per_cpu_pageset);
@@ -7418,7 +7421,7 @@ unsigned long __offline_isolated_pages(unsigned long start_pfn,offline_mem_sections (pfn, end_pfn); zone =page_zone (pfn_to_page (pfn)); -luf_takeoff_start (); +luf_takeoff_start (zone);spin_lock_irqsave (&zone->lock, flags);while (pfn < end_pfn) { page =pfn_to_page (pfn);
@@ -7536,7 +7539,7 @@ bool take_page_off_buddy(struct page *page)unsigned int order;bool ret = false; -luf_takeoff_start (); +luf_takeoff_start (zone);spin_lock_irqsave (&zone->lock, flags);for (order =0 ; order <NR_PAGE_ORDERS ; order++) {struct page *page_head = page - (pfn & ((1 << order) -1 ));
mm/page_isolation.c +1 -1
@@ -211,7 +211,7 @@ static void unset_migratetype_isolate(struct page *page, int migratetype)struct page *buddy; zone =page_zone (page); -luf_takeoff_start (); +luf_takeoff_start (zone);spin_lock_irqsave (&zone->lock, flags);if (!is_migrate_isolate_page (page)) goto out;
mm/page_reporting.c +2 -2
@@ -170,7 +170,7 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,if (free_area_empty (area, mt))return err; - can_shootdown =luf_takeoff_start (); + can_shootdown =luf_takeoff_start (zone);spin_lock_irq (&zone->lock);/*
@@ -250,7 +250,7 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,/* update budget to reflect call to report function */ budget--; -luf_takeoff_start (); +luf_takeoff_start (zone);/* reacquire zone lock and resume processing */ spin_lock_irq (&zone->lock);
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 23/26 mm: separate move/undo parts from migrate_pages_batch()
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that requires to use separated folio lists for its own handling during migration. Refactored migrate_pages_batch() so as to separate move/undo parts from migrate_pages_batch().
--- mm/migrate.c | 134 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 83 insertions(+), 51 deletions(-)
mm/migrate.c +83 -51
@@ -1695,6 +1695,81 @@ static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio,return nr_failed; } +static void migrate_folios_move (struct list_head *src_folios, +struct list_head *dst_folios, +free_folio_t put_new_folio,unsigned long private, +enum migrate_mode mode,int reason, +struct list_head *ret_folios, +struct migrate_pages_stats *stats, +int *retry,int *thp_retry,int *nr_failed, +int *nr_retry_pages) +{ +struct folio *folio, *folio2, *dst, *dst2; +bool is_thp; +int nr_pages; +int rc; + + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 =list_next_entry (dst, lru); +list_for_each_entry_safe (folio, folio2, src_folios, lru) { + is_thp =folio_test_large (folio) &&folio_test_pmd_mappable (folio); + nr_pages =folio_nr_pages (folio); + +cond_resched (); + + rc =migrate_folio_move (put_new_folio, private, + folio, dst, mode, + reason, ret_folios); +/* +* The rules are: +* Success: folio will be freed +* -EAGAIN: stay on the unmap_folios list +* Other errno: put on ret_folios list +*/ +switch (rc) { +case -EAGAIN : + *retry +=1 ; + *thp_retry += is_thp; + *nr_retry_pages += nr_pages; +break ; +case MIGRATEPAGE_SUCCESS : + stats->nr_succeeded += nr_pages; + stats->nr_thp_succeeded += is_thp; +break ; +default : + *nr_failed +=1 ; + stats->nr_thp_failed += is_thp; + stats->nr_failed_pages += nr_pages; +break ; + } + dst = dst2; + dst2 =list_next_entry (dst, lru); + } +} + +static void migrate_folios_undo (struct list_head *src_folios, +struct list_head *dst_folios, +free_folio_t put_new_folio,unsigned long private, +struct list_head *ret_folios) +{ +struct folio *folio, *folio2, *dst, *dst2; + + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 =list_next_entry (dst, lru); +list_for_each_entry_safe (folio, folio2, src_folios, lru) { +int old_page_state =0 ; +struct anon_vma *anon_vma =NULL ; + +__migrate_folio_extract (dst, &old_page_state, &anon_vma); +migrate_folio_undo_src (folio, old_page_state &PAGE_WAS_MAPPED , + anon_vma, true, ret_folios); +list_del (&dst->lru); +migrate_folio_undo_dst (dst, true, put_new_folio, private); + dst = dst2; + dst2 =list_next_entry (dst, lru); + } +} +/* * migrate_pages_batch() first unmaps folios in the from list as many as * possible, then move the unmapped folios.
@@ -1717,7 +1792,7 @@ static int migrate_pages_batch(struct list_head *from,int pass =0 ;bool is_thp = false;bool is_large = false; -struct folio *folio, *folio2, *dst =NULL , *dst2; +struct folio *folio, *folio2, *dst =NULL ;int rc, rc_saved =0 , nr_pages;LIST_HEAD (unmap_folios);LIST_HEAD (dst_folios);
@@ -1888,42 +1963,11 @@ static int migrate_pages_batch(struct list_head *from, thp_retry =0 ; nr_retry_pages =0 ; - dst = list_first_entry(&dst_folios, struct folio, lru); - dst2 =list_next_entry (dst, lru); -list_for_each_entry_safe (folio, folio2, &unmap_folios, lru) { - is_thp =folio_test_large (folio) &&folio_test_pmd_mappable (folio); - nr_pages =folio_nr_pages (folio); - -cond_resched (); - - rc =migrate_folio_move (put_new_folio, private, - folio, dst, mode, - reason, ret_folios); -/* -* The rules are: -* Success: folio will be freed -* -EAGAIN: stay on the unmap_folios list -* Other errno: put on ret_folios list -*/ -switch (rc) { -case -EAGAIN : - retry++; - thp_retry += is_thp; - nr_retry_pages += nr_pages; -break ; -case MIGRATEPAGE_SUCCESS : - stats->nr_succeeded += nr_pages; - stats->nr_thp_succeeded += is_thp; -break ; -default : - nr_failed++; - stats->nr_thp_failed += is_thp; - stats->nr_failed_pages += nr_pages; -break ; - } - dst = dst2; - dst2 =list_next_entry (dst, lru); - } +/* Move the unmapped folios */ +migrate_folios_move (&unmap_folios, &dst_folios, + put_new_folio, private, mode, reason, + ret_folios, stats, &retry, &thp_retry, + &nr_failed, &nr_retry_pages); } nr_failed += retry; stats->nr_thp_failed += thp_retry;
@@ -1932,20 +1976,8 @@ static int migrate_pages_batch(struct list_head *from, rc = rc_saved ? : nr_failed; out:/* Cleanup remaining folios */ - dst = list_first_entry(&dst_folios, struct folio, lru); - dst2 =list_next_entry (dst, lru); -list_for_each_entry_safe (folio, folio2, &unmap_folios, lru) { -int old_page_state =0 ; -struct anon_vma *anon_vma =NULL ; - -__migrate_folio_extract (dst, &old_page_state, &anon_vma); -migrate_folio_undo_src (folio, old_page_state &PAGE_WAS_MAPPED , - anon_vma, true, ret_folios); -list_del (&dst->lru); -migrate_folio_undo_dst (dst, true, put_new_folio, private); - dst = dst2; - dst2 =list_next_entry (dst, lru); - } +migrate_folios_undo (&unmap_folios, &dst_folios, + put_new_folio, private, ret_folios);return rc; }
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 24/26 mm/migrate: apply luf mechanism to unmapping during migration
In reply to Byungchul ParkA new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read only and were unmapped, since the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.
Applied the mechanism to unmapping during migration.
--- include/linux/mm.h | 2 ++ include/linux/rmap.h | 2 +- mm/migrate.c | 65 ++++++++++++++++++++++++++++++++++---------- mm/rmap.c | 15 ++++++---- mm/swap.c | 2 +- 5 files changed, 63 insertions(+), 23 deletions(-)
include/linux/mm.h +2 -0
@@ -1476,6 +1476,8 @@ static inline void folio_put(struct folio *folio)__folio_put (folio); } +void page_cache_release (struct folio *folio); +/** * folio_put_refs - Reduce the reference count on a folio. * @folio: The folio.
include/linux/rmap.h +1 -1
@@ -660,7 +660,7 @@ static inline int folio_try_share_anon_rmap_pmd(struct folio *folio,int folio_referenced (struct folio *,int is_locked,struct mem_cgroup *memcg,unsigned long *vm_flags); -void try_to_migrate (struct folio *folio,enum ttu_flags flags); +bool try_to_migrate (struct folio *folio,enum ttu_flags flags);void try_to_unmap (struct folio *,enum ttu_flags flags); int make_device_exclusive_range(struct mm_struct *mm, unsigned long start,
mm/migrate.c +50 -15
165 lines not on this page · raw message
mm/rmap.c +9 -6
@@ -2750,8 +2750,9 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,* * Tries to remove all the page table entries which are mapping this folio and * replace them with special swap entries. Caller must hold the folio lock. +* Return true if all the mappings are read-only, otherwise false. */ -void try_to_migrate (struct folio *folio,enum ttu_flags flags) +bool try_to_migrate (struct folio *folio,enum ttu_flags flags) {struct rmap_walk_control rwc = { .rmap_one = try_to_migrate_one,
@@ -2769,11 +2770,11 @@ void try_to_migrate(struct folio *folio, enum ttu_flags flags)*/ if (WARN_ON_ONCE (flags & ~(TTU_RMAP_LOCKED |TTU_SPLIT_HUGE_PMD |TTU_SYNC |TTU_BATCH_FLUSH ))) -return ; +return false;if (folio_is_zone_device (folio) && (!folio_is_device_private (folio) && !folio_is_device_coherent (folio))) -return ; +return false;/* * During exec, a temporary VMA is setup and later moved.
@@ -2793,10 +2794,12 @@ void try_to_migrate(struct folio *folio, enum ttu_flags flags)else rmap_walk (folio, &rwc); -if (can_luf_test ()) +if (can_luf_test ()) {fold_batch (tlb_ubc_luf, tlb_ubc_ro, true); -else -fold_batch (tlb_ubc, tlb_ubc_ro, true); +return true; + } +fold_batch (tlb_ubc, tlb_ubc_ro, true); +return false; }#ifdef CONFIG_DEVICE_PRIVATE
mm/swap.c +1 -1
@@ -84,7 +84,7 @@ static void __page_cache_release(struct folio *folio, struct lruvec **lruvecp,* This path almost never happens for VM activity - pages are normally freed * in batches. But it gets used by networking - and for compound pages. */ -static void page_cache_release (struct folio *folio) +void page_cache_release (struct folio *folio) {struct lruvec *lruvec =NULL ;unsigned long flags;
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 25/26 mm/vmscan: apply luf mechanism to unmapping during folio reclaim
In reply to Byungchul ParkA new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read only and were unmapped, since the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.
Applied the mechanism to unmapping during folio reclaim.
--- include/linux/rmap.h | 5 +++-- mm/rmap.c | 11 +++++++---- mm/vmscan.c | 37 ++++++++++++++++++++++++++++++++----- 3 files changed, 42 insertions(+), 11 deletions(-)
include/linux/rmap.h +3 -2
@@ -661,7 +661,7 @@ int folio_referenced(struct folio *, int is_locked, struct mem_cgroup *memcg, unsigned long *vm_flags);bool try_to_migrate (struct folio *folio,enum ttu_flags flags); -void try_to_unmap (struct folio *,enum ttu_flags flags); +bool try_to_unmap (struct folio *,enum ttu_flags flags); int make_device_exclusive_range(struct mm_struct *mm, unsigned long start, unsigned long end, struct page **pages,
@@ -794,8 +794,9 @@ static inline int folio_referenced(struct folio *folio, int is_locked,return 0 ; } -static inline void try_to_unmap (struct folio *folio,enum ttu_flags flags) +static inline bool try_to_unmap (struct folio *folio,enum ttu_flags flags) { +return false; }static inline int folio_mkclean (struct folio *folio)
mm/rmap.c +7 -4
@@ -2386,10 +2386,11 @@ static int folio_not_mapped(struct folio *folio)* Tries to remove all the page table entries which are mapping this * folio. It is the caller's responsibility to check if the folio is * still mapped if needed (use TTU_SYNC to prevent accounting races). +* Return true if all the mappings are read-only, otherwise false. * * Context: Caller must hold the folio lock. */ -void try_to_unmap (struct folio *folio,enum ttu_flags flags) +bool try_to_unmap (struct folio *folio,enum ttu_flags flags) {struct rmap_walk_control rwc = { .rmap_one = try_to_unmap_one,
@@ -2408,10 +2409,12 @@ void try_to_unmap(struct folio *folio, enum ttu_flags flags)else rmap_walk (folio, &rwc); -if (can_luf_test ()) +if (can_luf_test ()) {fold_batch (tlb_ubc_luf, tlb_ubc_ro, true); -else -fold_batch (tlb_ubc, tlb_ubc_ro, true); +return true; + } +fold_batch (tlb_ubc, tlb_ubc_ro, true); +return false; }/*
mm/vmscan.c +32 -5
@@ -1052,14 +1052,17 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, struct reclaim_stat *stat, bool ignore_references) {struct folio_batch free_folios; +struct folio_batch free_folios_luf;LIST_HEAD (ret_folios);LIST_HEAD (demote_folios);unsigned int nr_reclaimed =0 ;unsigned int pgactivate =0 ;bool do_demote_pass;struct swap_iocb *plug =NULL ; +unsigned short luf_key;folio_batch_init (&free_folios); +folio_batch_init (&free_folios_luf);memset (stat,0 ,sizeof (*stat));cond_resched (); do_demote_pass =can_demote (pgdat->node_id, sc);
@@ -1071,6 +1074,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,enum folio_references references =FOLIOREF_RECLAIM ;bool dirty, writeback;unsigned int nr_pages; +bool can_luf = false;cond_resched ();
@@ -1309,7 +1313,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (folio_test_large (folio)) flags |=TTU_SYNC ; -try_to_unmap (folio, flags); + can_luf =try_to_unmap (folio, flags);if (folio_mapped (folio)) { stat->nr_unmap_fail += nr_pages; if (!was_swapbacked &&
@@ -1453,6 +1457,8 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,* leave it off the LRU). */ nr_reclaimed += nr_pages; +if (can_luf) +luf_flush (fold_unmap_luf ());continue ; } }
@@ -1485,6 +1491,19 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, nr_reclaimed += nr_pages;folio_unqueue_deferred_split (folio); + +if (can_luf) { +if (folio_batch_add (&free_folios_luf, folio) ==0 ) { +mem_cgroup_uncharge_folios (&free_folios); +mem_cgroup_uncharge_folios (&free_folios_luf); + luf_key =fold_unmap_luf (); +try_to_unmap_flush (); +free_unref_folios (&free_folios,0 ); +free_unref_folios (&free_folios_luf, luf_key); + } +continue ; + } +if (folio_batch_add (&free_folios, folio) ==0 ) {mem_cgroup_uncharge_folios (&free_folios);try_to_unmap_flush ();
@@ -1519,9 +1538,21 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,list_add (&folio->lru, &ret_folios);VM_BUG_ON_FOLIO (folio_test_lru (folio) ||folio_test_unevictable (folio), folio); +if (can_luf) +luf_flush (fold_unmap_luf ()); }/* 'folio_list' is always empty here */ +/* +* Finalize this turn before demote_folio_list(). +*/ +mem_cgroup_uncharge_folios (&free_folios); +mem_cgroup_uncharge_folios (&free_folios_luf); + luf_key =fold_unmap_luf (); +try_to_unmap_flush (); +free_unref_folios (&free_folios,0 ); +free_unref_folios (&free_folios_luf, luf_key); +/* Migrate folios selected for demotion */ stat->nr_demoted =demote_folio_list (&demote_folios, pgdat); nr_reclaimed += stat->nr_demoted;
@@ -1554,10 +1585,6 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, pgactivate = stat->nr_activate[0 ] + stat->nr_activate[1 ]; -mem_cgroup_uncharge_folios (&free_folios); -try_to_unmap_flush (); -free_unref_folios (&free_folios,0 ); -list_splice (&ret_folios, folio_list);count_vm_events (PGACTIVATE , pgactivate);
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 26/26 mm/luf: implement luf debug feature
In reply to Byungchul ParkWe need luf debug feature to detect when luf goes wrong by any chance. As a RFC, suggest a simple implementation to report problematic situations by luf.
--- arch/riscv/include/asm/tlbflush.h | 3 + arch/riscv/mm/tlbflush.c | 35 ++++- arch/x86/include/asm/pgtable.h | 10 ++ arch/x86/include/asm/tlbflush.h | 3 + arch/x86/mm/pgtable.c | 10 ++ arch/x86/mm/tlb.c | 35 ++++- include/linux/highmem-internal.h | 5 + include/linux/mm.h | 20 ++- include/linux/mm_types.h | 16 +-- include/linux/mm_types_task.h | 16 +++ include/linux/sched.h | 5 + mm/highmem.c | 1 + mm/memory.c | 12 ++ mm/page_alloc.c | 34 ++++- mm/page_ext.c | 3 + mm/rmap.c | 229 ++++++++++++++++++++++++++++++ 16 files changed, 418 insertions(+), 19 deletions(-)
arch/riscv/include/asm/tlbflush.h +3 -0
@@ -69,6 +69,9 @@ bool arch_tlbbatch_check_done(struct arch_tlbflush_unmap_batch *batch, unsignedbool arch_tlbbatch_diet (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen);void arch_tlbbatch_mark_ugen (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen);void arch_mm_mark_ugen (struct mm_struct *mm,unsigned long ugen); +#ifdef CONFIG_LUF_DEBUG +extern void print_lufd_arch (void ); +#endif static inline void arch_tlbbatch_clear (struct arch_tlbflush_unmap_batch *batch) {
arch/riscv/mm/tlbflush.c +34 -1
@@ -216,6 +216,25 @@ static int __init luf_init_arch(void) }early_initcall (luf_init_arch); +#ifdef CONFIG_LUF_DEBUG +static DEFINE_SPINLOCK (luf_debug_lock); +#define lufd_lock (f) spin_lock_irqsave(&luf_debug_lock, (f)) +#define lufd_unlock (f) spin_unlock_irqrestore(&luf_debug_lock, (f)) + +void print_lufd_arch (void ) +{ +int cpu; + +pr_cont ("LUFD ARCH:" ); +for_each_cpu (cpu, cpu_possible_mask) +pr_cont (" %lu" ,atomic_long_read (per_cpu_ptr (&ugen_done, cpu))); +pr_cont ("\n" ); +} +#else +#define lufd_lock (f) do { (void)(f); } while(0) +#define lufd_unlock (f) do { (void)(f); } while(0) +#endif +/* * batch will not be updated. */
@@ -223,17 +242,22 @@ bool arch_tlbbatch_check_done(struct arch_tlbflush_unmap_batch *batch, unsigned long ugen) {int cpu; +unsigned long flags;if (!ugen) goto out; +lufd_lock (flags);for_each_cpu (cpu, &batch->cpumask) {unsigned long done; done =atomic_long_read (per_cpu_ptr (&ugen_done, cpu)); -if (ugen_before (done, ugen)) +if (ugen_before (done, ugen)) { +lufd_unlock (flags);return false; + } } +lufd_unlock (flags);return true; out:return cpumask_empty (&batch->cpumask);
@@ -243,10 +267,12 @@ bool arch_tlbbatch_diet(struct arch_tlbflush_unmap_batch *batch, unsigned long ugen) {int cpu; +unsigned long flags;if (!ugen) goto out; +lufd_lock (flags);for_each_cpu (cpu, &batch->cpumask) {unsigned long done;
@@ -254,6 +280,7 @@ bool arch_tlbbatch_diet(struct arch_tlbflush_unmap_batch *batch,if (!ugen_before (done, ugen))cpumask_clear_cpu (cpu, &batch->cpumask); } +lufd_unlock (flags); out:return cpumask_empty (&batch->cpumask); }
@@ -262,10 +289,12 @@ void arch_tlbbatch_mark_ugen(struct arch_tlbflush_unmap_batch *batch, unsigned long ugen) {int cpu; +unsigned long flags;if (!ugen)return ; +lufd_lock (flags);for_each_cpu (cpu, &batch->cpumask) {atomic_long_t *done =per_cpu_ptr (&ugen_done, cpu);unsigned long old =atomic_long_read (done);
@@ -283,15 +312,18 @@ void arch_tlbbatch_mark_ugen(struct arch_tlbflush_unmap_batch *batch,*/ atomic_long_cmpxchg (done, old, ugen); } +lufd_unlock (flags); }void arch_mm_mark_ugen (struct mm_struct *mm,unsigned long ugen) {int cpu; +unsigned long flags;if (!ugen)return ; +lufd_lock (flags);for_each_cpu (cpu,mm_cpumask (mm)) {atomic_long_t *done =per_cpu_ptr (&ugen_done, cpu);unsigned long old =atomic_long_read (done);
@@ -309,4 +341,5 @@ void arch_mm_mark_ugen(struct mm_struct *mm, unsigned long ugen)*/ atomic_long_cmpxchg (done, old, ugen); } +lufd_unlock (flags); }
arch/x86/include/asm/pgtable.h +10 -0
@@ -695,12 +695,22 @@ static inline pud_t pud_mkyoung(pud_t pud)return pud_set_flags (pud, _PAGE_ACCESSED); } +#ifdef CONFIG_LUF_DEBUG +pud_t pud_mkwrite (pud_t pud); +static inline pud_t __pud_mkwrite (pud_t pud) +{ + pud =pud_set_flags (pud, _PAGE_RW); + +return pud_clear_saveddirty (pud); +} +#else static inline pud_t pud_mkwrite (pud_t pud) { pud =pud_set_flags (pud, _PAGE_RW);return pud_clear_saveddirty (pud); } +#endif #ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY static inline int pte_soft_dirty (pte_t pte)
arch/x86/include/asm/tlbflush.h +3 -0
@@ -297,6 +297,9 @@ extern bool arch_tlbbatch_check_done(struct arch_tlbflush_unmap_batch *batch, unextern bool arch_tlbbatch_diet (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen);extern void arch_tlbbatch_mark_ugen (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen);extern void arch_mm_mark_ugen (struct mm_struct *mm,unsigned long ugen); +#ifdef CONFIG_LUF_DEBUG +extern void print_lufd_arch (void ); +#endif static inline void arch_tlbbatch_clear (struct arch_tlbflush_unmap_batch *batch) {
arch/x86/mm/pgtable.c +10 -0
@@ -901,6 +901,7 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) { +lufd_check_pages (pte_page (pte),0 );if (vma->vm_flags &VM_SHADOW_STACK )return pte_mkwrite_shstk (pte);
@@ -911,6 +912,7 @@ pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)pmd_t pmd_mkwrite (pmd_t pmd,struct vm_area_struct *vma) { +lufd_check_pages (pmd_page (pmd),PMD_ORDER );if (vma->vm_flags &VM_SHADOW_STACK )return pmd_mkwrite_shstk (pmd);
@@ -919,6 +921,14 @@ pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)return pmd_clear_saveddirty (pmd); } +#ifdef CONFIG_LUF_DEBUG +pud_t pud_mkwrite (pud_t pud) +{ +lufd_check_pages (pud_page (pud),PUD_ORDER ); +return __pud_mkwrite (pud); +} +#endif +void arch_check_zapped_pte (struct vm_area_struct *vma,pte_t pte) {/*
arch/x86/mm/tlb.c +34 -1
@@ -1253,6 +1253,25 @@ static int __init luf_init_arch(void) }early_initcall (luf_init_arch); +#ifdef CONFIG_LUF_DEBUG +static DEFINE_SPINLOCK (luf_debug_lock); +#define lufd_lock (f) spin_lock_irqsave(&luf_debug_lock, (f)) +#define lufd_unlock (f) spin_unlock_irqrestore(&luf_debug_lock, (f)) + +void print_lufd_arch (void ) +{ +int cpu; + +pr_cont ("LUFD ARCH:" ); +for_each_cpu (cpu, cpu_possible_mask) +pr_cont (" %lu" ,atomic_long_read (per_cpu_ptr (&ugen_done, cpu))); +pr_cont ("\n" ); +} +#else +#define lufd_lock (f) do { (void)(f); } while(0) +#define lufd_unlock (f) do { (void)(f); } while(0) +#endif +/* * batch will not be updated. */
@@ -1260,17 +1279,22 @@ bool arch_tlbbatch_check_done(struct arch_tlbflush_unmap_batch *batch, unsigned long ugen) {int cpu; +unsigned long flags;if (!ugen) goto out; +lufd_lock (flags);for_each_cpu (cpu, &batch->cpumask) {unsigned long done; done =atomic_long_read (per_cpu_ptr (&ugen_done, cpu)); -if (ugen_before (done, ugen)) +if (ugen_before (done, ugen)) { +lufd_unlock (flags);return false; + } } +lufd_unlock (flags);return true; out:return cpumask_empty (&batch->cpumask);
@@ -1280,10 +1304,12 @@ bool arch_tlbbatch_diet(struct arch_tlbflush_unmap_batch *batch, unsigned long ugen) {int cpu; +unsigned long flags;if (!ugen) goto out; +lufd_lock (flags);for_each_cpu (cpu, &batch->cpumask) {unsigned long done;
@@ -1291,6 +1317,7 @@ bool arch_tlbbatch_diet(struct arch_tlbflush_unmap_batch *batch,if (!ugen_before (done, ugen))cpumask_clear_cpu (cpu, &batch->cpumask); } +lufd_unlock (flags); out:return cpumask_empty (&batch->cpumask); }
@@ -1299,10 +1326,12 @@ void arch_tlbbatch_mark_ugen(struct arch_tlbflush_unmap_batch *batch, unsigned long ugen) {int cpu; +unsigned long flags;if (!ugen)return ; +lufd_lock (flags);for_each_cpu (cpu, &batch->cpumask) {atomic_long_t *done =per_cpu_ptr (&ugen_done, cpu);unsigned long old =atomic_long_read (done);
@@ -1320,15 +1349,18 @@ void arch_tlbbatch_mark_ugen(struct arch_tlbflush_unmap_batch *batch,*/ atomic_long_cmpxchg (done, old, ugen); } +lufd_unlock (flags); }void arch_mm_mark_ugen (struct mm_struct *mm,unsigned long ugen) {int cpu; +unsigned long flags;if (!ugen)return ; +lufd_lock (flags);for_each_cpu (cpu,mm_cpumask (mm)) {atomic_long_t *done =per_cpu_ptr (&ugen_done, cpu);unsigned long old =atomic_long_read (done);
@@ -1346,6 +1378,7 @@ void arch_mm_mark_ugen(struct mm_struct *mm, unsigned long ugen)*/ atomic_long_cmpxchg (done, old, ugen); } +lufd_unlock (flags); }void arch_tlbbatch_flush (struct arch_tlbflush_unmap_batch *batch)
include/linux/highmem-internal.h +5 -0
@@ -41,6 +41,7 @@ static inline void *kmap(struct page *page) {void *addr; +lufd_check_pages (page,0 );might_sleep ();if (!PageHighMem (page)) addr =page_address (page);
@@ -161,6 +162,7 @@ static inline struct page *kmap_to_page(void *addr)static inline void *kmap (struct page *page) { +lufd_check_pages (page,0 );might_sleep ();return page_address (page); }
@@ -177,11 +179,13 @@ static inline void kunmap(struct page *page)static inline void *kmap_local_page (struct page *page) { +lufd_check_pages (page,0 );return page_address (page); }static inline void *kmap_local_folio (struct folio *folio,size_t offset) { +lufd_check_folio (folio);return page_address (&folio->page) + offset; }
@@ -204,6 +208,7 @@ static inline void __kunmap_local(const void *addr)static inline void *kmap_atomic (struct page *page) { +lufd_check_pages (page,0 );if (IS_ENABLED (CONFIG_PREEMPT_RT ))migrate_disable ();else
include/linux/mm.h +19 -1
@@ -44,6 +44,24 @@ extern int sysctl_page_lock_unfairness;void mm_core_init (void );void init_mm_internals (void ); +#ifdef CONFIG_LUF_DEBUG +void lufd_check_folio (struct folio *f); +void lufd_check_pages (const struct page *p,unsigned int order); +void lufd_check_zone_pages (struct zone *zone,struct page *page,unsigned int order); +void lufd_check_queued_pages (void ); +void lufd_queue_page_for_check (struct page *page,int order); +void lufd_mark_folio (struct folio *f,unsigned short luf_key); +void lufd_mark_pages (struct page *p,unsigned int order,unsigned short luf_key); +#else +static inline void lufd_check_folio (struct folio *f) {} +static inline void lufd_check_pages (const struct page *p,unsigned int order) {} +static inline void lufd_check_zone_pages (struct zone *zone,struct page *page,unsigned int order) {} +static inline void lufd_check_queued_pages (void ) {} +static inline void lufd_queue_page_for_check (struct page *page,int order) {} +static inline void lufd_mark_folio (struct folio *f,unsigned short luf_key) {} +static inline void lufd_mark_pages (struct page *p,unsigned int order,unsigned short luf_key) {} +#endif +#ifndef CONFIG_NUMA /* Don't use mapnrs, do it properly */ extern unsigned long max_mapnr;
@@ -113,7 +131,7 @@ extern int mmap_rnd_compat_bits __read_mostly;#endif #ifndef page_to_virt -#define page_to_virt (x) __va(PFN_PHYS(page_to_pfn(x))) +#define page_to_virt (x) ({ lufd_check_pages(x, 0); __va(PFN_PHYS(page_to_pfn(x)));})#endif #ifndef lm_alias
include/linux/mm_types.h +4 -12
@@ -22,6 +22,10 @@#include <asm/mmu.h> +#ifdef CONFIG_LUF_DEBUG +extern struct page_ext_operations luf_debug_ops; +#endif +#ifndef AT_VECTOR_SIZE_ARCH #define AT_VECTOR_SIZE_ARCH 0#endif
@@ -32,18 +36,6 @@struct address_space ;struct mem_cgroup ; -#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH -struct luf_batch { -struct tlbflush_unmap_batch batch; -unsigned long ugen; -rwlock_t lock; -}; -void luf_batch_init (struct luf_batch *lb); -#else -struct luf_batch {}; -static inline void luf_batch_init (struct luf_batch *lb) {} -#endif -/* * Each physical page in the system has a struct page associated with * it to keep track of whatever it is we are using the page for at the
include/linux/mm_types_task.h +16 -0
@@ -9,6 +9,7 @@*/ #include <linux/types.h> +#include <linux/spinlock_types.h> #include <asm/page.h>
@@ -67,4 +68,19 @@ struct tlbflush_unmap_batch { #endif }; +#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH +struct luf_batch { +struct tlbflush_unmap_batch batch; +unsigned long ugen; +rwlock_t lock; +}; +void luf_batch_init (struct luf_batch *lb); +#else +struct luf_batch {}; +static inline void luf_batch_init (struct luf_batch *lb) {} +#endif + +#if defined(CONFIG_LUF_DEBUG ) +#define NR_LUFD_PAGES 512 +#endif #endif /* _LINUX_MM_TYPES_TASK_H */
include/linux/sched.h +5 -0
@@ -1380,6 +1380,11 @@ struct task_struct {unsigned long luf_ugen;unsigned long zone_ugen;unsigned long wait_zone_ugen; +#if defined(CONFIG_LUF_DEBUG ) +struct page *lufd_pages[NR_LUFD_PAGES ]; +int lufd_pages_order[NR_LUFD_PAGES ]; +int lufd_pages_nr; +#endif #endif struct tlbflush_unmap_batch tlb_ubc;
mm/highmem.c +1 -0
@@ -576,6 +576,7 @@ void *__kmap_local_page_prot(struct page *page, pgprot_t prot) {void *kmap; +lufd_check_pages (page,0 );/* * To broaden the usage of the actual kmap_local() machinery always map * pages when debugging is enabled and the architecture has no problems
mm/memory.c +12 -0
@@ -6124,6 +6124,18 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, mapping = vma->vm_file->f_mapping; } +#ifdef CONFIG_LUF_DEBUG +if (luf_flush) { +/* +* If it has a VM_SHARED mapping, all the mms involved +* in the struct address_space should be luf_flush'ed. +*/ +if (mapping) +luf_flush_mapping (mapping); +luf_flush_mm (mm); + } +#endif +if (unlikely (is_vm_hugetlb_page (vma))) ret =hugetlb_fault (vma->vm_mm, vma, address, flags);else
mm/page_alloc.c +30 -4
@@ -758,6 +758,8 @@ void luf_takeoff_end(struct zone *zone)VM_WARN_ON (current->zone_ugen);VM_WARN_ON (current->wait_zone_ugen); } + +lufd_check_queued_pages (); }/*
@@ -853,8 +855,10 @@ bool luf_takeoff_check_and_fold(struct zone *zone, struct page *page)struct luf_batch *lb;unsigned long lb_ugen; -if (!luf_key) +if (!luf_key) { +lufd_check_pages (page,buddy_order (page));return true; + } lb = &luf_batch[luf_key];read_lock_irqsave (&lb->lock, flags);
@@ -875,12 +879,15 @@ bool luf_takeoff_check_and_fold(struct zone *zone, struct page *page)if (!current->luf_ugen ||ugen_before (current->luf_ugen, lb_ugen)) current->luf_ugen = lb_ugen; +lufd_queue_page_for_check (page,buddy_order (page));return true; } zone_ugen =page_zone_ugen (zone, page); -if (!zone_ugen) +if (!zone_ugen) { +lufd_check_pages (page,buddy_order (page));return true; + }/* * Should not be zero since zone-zone_ugen has been updated in
@@ -888,17 +895,23 @@ bool luf_takeoff_check_and_fold(struct zone *zone, struct page *page)*/ VM_WARN_ON (!zone->zone_ugen); -if (!ugen_before (READ_ONCE (zone->zone_ugen_done), zone_ugen)) +if (!ugen_before (READ_ONCE (zone->zone_ugen_done), zone_ugen)) { +lufd_check_pages (page,buddy_order (page));return true; + }if (current->luf_no_shootdown)return false; +lufd_check_zone_pages (zone, page,buddy_order (page)); +/* * zone batched flush has been already set. */ -if (current->zone_ugen) +if (current->zone_ugen) { +lufd_queue_page_for_check (page,buddy_order (page));return true; + }/* * Others are already performing tlb shootdown for us. All we
@@ -933,6 +946,7 @@ bool luf_takeoff_check_and_fold(struct zone *zone, struct page *page)atomic_long_set (&zone->nr_luf_pages,0 ); fold_batch(tlb_ubc_takeoff, &zone->zone_batch, true); } +lufd_queue_page_for_check (page,buddy_order (page));return true; }#endif
@@ -1238,6 +1252,11 @@ static inline void __free_one_page(struct page *page, }else zone_ugen =page_zone_ugen (zone, page); +if (!zone_ugen) +lufd_check_pages (page, order); +else +lufd_check_zone_pages (zone, page, order); +while (order <MAX_PAGE_ORDER ) {int buddy_mt = migratetype;unsigned long buddy_zone_ugen;
@@ -1299,6 +1318,10 @@ static inline void __free_one_page(struct page *page,set_page_zone_ugen (page, zone_ugen); pfn = combined_pfn; order++; +if (!zone_ugen) +lufd_check_pages (page, order); +else +lufd_check_zone_pages (zone, page, order); } done_merging:
@@ -3201,6 +3224,8 @@ void free_unref_page(struct page *page, unsigned int order,unsigned long pfn =page_to_pfn (page);int migratetype; +lufd_mark_pages (page, order, luf_key); +if (!pcp_allowed_order (order)) {__free_pages_ok (page, order,FPI_NONE , luf_key);return ;
@@ -3253,6 +3278,7 @@ void free_unref_folios(struct folio_batch *folios, unsigned short luf_key)unsigned long pfn =folio_pfn (folio);unsigned int order =folio_order (folio); +lufd_mark_folio (folio, luf_key);if (!free_pages_prepare (&folio->page, order))continue ;/*
mm/page_ext.c +3 -0
@@ -89,6 +89,9 @@ static struct page_ext_operations *page_ext_ops[] __initdata = {#ifdef CONFIG_PAGE_TABLE_CHECK &page_table_check_ops,#endif +#ifdef CONFIG_LUF_DEBUG + &luf_debug_ops, +#endif };unsigned long page_ext_size;
mm/rmap.c +229 -0
235 lines not on this page · raw message
Signature · 1 line
2.17.1
On Thu, 20 Feb 2025 14:20:01 +0900 Byungchul Park <[email protected]>
Quoted 33 lines
To check luf's stability, I ran a heavy LLM inference workload consuming 210GiB over 7 days on a machine with 140GiB memory, and decided it's stable enough.
I'm posting the latest version so that anyone can try luf mechanism if wanted by any chance. However, I tagged RFC again because there are still issues that should be resolved to merge to mainline:
1. Even though system wide total cpu time for TLB shootdown is reduced over 95%, page allocation paths should take additional cpu time shifted from page reclaim to perform TLB shootdown.
2. We need luf debug feature to detect when luf goes wrong by any chance. I implemented just a draft version that checks the sanity on
mkwrite(),kmap(), and so on. I need to gather better ideas to improve the debug feature.---
Hi everyone,
While I'm working with a tiered memory system e.g. CXL memory, I have been facing migration overhead esp. tlb shootdown on promotion or demotion between different tiers. Yeah.. most tlb shootdowns on migration through hinting fault can be avoided thanks to Huang Ying's work, commit
4d4b6d66db("mm,unmap: avoid flushing tlb in batch if PTE is inaccessible").However, it's only for migration through hinting fault. I thought it'd be much better if we have a general mechanism to reduce all the tlb numbers that we can apply to any unmap code, that we normally believe tlb flush should be followed.
Quoted 7 lines
I'm suggesting a new mechanism, LUF(Lazy Unmap Flush), that defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read-only and were unmapped, as long as the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.
Given pcp or buddy, you are opening window for use after free which makes no sense in 99% cases.
Byungchul Park
reviewerQuoted 41 lines · Hillf Danton
On Thu, 20 Feb 2025 14:20:01 +0900 Byungchul Park <[email protected]>
Quoted 33 lines
To check luf's stability, I ran a heavy LLM inference workload consuming 210GiB over 7 days on a machine with 140GiB memory, and decided it's stable enough.
I'm posting the latest version so that anyone can try luf mechanism if wanted by any chance. However, I tagged RFC again because there are still issues that should be resolved to merge to mainline:
1. Even though system wide total cpu time for TLB shootdown is reduced over 95%, page allocation paths should take additional cpu time shifted from page reclaim to perform TLB shootdown.
2. We need luf debug feature to detect when luf goes wrong by any chance. I implemented just a draft version that checks the sanity on
mkwrite(),kmap(), and so on. I need to gather better ideas to improve the debug feature.---
Hi everyone,
While I'm working with a tiered memory system e.g. CXL memory, I have been facing migration overhead esp. tlb shootdown on promotion or demotion between different tiers. Yeah.. most tlb shootdowns on migration through hinting fault can be avoided thanks to Huang Ying's work, commit
4d4b6d66db("mm,unmap: avoid flushing tlb in batch if PTE is inaccessible").However, it's only for migration through hinting fault. I thought it'd be much better if we have a general mechanism to reduce all the tlb numbers that we can apply to any unmap code, that we normally believe tlb flush should be followed.
Quoted 7 lines
I'm suggesting a new mechanism, LUF(Lazy Unmap Flush), that defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read-only and were unmapped, as long as the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.
Quoted 2 lines
Given pcp or buddy, you are opening window for use after free which makes no sense in 99% cases.
It's kinda 'use(= read only) after free' but luf ensures the data of the interesting pages doesn't change. That's what luf works on.
Byungchul
Byungchul Park
reviewerQuoted 41 lines · Hillf Danton
On Thu, 20 Feb 2025 14:20:01 +0900 Byungchul Park <[email protected]>
Quoted 33 lines
To check luf's stability, I ran a heavy LLM inference workload consuming 210GiB over 7 days on a machine with 140GiB memory, and decided it's stable enough.
I'm posting the latest version so that anyone can try luf mechanism if wanted by any chance. However, I tagged RFC again because there are still issues that should be resolved to merge to mainline:
1. Even though system wide total cpu time for TLB shootdown is reduced over 95%, page allocation paths should take additional cpu time shifted from page reclaim to perform TLB shootdown.
2. We need luf debug feature to detect when luf goes wrong by any chance. I implemented just a draft version that checks the sanity on
mkwrite(),kmap(), and so on. I need to gather better ideas to improve the debug feature.---
Hi everyone,
While I'm working with a tiered memory system e.g. CXL memory, I have been facing migration overhead esp. tlb shootdown on promotion or demotion between different tiers. Yeah.. most tlb shootdowns on migration through hinting fault can be avoided thanks to Huang Ying's work, commit
4d4b6d66db("mm,unmap: avoid flushing tlb in batch if PTE is inaccessible").However, it's only for migration through hinting fault. I thought it'd be much better if we have a general mechanism to reduce all the tlb numbers that we can apply to any unmap code, that we normally believe tlb flush should be followed.
Quoted 7 lines
I'm suggesting a new mechanism, LUF(Lazy Unmap Flush), that defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read-only and were unmapped, as long as the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.
Quoted 2 lines
Given pcp or buddy, you are opening window for use after free which makes no sense in 99% cases.
Just in case that I don't understand what you meant and for better understanding, can you provide a simple and problematic example from the u-a-f?
Byungchul
Quoted 42 lines · On Thu, 20 Feb 2025 20:09:35 +0900 Byungchul Park wrote:
Quoted 41 lines · Hillf Danton
On Thu, 20 Feb 2025 14:20:01 +0900 Byungchul Park <[email protected]>
Quoted 40 lines
To check luf's stability, I ran a heavy LLM inference workload consuming 210GiB over 7 days on a machine with 140GiB memory, and decided it's stable enough.
I'm posting the latest version so that anyone can try luf mechanism if wanted by any chance. However, I tagged RFC again because there are still issues that should be resolved to merge to mainline:
1. Even though system wide total cpu time for TLB shootdown is reduced over 95%, page allocation paths should take additional cpu time shifted from page reclaim to perform TLB shootdown.
2. We need luf debug feature to detect when luf goes wrong by any chance. I implemented just a draft version that checks the sanity on
mkwrite(),kmap(), and so on. I need to gather better ideas to improve the debug feature.---
Hi everyone,
While I'm working with a tiered memory system e.g. CXL memory, I have been facing migration overhead esp. tlb shootdown on promotion or demotion between different tiers. Yeah.. most tlb shootdowns on migration through hinting fault can be avoided thanks to Huang Ying's work, commit
4d4b6d66db("mm,unmap: avoid flushing tlb in batch if PTE is inaccessible").However, it's only for migration through hinting fault. I thought it'd be much better if we have a general mechanism to reduce all the tlb numbers that we can apply to any unmap code, that we normally believe tlb flush should be followed.
I'm suggesting a new mechanism, LUF(Lazy Unmap Flush), that defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read-only and were unmapped, as long as the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.
Quoted 7 lines
Quoted 2 lines
Given pcp or buddy, you are opening window for use after free which makes no sense in 99% cases.
Just in case that I don't understand what you meant and for better understanding, can you provide a simple and problematic example from the u-a-f?
Tell us if it is illegal to commit rape without pregnancy in your home town?
PS defering flushing tlb [1,2] is no go.
Subject: Re: [PATCH v4 29/30] x86/mm, mm/vmalloc: Defer flush_tlb_kernel_range() targeting NOHZ_FULL CPUs
[1] https://lore.kernel.org/lkml/20250127155146.GB25757@willie-the-truck/
[2] https://lore.kernel.org/lkml/[email protected]/
Byungchul Park
reviewerQuoted 43 lines · Hillf Danton
Quoted 42 lines · On Thu, 20 Feb 2025 20:09:35 +0900 Byungchul Park wrote:
Quoted 41 lines · Hillf Danton
On Thu, 20 Feb 2025 14:20:01 +0900 Byungchul Park <[email protected]>
Quoted 40 lines
To check luf's stability, I ran a heavy LLM inference workload consuming 210GiB over 7 days on a machine with 140GiB memory, and decided it's stable enough.
I'm posting the latest version so that anyone can try luf mechanism if wanted by any chance. However, I tagged RFC again because there are still issues that should be resolved to merge to mainline:
1. Even though system wide total cpu time for TLB shootdown is reduced over 95%, page allocation paths should take additional cpu time shifted from page reclaim to perform TLB shootdown.
2. We need luf debug feature to detect when luf goes wrong by any chance. I implemented just a draft version that checks the sanity on
mkwrite(),kmap(), and so on. I need to gather better ideas to improve the debug feature.---
Hi everyone,
While I'm working with a tiered memory system e.g. CXL memory, I have been facing migration overhead esp. tlb shootdown on promotion or demotion between different tiers. Yeah.. most tlb shootdowns on migration through hinting fault can be avoided thanks to Huang Ying's work, commit
4d4b6d66db("mm,unmap: avoid flushing tlb in batch if PTE is inaccessible").However, it's only for migration through hinting fault. I thought it'd be much better if we have a general mechanism to reduce all the tlb numbers that we can apply to any unmap code, that we normally believe tlb flush should be followed.
I'm suggesting a new mechanism, LUF(Lazy Unmap Flush), that defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read-only and were unmapped, as long as the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Given pcp or buddy, you are opening window for use after free which makes no sense in 99% cases.
Just in case that I don't understand what you meant and for better understanding, can you provide a simple and problematic example from the u-a-f?
Tell us if it is illegal to commit rape without pregnancy in your home town?
Memory overcommit also looked cheating to someone like you. You definitely think it'd be totally non-sense that each task believes it can use its own full virtual space.
We say uaf is illegal only when it can cause access the free area without *appropriate permission*.
Quoted 1 line
PS defering flushing tlb [1,2] is no go.
I will check this shortly.
Byungchul
Quoted 4 lines
Subject: Re: [PATCH v4 29/30] x86/mm, mm/vmalloc: Defer
flush_tlb_kernel_range()targetingNOHZ_FULLCPUs
[1] https://lore.kernel.org/lkml/20250127155146.GB25757@willie-the-truck/
[2] https://lore.kernel.org/lkml/[email protected]/
Byungchul Park
reviewerQuoted 43 lines · Hillf Danton
Quoted 42 lines · On Thu, 20 Feb 2025 20:09:35 +0900 Byungchul Park wrote:
Quoted 41 lines · Hillf Danton
On Thu, 20 Feb 2025 14:20:01 +0900 Byungchul Park <[email protected]>
Quoted 40 lines
To check luf's stability, I ran a heavy LLM inference workload consuming 210GiB over 7 days on a machine with 140GiB memory, and decided it's stable enough.
I'm posting the latest version so that anyone can try luf mechanism if wanted by any chance. However, I tagged RFC again because there are still issues that should be resolved to merge to mainline:
1. Even though system wide total cpu time for TLB shootdown is reduced over 95%, page allocation paths should take additional cpu time shifted from page reclaim to perform TLB shootdown.
2. We need luf debug feature to detect when luf goes wrong by any chance. I implemented just a draft version that checks the sanity on
mkwrite(),kmap(), and so on. I need to gather better ideas to improve the debug feature.---
Hi everyone,
While I'm working with a tiered memory system e.g. CXL memory, I have been facing migration overhead esp. tlb shootdown on promotion or demotion between different tiers. Yeah.. most tlb shootdowns on migration through hinting fault can be avoided thanks to Huang Ying's work, commit
4d4b6d66db("mm,unmap: avoid flushing tlb in batch if PTE is inaccessible").However, it's only for migration through hinting fault. I thought it'd be much better if we have a general mechanism to reduce all the tlb numbers that we can apply to any unmap code, that we normally believe tlb flush should be followed.
I'm suggesting a new mechanism, LUF(Lazy Unmap Flush), that defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read-only and were unmapped, as long as the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Given pcp or buddy, you are opening window for use after free which makes no sense in 99% cases.
Just in case that I don't understand what you meant and for better understanding, can you provide a simple and problematic example from the u-a-f?
Tell us if it is illegal to commit rape without pregnancy in your home town?
+to Torvalds
Logical blame is welcome but I don't want to see potty-mouthed busters like him in Linux community any more. Please, ban him for better community.
Byungchul
Quoted 5 lines
PS defering flushing tlb [1,2] is no go.
Subject: Re: [PATCH v4 29/30] x86/mm, mm/vmalloc: Defer
flush_tlb_kernel_range()targetingNOHZ_FULLCPUs
[1] https://lore.kernel.org/lkml/20250127155146.GB25757@willie-the-truck/
[2] https://lore.kernel.org/lkml/[email protected]/
Matthew Wilcox
maintainerQuoted 6 lines · Hillf Danton
Quoted 4 lines · On Thu, 20 Feb 2025 20:09:35 +0900 Byungchul Park wrote:
Just in case that I don't understand what you meant and for better understanding, can you provide a simple and problematic example from the u-a-f?
Tell us if it is illegal to commit rape without pregnancy in your home town?
Hillf, this is unacceptable language. You need to apologise.
Steven Rostedt
reviewerQuoted 9 lines · Matthew Wilcox
Quoted 6 lines · Hillf Danton
Quoted 4 lines · On Thu, 20 Feb 2025 20:09:35 +0900 Byungchul Park wrote:
Just in case that I don't understand what you meant and for better understanding, can you provide a simple and problematic example from the u-a-f?
Tell us if it is illegal to commit rape without pregnancy in your home town?
Hillf, this is unacceptable language. You need to apologise.
Agreed. WTF Hillf? Where did that come from? Is this how you talk to your co-workers?
I'll tell you what would happen in my home town. If someone said that to a co-worker, they would likely be terminated.
-- Steve
Quoted 9 lines · Steven Rostedt
Quoted 8 lines · Matthew Wilcox
Quoted 6 lines · Hillf Danton
Quoted 4 lines · On Thu, 20 Feb 2025 20:09:35 +0900 Byungchul Park wrote:
Just in case that I don't understand what you meant and for better understanding, can you provide a simple and problematic example from the u-a-f?
Tell us if it is illegal to commit rape without pregnancy in your home town?
Quoted 7 lines
Quoted 1 line
Hillf, this is unacceptable language. You need to apologise.
Agreed. WTF Hillf? Where did that come from? Is this how you talk to your co-workers?
I'll tell you what would happen in my home town. If someone said that to a co-worker, they would likely be terminated.
I can't agree with the "this is a firing offence" approach.
We're a community, no one is employed by anyone else here; we work together because we have to and we have to figure out how to get along. We work via consensus, not appeals to authority.
However - language like that can _and has_ driven away valuable and long standing members of our community, so people will feel strongly about this.
Hillf, I'm going to share a story off list.
Steven Rostedt
reviewerQuoted 4 lines · On Thu, 20 Feb 2025 17:53:41 -0500 Kent Overstreet <[email protected]>…
Quoted 2 lines
I'll tell you what would happen in my home town. If someone said that to a co-worker, they would likely be terminated.
I can't agree with the "this is a firing offence" approach.
My point was, if this was in a company, it could very well be a firing offense.
Quoted 4 lines
We're a community, no one is employed by anyone else here; we work together because we have to and we have to figure out how to get along. We work via consensus, not appeals to authority.
As a community, yes, things are different. But we should not have to tolerate such language.
-- Steve
Quoted 9 lines · Steven Rostedt
Quoted 4 lines · On Thu, 20 Feb 2025 17:53:41 -0500 Kent Overstreet <[email protected]>…
Quoted 2 lines
I'll tell you what would happen in my home town. If someone said that to a co-worker, they would likely be terminated.
I can't agree with the "this is a firing offence" approach.
My point was, if this was in a company, it could very well be a firing offense.
Well, to a white color worker, yes. But to someone working a blue collor safety critical industry, that's going to come across as rather tame.
(And I do get annoyed when people get overly focused on language and forget that _we_ are a safety critical industry. To a first approximation, all the critical infrastructure throughout the world runs on Linux, stuff that doesn't is a rounding error, and all the testing and validation that exists only provides a safety factor. We have to have our shit together, and that does need to come first).
That aside - my point isn't about what should and shouldn't be allowed, it's just that norms are arbitrary and it's not the best argument if you want someone to change their behavior.
Quoted 6 lines
Quoted 3 lines
We're a community, no one is employed by anyone else here; we work together because we have to and we have to figure out how to get along. We work via consensus, not appeals to authority.
As a community, yes, things are different. But we should not have to tolerate such language.
Agreed.
And I think we're all aware at this point at how that sort of thing does drive people away, so best not take it so far people start to consider you a liability - or one way or another there's going to be an "or else".
This place functions by making people feel respected and valued for the work they do, so a degree of respect and consideration is required.
On Thu, 20 Feb 2025 10:09:45 -0500 Steven Rostedt <[email protected]>
Quoted 9 lines · Matthew Wilcox
Quoted 8 lines · Hillf Danton
Quoted 6 lines · On Thu, 20 Feb 2025 20:09:35 +0900 Byungchul Park wrote:
Quoted 4 lines
Just in case that I don't understand what you meant and for better understanding, can you provide a simple and problematic example from the u-a-f?
Tell us if it is illegal to commit rape without pregnancy in your home town?
Quoted 8 lines
Quoted 1 line
Hillf, this is unacceptable language. You need to apologise.
Agreed. WTF Hillf? Where did that come from? Is this how you talk to your co-workers?
I'll tell you what would happen in my home town. If someone said that to a co-worker, they would likely be terminated.
Interesting, I want to know if the three words, rape, pregnancy and WTK, could be used before judge in your hometown court by anyone like your lawyer.
Steven Rostedt
reviewerQuoted 5 lines · On Fri, 21 Feb 2025 07:25:02 +0800 Hillf Danton <[email protected]> wrote:
Quoted 3 lines
I'll tell you what would happen in my home town. If someone said that to a co-worker, they would likely be terminated.
Interesting, I want to know if the three words, rape, pregnancy and WTK, could be used before judge in your hometown court by anyone like your lawyer.
Hillf,
This isn't a court. And there's no reason to use the word "rape" in a technical conversation on the Linux kernel mailing list. Perhaps a person reading this was a victim of rape. How do you think that would make them feel? Welcomed to our community? Absolutely not. Which is why it's totally unacceptable.
-- Steve
On Thu, 20 Feb 2025 18:44:12 -0500 Steven Rostedt <[email protected]>
Quoted 13 lines · On Fri, 21 Feb 2025 07:25:02 +0800 Hillf Danton <[email protected]> wrote:
Quoted 5 lines
Quoted 3 lines
I'll tell you what would happen in my home town. If someone said that to a co-worker, they would likely be terminated.
Interesting, I want to know if the three words, rape, pregnancy and WTK, could be used before judge in your hometown court by anyone like your lawyer.
This isn't a court. And there's no reason to use the word "rape" in a technical conversation on the Linux kernel mailing list. Perhaps a person reading this was a victim of rape. How do you think that would make them feel? Welcomed to our community? Absolutely not. Which is why it's totally unacceptable.
There are NAK victims. Did you nak more than twice a week, Steve?
Quoted 8 lines · Hillf Danton
On Thu, 20 Feb 2025 18:44:12 -0500 Steven Rostedt <[email protected]>
Quoted 7 lines · On Fri, 21 Feb 2025 07:25:02 +0800 Hillf Danton <[email protected]> wrote:
Quoted 5 lines
Quoted 3 lines
I'll tell you what would happen in my home town. If someone said that to a co-worker, they would likely be terminated.
Interesting, I want to know if the three words, rape, pregnancy and WTK, could be used before judge in your hometown court by anyone like your lawyer.
Quoted 7 lines
Quoted 6 lines
This isn't a court. And there's no reason to use the word "rape" in a technical conversation on the Linux kernel mailing list. Perhaps a person reading this was a victim of rape. How do you think that would make them feel? Welcomed to our community? Absolutely not. Which is why it's totally unacceptable.
There are NAK victims. Did you nak more than twice a week, Steve?
Hillf,
This is not the way to work with your fellow developers in the community to express disagreements. I would recommend following up with an apology.
thanks,
greg k-h (On behalf of the Code of Conduct Committee)
On Sat, 22 Feb 2025 08:16:09 +0100 Greg KH <[email protected]>
Quoted 15 lines · Hillf Danton
Quoted 14 lines
On Thu, 20 Feb 2025 18:44:12 -0500 Steven Rostedt <[email protected]>
Quoted 13 lines · On Fri, 21 Feb 2025 07:25:02 +0800 Hillf Danton <[email protected]> wrote:
Quoted 5 lines
Quoted 3 lines
I'll tell you what would happen in my home town. If someone said that to a co-worker, they would likely be terminated.
Interesting, I want to know if the three words, rape, pregnancy and WTK, could be used before judge in your hometown court by anyone like your lawyer.
This isn't a court. And there's no reason to use the word "rape" in a technical conversation on the Linux kernel mailing list. Perhaps a person reading this was a victim of rape. How do you think that would make them feel? Welcomed to our community? Absolutely not. Which is why it's totally unacceptable.
Quoted 5 lines
Quoted 1 line
There are NAK victims. Did you nak more than twice a week, Steve?
This is not the way to work with your fellow developers in the community to express disagreements.
No comment because you are free to express disagreements.
Quoted 2 lines
I would recommend following up with an apology.
It would take some time for me to opt to follow/ignore what you recommended.
Quoted 16 lines · Hillf Danton
On Sat, 22 Feb 2025 08:16:09 +0100 Greg KH <[email protected]>
Quoted 15 lines · Hillf Danton
Quoted 14 lines
On Thu, 20 Feb 2025 18:44:12 -0500 Steven Rostedt <[email protected]>
Quoted 13 lines · On Fri, 21 Feb 2025 07:25:02 +0800 Hillf Danton <[email protected]> wrote:
Quoted 5 lines
Quoted 3 lines
I'll tell you what would happen in my home town. If someone said that to a co-worker, they would likely be terminated.
Interesting, I want to know if the three words, rape, pregnancy and WTK, could be used before judge in your hometown court by anyone like your lawyer.
This isn't a court. And there's no reason to use the word "rape" in a technical conversation on the Linux kernel mailing list. Perhaps a person reading this was a victim of rape. How do you think that would make them feel? Welcomed to our community? Absolutely not. Which is why it's totally unacceptable.
Quoted 6 lines
Quoted 5 lines
Quoted 1 line
There are NAK victims. Did you nak more than twice a week, Steve?
This is not the way to work with your fellow developers in the community to express disagreements.
No comment because you are free to express disagreements.
Disagreements are fine, but not with words like what you used, sorry, that was unacceptable and requires an apology.
Quoted 3 lines
Quoted 2 lines
I would recommend following up with an apology.
It would take some time for me to opt to follow/ignore what you recommended.
Please take the time to do so very soon.
thanks,
greg k-h (On behalf of the Code of Conduct Committee)
On Sat, 22 Feb 2025 14:57:43 +0100 Greg KH <[email protected]>
Quoted 19 lines · Hillf Danton
Quoted 18 lines
On Sat, 22 Feb 2025 08:16:09 +0100 Greg KH <[email protected]>
Quoted 17 lines · Hillf Danton
Quoted 15 lines
On Thu, 20 Feb 2025 18:44:12 -0500 Steven Rostedt <[email protected]>
Quoted 13 lines · On Fri, 21 Feb 2025 07:25:02 +0800 Hillf Danton <[email protected]> wrote:
Quoted 5 lines
Quoted 3 lines
I'll tell you what would happen in my home town. If someone said that to a co-worker, they would likely be terminated.
Interesting, I want to know if the three words, rape, pregnancy and WTK, could be used before judge in your hometown court by anyone like your lawyer.
This isn't a court. And there's no reason to use the word "rape" in a technical conversation on the Linux kernel mailing list. Perhaps a person reading this was a victim of rape. How do you think that would make them feel? Welcomed to our community? Absolutely not. Which is why it's totally unacceptable.
There are NAK victims. Did you nak more than twice a week, Steve?
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
This is not the way to work with your fellow developers in the community to express disagreements.
No comment because you are free to express disagreements.
Disagreements are fine, but not with words like what you used, sorry, that was unacceptable and requires an apology.
Feel free to show us how the three words, rape, pregnancy and WTK, are defined to be unacceptable, if you are correct.
Quoted 3 lines · Hillf Danton
Quoted 2 lines · On Thu, 20 Feb 2025 20:09:35 +0900 Byungchul Park wrote:
Quoted 1 line · Hillf Danton
On Thu, 20 Feb 2025 14:20:01 +0900 Byungchul Park <[email protected]>
[..]
Hillf,
The Code of Conduct Committee received reports about your conduct in this email discussion.
Link to email where the violation took place:
https://lore.kernel.org/lkml/[email protected]/
Our community works on trust and respect and has agreed to abide by the Code of Conduct:
Reference: https://docs.kernel.org/process/code-of-conduct.html
The Code of Conduct Committee has determined that your written abuse of another community member required action on your part to repair the damage to the individual and the community. You took insufficient action to restore the community's faith in having otherwise productive technical discussions without the fear of personal attacks.
Following the Code of Conduct Interpretation process the TAB has approved the following recommendation:
-- Restrict Hillf Danton's participation in the kernel development
process for 3 months.
- Scope: Ban Hillf Danton from Linux kernel mailing lists for a period of 3 months.
On Mon, 10 Mar 2025 16:24:09 -0700 Dan Williams <[email protected]>
Quoted 4 lines
Following the Code of Conduct Interpretation process the TAB has approved the following recommendation:
Illegal before court hearing in public.
Quoted 6 lines
-- Restrict Hillf Danton's participation in the kernel development
process for 3 months.- Scope: Ban Hillf Danton from Linux kernel mailing lists for a period of 3 months.
Quoted 27 lines · Dan Williams
Quoted 3 lines · Hillf Danton
Quoted 2 lines · On Thu, 20 Feb 2025 20:09:35 +0900 Byungchul Park wrote:
Quoted 1 line · Hillf Danton
On Thu, 20 Feb 2025 14:20:01 +0900 Byungchul Park <[email protected]>
[..]
Hillf,
The Code of Conduct Committee received reports about your conduct in this email discussion.
Link to email where the violation took place:
https://lore.kernel.org/lkml/[email protected]/
Our community works on trust and respect and has agreed to abide by the Code of Conduct:
Reference: https://docs.kernel.org/process/code-of-conduct.html
The Code of Conduct Committee has determined that your written abuse of another community member required action on your part to repair the damage to the individual and the community. You took insufficient action to restore the community's faith in having otherwise productive technical discussions without the fear of personal attacks.
Quoted 8 lines
Following the Code of Conduct Interpretation process the TAB has approved the following recommendation:
-- Restrict Hillf Danton's participation in the kernel development
process for 3 months.- Scope: Ban Hillf Danton from Linux kernel mailing lists for a period of 3 months.
Please ban this guy for another 3 months, as I have another case here[1]. This kind of random and insane personal attack is unacceptable.
[1] https://lore.kernel.org/all/[email protected]/#t
Quoted 1 line
Thanks
Barry
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 5 lines · Hillf Danton
Quoted 5 lines
Date: Mon, 10 Mar 2025 16:24:09 -0700 Dan Williams wrote:
Following the Code of Conduct Interpretation process the TAB has approved the following recommendation:
Quoted 7 lines
Quoted 6 lines
-- Restrict Hillf Danton's participation in the kernel development
process for 3 months.- Scope: Ban Hillf Danton from Linux kernel mailing lists for a period of 3 months.
Dan, the ban expires.
Acknowledged, lifted.
Quoted 3 lines · Byungchul Park
I'm posting the latest version so that anyone can try luf mechanism if wanted by any chance. However, I tagged RFC again because there are still issues that should be resolved to merge to mainline:
I don't see anything fundamentally different here from the last 11 versions. I think the entire approach is dangerous and basically makes things impossible to debug. It's not clear that some of the failure scenarios that I've brought up in the past have actually been fixed.
What I've said here still stands:
Quoted 9 lines
https://lore.kernel.org/all/[email protected]/
I think tglx would call all of this "tinkering". The approach to this series is to "fix" narrow, specific cases that reviewers point out, make it compile, then send it out again, hoping someone will apply it.
So, for me, until the approach to this series changes: NAK, for x86. Andrew, please don't take this series. Or, if you do, please drop the patch enabling it on x86.
I think I'd also like to stop being cc'd on this. If LUF is merged into mainline and proven to work on arm64 or riscv for a year, I'd be happy to take another look at enabling it on x86. I think that's just about the only thing that would make me reconsider.
Quoted 9 lines · Dave Hansen
Quoted 3 lines · Byungchul Park
I'm posting the latest version so that anyone can try luf mechanism if wanted by any chance. However, I tagged RFC again because there are still issues that should be resolved to merge to mainline:
I don't see anything fundamentally different here from the last 11 versions. I think the entire approach is dangerous and basically makes things impossible to debug. It's not clear that some of the failure scenarios that I've brought up in the past have actually been fixed.
Yes, and it's still an invasive change to the buddy allocator. IIRC at Plumbers the opinion in the audience was that there might be ways to improve the batching on unmap to reduce the flushes without such an invasive and potentially dangerous change? Has that been investigated?
Also "Rebase on akpm/mm.git mm-unstable(5a7056135b) as of Nov 22, 2024." is very outdated at this point?
Thanks,
Vlastimil
Quoted 17 lines
What I've said here still stands:
Quoted 9 lines
https://lore.kernel.org/all/[email protected]/
I think tglx would call all of this "tinkering". The approach to this series is to "fix" narrow, specific cases that reviewers point out, make it compile, then send it out again, hoping someone will apply it.
So, for me, until the approach to this series changes: NAK, for x86. Andrew, please don't take this series. Or, if you do, please drop the patch enabling it on x86.
I think I'd also like to stop being cc'd on this. If LUF is merged into mainline and proven to work on arm64 or riscv for a year, I'd be happy to take another look at enabling it on x86. I think that's just about the only thing that would make me reconsider.
Byungchul Park
reviewerQuoted 6 lines · Vlastimil Babka
Quoted 5 lines · Dave Hansen
Quoted 3 lines · Byungchul Park
I'm posting the latest version so that anyone can try luf mechanism if wanted by any chance. However, I tagged RFC again because there are still issues that should be resolved to merge to mainline:
Quoted 6 lines
Quoted 4 lines
I don't see anything fundamentally different here from the last 11 versions. I think the entire approach is dangerous and basically makes things impossible to debug. It's not clear that some of the failure scenarios that I've brought up in the past have actually been fixed.
Yes, and it's still an invasive change to the buddy allocator.
Didn't want.. but admit.
Quoted 3 lines
IIRC at Plumbers the opinion in the audience was that there might be ways to improve the batching on unmap to reduce the flushes without such an invasive and potentially dangerous change? Has that been investigated?
Sure. I tried like, by holding those pages not freed until either no one accesses the interesting pages or memory pressure is high. However, unfortunately it was super hard to fix performance degradation by the number of page reclaim increased due to the unfreed pages.
Quoted 2 lines
Also "Rebase on akpm/mm.git mm-unstable(5a7056135b) as of Nov 22, 2024." is very outdated at this point?
Sorry for that. I will rebase and share.
Byungchul
Quoted 21 lines
Thanks,
VlastimilQuoted 17 lines
What I've said here still stands:
Quoted 9 lines
https://lore.kernel.org/all/[email protected]/
I think tglx would call all of this "tinkering". The approach to this series is to "fix" narrow, specific cases that reviewers point out, make it compile, then send it out again, hoping someone will apply it.
So, for me, until the approach to this series changes: NAK, for x86. Andrew, please don't take this series. Or, if you do, please drop the patch enabling it on x86.
I think I'd also like to stop being cc'd on this. If LUF is merged into mainline and proven to work on arm64 or riscv for a year, I'd be happy to take another look at enabling it on x86. I think that's just about the only thing that would make me reconsider.
Byungchul Park
reviewerRFC v12 rebased on v6.14-rc4
Quoted 25 lines · Byungchul Park
Quoted 6 lines · Vlastimil Babka
Quoted 5 lines · Dave Hansen
Quoted 3 lines · Byungchul Park
I'm posting the latest version so that anyone can try luf mechanism if wanted by any chance. However, I tagged RFC again because there are still issues that should be resolved to merge to mainline:
Quoted 6 lines
Quoted 4 lines
I don't see anything fundamentally different here from the last 11 versions. I think the entire approach is dangerous and basically makes things impossible to debug. It's not clear that some of the failure scenarios that I've brought up in the past have actually been fixed.
Yes, and it's still an invasive change to the buddy allocator.
Didn't want.. but admit.
Quoted 3 lines
IIRC at Plumbers the opinion in the audience was that there might be ways to improve the batching on unmap to reduce the flushes without such an invasive and potentially dangerous change? Has that been investigated?
Sure. I tried like, by holding those pages not freed until either no one accesses the interesting pages or memory pressure is high. However, unfortunately it was super hard to fix performance degradation by the number of page reclaim increased due to the unfreed pages.
Quoted 4 lines
Quoted 2 lines
Also "Rebase on akpm/mm.git mm-unstable(5a7056135b) as of Nov 22, 2024." is very outdated at this point?
Sorry for that. I will rebase and share.
This is the same patch set but rebased on v6.14-rc4.
Byungchul
Quoted 23 lines
Byungchul
Quoted 21 lines
Thanks,
VlastimilQuoted 17 lines
What I've said here still stands:
Quoted 9 lines
https://lore.kernel.org/all/[email protected]/
I think tglx would call all of this "tinkering". The approach to this series is to "fix" narrow, specific cases that reviewers point out, make it compile, then send it out again, hoping someone will apply it.
So, for me, until the approach to this series changes: NAK, for x86. Andrew, please don't take this series. Or, if you do, please drop the patch enabling it on x86.
I think I'd also like to stop being cc'd on this. If LUF is merged into mainline and proven to work on arm64 or riscv for a year, I'd be happy to take another look at enabling it on x86. I think that's just about the only thing that would make me reconsider.
Byungchul Park
reviewerRFC PATCH v12 1/25 x86/tlb: add APIs manipulating tlb batch's arch data
A new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read-only and were unmapped, since the contents of the folios wouldn't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.
This is a preparation for the mechanism that needs to recognize read-only tlb entries by separating tlb batch arch data into two, one is for read-only entries and the other is for writable ones, and merging those two when needed.
It also optimizes tlb shootdown by skipping CPUs that have already performed tlb flush needed since. To support it, added APIs manipulating arch data for x86.
--- arch/x86/include/asm/tlbflush.h | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
arch/x86/include/asm/tlbflush.h +24 -0
@@ -5,6 +5,7 @@#include <linux/mm_types.h> #include <linux/mmu_notifier.h> #include <linux/sched.h> +#include <linux/cpumask.h> #include <asm/processor.h> #include <asm/cpufeature.h>
@@ -294,6 +295,29 @@ static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm)extern void arch_tlbbatch_flush (struct arch_tlbflush_unmap_batch *batch); +static inline void arch_tlbbatch_clear (struct arch_tlbflush_unmap_batch *batch) +{ +cpumask_clear (&batch->cpumask); +} + +static inline void arch_tlbbatch_fold (struct arch_tlbflush_unmap_batch *bdst, +struct arch_tlbflush_unmap_batch *bsrc) +{ +cpumask_or (&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + +static inline bool arch_tlbbatch_need_fold (struct arch_tlbflush_unmap_batch *batch, +struct mm_struct *mm) +{ +return !cpumask_subset (mm_cpumask (mm), &batch->cpumask); +} + +static inline bool arch_tlbbatch_done (struct arch_tlbflush_unmap_batch *bdst, +struct arch_tlbflush_unmap_batch *bsrc) +{ +return !cpumask_andnot (&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} +static inline bool pte_flags_need_flush (unsigned long oldflags,unsigned long newflags,bool ignore_access)
base-commit: d082ecbc71e9e0bf49883ee4afd435a77a5101b6
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 2/25 arm64/tlbflush: add APIs manipulating tlb batch's arch data
A new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read only and were unmapped, since the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.
This is a preparation for the mechanism that requires to manipulate tlb batch's arch data. Even though arm64 does nothing for tlb things, arch with CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH should provide the APIs.
--- arch/arm64/include/asm/tlbflush.h | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
arch/arm64/include/asm/tlbflush.h +27 -0
@@ -354,6 +354,33 @@ static inline void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)dsb (ish); } +static inline void arch_tlbbatch_clear (struct arch_tlbflush_unmap_batch *batch) +{ +/* nothing to do */ +} + +static inline void arch_tlbbatch_fold (struct arch_tlbflush_unmap_batch *bdst, +struct arch_tlbflush_unmap_batch *bsrc) +{ +/* nothing to do */ +} + +static inline bool arch_tlbbatch_need_fold (struct arch_tlbflush_unmap_batch *batch, +struct mm_struct *mm) +{ +/* +* Nothing is needed in this architecture. +*/ +return false; +} + +static inline bool arch_tlbbatch_done (struct arch_tlbflush_unmap_batch *bdst, +struct arch_tlbflush_unmap_batch *bsrc) +{ +/* Kernel can consider tlb batch always has been done. */ +return true; +} +/* * This is meant to avoid soft lock-ups on large TLB flushing ranges and not * necessarily a performance improvement.
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 3/25 riscv/tlb: add APIs manipulating tlb batch's arch data
In reply to Byungchul ParkA new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read only and were unmapped, since the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.
This is a preparation for the mechanism that needs to recognize read-only tlb entries by separating tlb batch arch data into two, one is for read-only entries and the other is for writable ones, and merging those two when needed.
It also optimizes tlb shootdown by skipping CPUs that have already performed tlb flush needed since. To support it, added APIs manipulating arch data for riscv.
--- arch/riscv/include/asm/tlbflush.h | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)
arch/riscv/include/asm/tlbflush.h +28 -0
@@ -8,6 +8,7 @@#define _ASM_RISCV_TLBFLUSH_H#include <linux/mm_types.h> +#include <linux/cpumask.h> #include <asm/smp.h> #include <asm/errata_list.h>
@@ -65,6 +66,33 @@ void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch,void arch_flush_tlb_batched_pending (struct mm_struct *mm);void arch_tlbbatch_flush (struct arch_tlbflush_unmap_batch *batch); +static inline void arch_tlbbatch_clear (struct arch_tlbflush_unmap_batch *batch) +{ +cpumask_clear (&batch->cpumask); + +} + +static inline void arch_tlbbatch_fold (struct arch_tlbflush_unmap_batch *bdst, +struct arch_tlbflush_unmap_batch *bsrc) +{ +cpumask_or (&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); + +} + +static inline bool arch_tlbbatch_need_fold (struct arch_tlbflush_unmap_batch *batch, +struct mm_struct *mm) +{ +return !cpumask_subset (mm_cpumask (mm), &batch->cpumask); + +} + +static inline bool arch_tlbbatch_done (struct arch_tlbflush_unmap_batch *bdst, +struct arch_tlbflush_unmap_batch *bsrc) +{ +return !cpumask_andnot (&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); + +} +extern unsigned long tlb_flush_all_threshold;#else /* CONFIG_MMU */ #define local_flush_tlb_all () do { } while (0)
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 4/25 x86/tlb, riscv/tlb, mm/rmap: separate arch_tlbbatch_clear() out of arch_tlbbatch_flush()
In reply to Byungchul ParkA new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read only and were unmapped, since the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.
This is a preparation for the mechanism that requires to avoid redundant tlb flush by manipulating tlb batch's arch data. To achieve that, we need to separate the part clearing the tlb batch's arch data out of arch_tlbbatch_flush().
--- arch/riscv/mm/tlbflush.c | 1 - arch/x86/mm/tlb.c | 2 -- mm/rmap.c | 1 + 3 files changed, 1 insertion(+), 3 deletions(-)
arch/riscv/mm/tlbflush.c +0 -1
@@ -201,5 +201,4 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) {__flush_tlb_range (&batch->cpumask,FLUSH_TLB_NO_ASID ,0 ,FLUSH_TLB_MAX_SIZE ,PAGE_SIZE ); -cpumask_clear (&batch->cpumask); }
arch/x86/mm/tlb.c +0 -2
@@ -1292,8 +1292,6 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)local_irq_enable (); } -cpumask_clear (&batch->cpumask); -put_flush_tlb_info ();put_cpu (); }
mm/rmap.c +1 -0
@@ -648,6 +648,7 @@ void try_to_unmap_flush(void)return ;arch_tlbbatch_flush (&tlb_ubc->arch); +arch_tlbbatch_clear (&tlb_ubc->arch); tlb_ubc->flush_required = false; tlb_ubc->writable = false; }
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 5/25 mm/buddy: make room for a new variable, luf_key, in struct page
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that tracks need of tlb flush for each page residing in buddy.
Since the private field in struct page is used only to store page order in buddy, ranging from 0 to MAX_PAGE_ORDER, that can be covered with unsigned short. So splitted it into two smaller ones, order and luf_key, so that the both can be used in buddy at the same time.
--- include/linux/mm_types.h | 42 +++++++++++++++++++++++++++++++++------- mm/internal.h | 4 ++-- mm/page_alloc.c | 2 +- 3 files changed, 38 insertions(+), 10 deletions(-)
include/linux/mm_types.h +35 -7
@@ -106,13 +106,27 @@ struct page {pgoff_t index;/* Our offset within mapping. */ unsigned long share;/* share count for fsdax */ }; -/** -* @private: Mapping-private opaque data. -* Usually used for buffer_heads if PagePrivate. -* Used for swp_entry_t if swapcache flag set. -* Indicates order in the buddy system if PageBuddy. -*/ -unsigned long private; +union { +/** +* @private: Mapping-private opaque data. +* Usually used for buffer_heads if PagePrivate. +* Used for swp_entry_t if swapcache flag set. +* Indicates order in the buddy system if PageBuddy. +*/ +unsigned long private; +struct { +/* +* Indicates order in the buddy system if PageBuddy. +*/ +unsigned short order; + +/* +* For tracking need of tlb flush, +* by luf(lazy unmap flush). +*/ +unsigned short luf_key; + }; + }; };struct {/* page_pool used by netstack */ /**
@@ -566,6 +580,20 @@ static inline void set_page_private(struct page *page, unsigned long private) page->private = private; } +#define page_buddy_order (page) ((page)->order) + +static inline void set_page_buddy_order (struct page *page,unsigned int order) +{ + page->order = (unsigned short )order; +} + +#define page_luf_key (page) ((page)->luf_key) + +static inline void set_page_luf_key (struct page *page,unsigned short luf_key) +{ + page->luf_key = luf_key; +} +static inline void *folio_get_private (struct folio *folio) {return folio->private;
mm/internal.h +2 -2
@@ -543,7 +543,7 @@ struct alloc_context {static inline unsigned int buddy_order (struct page *page) {/* PageBuddy() must be checked by the caller */ -return page_private (page); +return page_buddy_order (page); }/*
@@ -557,7 +557,7 @@ static inline unsigned int buddy_order(struct page *page)* times, potentially observing different values in the tests and the actual * use of the result. */ -#define buddy_order_unsafe (page) READ_ONCE(page_private(page)) +#define buddy_order_unsafe (page) READ_ONCE(page_buddy_order(page))/* * This function checks whether a page is free && is the buddy
mm/page_alloc.c +1 -1
@@ -576,7 +576,7 @@ void prep_compound_page(struct page *page, unsigned int order)static inline void set_buddy_order (struct page *page,unsigned int order) { -set_page_private (page, order); +set_page_buddy_order (page, order);__SetPageBuddy (page); }
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 6/25 mm: move should_skip_kasan_poison() to mm/internal.h
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that needs to use should_skip_kasan_poison() function in mm/internal.h.
--- mm/internal.h | 47 +++++++++++++++++++++++++++++++++++++++++++++++ mm/page_alloc.c | 47 ----------------------------------------------- 2 files changed, 47 insertions(+), 47 deletions(-)
mm/internal.h +47 -0
@@ -1051,8 +1051,55 @@ static inline void vunmap_range_noflush(unsigned long start, unsigned long end)DECLARE_STATIC_KEY_TRUE (deferred_pages);bool __init deferred_grow_zone (struct zone *zone,unsigned int order); + +static inline bool deferred_pages_enabled (void ) +{ +return static_branch_unlikely (&deferred_pages); +} +#else +static inline bool deferred_pages_enabled (void ) +{ +return false; +}#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ +/* +* Skip KASAN memory poisoning when either: +* +* 1. For generic KASAN: deferred memory initialization has not yet completed. +* Tag-based KASAN modes skip pages freed via deferred memory initialization +* using page tags instead (see below). +* 2. For tag-based KASAN modes: the page has a match-all KASAN tag, indicating +* that error detection is disabled for accesses via the page address. +* +* Pages will have match-all tags in the following circumstances: +* +* 1. Pages are being initialized for the first time, including during deferred +* memory init; see the call to page_kasan_tag_reset in __init_single_page. +* 2. The allocation was not unpoisoned due to __GFP_SKIP_KASAN, with the +* exception of pages unpoisoned by kasan_unpoison_vmalloc. +* 3. The allocation was excluded from being checked due to sampling, +* see the call to kasan_unpoison_pages. +* +* Poisoning pages during deferred memory init will greatly lengthen the +* process and cause problem in large memory systems as the deferred pages +* initialization is done with interrupt disabled. +* +* Assuming that there will be no reference to those newly initialized +* pages before they are ever allocated, this should have no effect on +* KASAN memory tracking as the poison will be properly inserted at page +* allocation time. The only corner case is when pages are allocated by +* on-demand allocation and then freed again before the deferred pages +* initialization is done, but this is not likely to happen. +*/ +static inline bool should_skip_kasan_poison (struct page *page) +{ +if (IS_ENABLED (CONFIG_KASAN_GENERIC )) +return deferred_pages_enabled (); + +return page_kasan_tag (page) ==KASAN_TAG_KERNEL ; +} +enum mminit_level {MMINIT_WARNING ,MMINIT_VERIFY ,
mm/page_alloc.c +0 -47
65 lines not on this page · raw message
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 7/25 mm: introduce luf_ugen to be used as a global timestamp
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that needs to evaluate the temporal sequence of events to determine whether tlb flush required has been done on each CPU.
To achieve that, this patch introduced a generation number, luf_ugen, and a few APIs manipulating the number. It's worth noting the number is designed to wraparound so care must be taken when using it.
--- include/linux/mm.h | 34 ++++++++++++++++++++++++++++++++++ mm/rmap.c | 22 ++++++++++++++++++++++ 2 files changed, 56 insertions(+)
include/linux/mm.h +34 -0
@@ -4155,4 +4155,38 @@ int arch_get_shadow_stack_status(struct task_struct *t, unsigned long __user *stint arch_set_shadow_stack_status (struct task_struct *t,unsigned long status);int arch_lock_shadow_stack_status (struct task_struct *t,unsigned long status); +#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH ) +/* +* luf_ugen will start with 2 so that 1 can be regarded as a passed one. +*/ +#define LUF_UGEN_INIT 2 + +static inline bool ugen_before (unsigned long a,unsigned long b) +{ +/* +* Consider wraparound. +*/ +return (long )(a - b) <0 ; +} + +static inline unsigned long next_ugen (unsigned long ugen) +{ +if (ugen +1 ) +return ugen +1 ; +/* +* Avoid invalid ugen, zero. +*/ +return ugen +2 ; +} + +static inline unsigned long prev_ugen (unsigned long ugen) +{ +if (ugen -1 ) +return ugen -1 ; +/* +* Avoid invalid ugen, zero. +*/ +return ugen -2 ; +} +#endif #endif /* _LINUX_MM_H */
mm/rmap.c +22 -0
@@ -634,6 +634,28 @@ struct anon_vma *folio_lock_anon_vma_read(const struct folio *folio, }#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH + +/* +* This generation number is primarily used as a global timestamp to +* determine whether tlb flush required has been done on each CPU. The +* function, ugen_before(), should be used to evaluate the temporal +* sequence of events because the number is designed to wraparound. +*/ +static atomic_long_t __maybe_unused luf_ugen =ATOMIC_LONG_INIT (LUF_UGEN_INIT ); + +/* +* Don't return invalid luf_ugen, zero. +*/ +static unsigned long __maybe_unused new_luf_ugen (void ) +{ +unsigned long ugen =atomic_long_inc_return (&luf_ugen); + +if (!ugen) + ugen =atomic_long_inc_return (&luf_ugen); + +return ugen; +} +/* * Flush TLB entries for recently unmapped pages from remote CPUs. It is * important if a PTE was dirty when it was unmapped that it's flushed
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 8/25 mm: introduce luf_batch to be used as hash table to store luf meta data
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that needs to keep luf meta data per page while staying in pcp or buddy allocator. The meta data includes cpumask for tlb shootdown and luf's request generation number.
Since struct page doesn't have enough room to store luf meta data, this patch introduces a hash table to store them and makes each page keep its hash key instead.
Since all the pages in pcp or buddy share the hash table, confliction is inevitable so care must be taken when reading or updating its entry.
--- include/linux/mm_types.h | 10 ++++ mm/internal.h | 8 +++ mm/rmap.c | 122 +++++++++++++++++++++++++++++++++++++-- 3 files changed, 136 insertions(+), 4 deletions(-)
include/linux/mm_types.h +10 -0
@@ -32,6 +32,16 @@struct address_space ;struct mem_cgroup ; +#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH +struct luf_batch { +struct tlbflush_unmap_batch batch; +unsigned long ugen; +rwlock_t lock; +}; +#else +struct luf_batch {}; +#endif +/* * Each physical page in the system has a struct page associated with * it to keep track of whatever it is we are using the page for at the
mm/internal.h +8 -0
@@ -1253,6 +1253,8 @@ extern struct workqueue_struct *mm_percpu_wq;void try_to_unmap_flush (void );void try_to_unmap_flush_dirty (void );void flush_tlb_batched_pending (struct mm_struct *mm); +void fold_batch (struct tlbflush_unmap_batch *dst,struct tlbflush_unmap_batch *src,bool reset); +void fold_luf_batch (struct luf_batch *dst,struct luf_batch *src);#else static inline void try_to_unmap_flush (void ) {
@@ -1263,6 +1265,12 @@ static inline void try_to_unmap_flush_dirty(void)static inline void flush_tlb_batched_pending (struct mm_struct *mm) { } +static inline void fold_batch (struct tlbflush_unmap_batch *dst,struct tlbflush_unmap_batch *src,bool reset) +{ +} +static inline void fold_luf_batch (struct luf_batch *dst,struct luf_batch *src) +{ +}#endif /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ extern const struct trace_print_flags pageflag_names[];
mm/rmap.c +118 -4
140 lines not on this page · raw message
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 9/25 mm: introduce API to perform tlb shootdown on exit from page allocator
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that performs tlb shootdown required on exit from page allocator.
This patch introduced a new API rather than making use of existing try_to_unmap_flush() to avoid repeated and redundant tlb shootdown due to frequent page allocations during a session of batched unmap flush.
--- include/linux/sched.h | 1 + mm/internal.h | 4 ++++ mm/rmap.c | 20 ++++++++++++++++++++ 3 files changed, 25 insertions(+)
include/linux/sched.h +1 -0
@@ -1401,6 +1401,7 @@ struct task_struct {#endif struct tlbflush_unmap_batch tlb_ubc; +struct tlbflush_unmap_batch tlb_ubc_takeoff;/* Cache last used pipe for splice(): */ struct pipe_inode_info *splice_pipe;
mm/internal.h +4 -0
@@ -1252,6 +1252,7 @@ extern struct workqueue_struct *mm_percpu_wq;#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH void try_to_unmap_flush (void );void try_to_unmap_flush_dirty (void ); +void try_to_unmap_flush_takeoff (void );void flush_tlb_batched_pending (struct mm_struct *mm);void fold_batch (struct tlbflush_unmap_batch *dst,struct tlbflush_unmap_batch *src,bool reset);void fold_luf_batch (struct luf_batch *dst,struct luf_batch *src);
@@ -1262,6 +1263,9 @@ static inline void try_to_unmap_flush(void)static inline void try_to_unmap_flush_dirty (void ) { } +static inline void try_to_unmap_flush_takeoff (void ) +{ +}static inline void flush_tlb_batched_pending (struct mm_struct *mm) { }
mm/rmap.c +20 -0
@@ -772,6 +772,26 @@ void fold_luf_batch(struct luf_batch *dst, struct luf_batch *src)read_unlock_irqrestore (&src->lock, flags); } +void try_to_unmap_flush_takeoff (void ) +{ +struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; +struct tlbflush_unmap_batch *tlb_ubc_takeoff = ¤t->tlb_ubc_takeoff; + +if (!tlb_ubc_takeoff->flush_required) +return ; + +arch_tlbbatch_flush (&tlb_ubc_takeoff->arch); + +/* +* Now that tlb shootdown of tlb_ubc_takeoff has been performed, +* it's good chance to shrink tlb_ubc if possible. +*/ +if (arch_tlbbatch_done (&tlb_ubc->arch, &tlb_ubc_takeoff->arch)) +reset_batch (tlb_ubc); + +reset_batch (tlb_ubc_takeoff); +} +/* * Flush TLB entries for recently unmapped pages from remote CPUs. It is * important if a PTE was dirty when it was unmapped that it's flushed
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 10/25 mm: introduce APIs to check if the page allocation is tlb shootdownable
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that should indentify if tlb shootdown can be performed on page allocation.
In a context with irq disabled or non-task, tlb shootdown cannot be performed because of deadlock issue. Thus, page allocator should work being aware of whether tlb shootdown can be performed on returning page.
This patch introduced APIs that pcp or buddy page allocator can use to delimit the critical sections taking off pages and indentify whether tlb shootdown can be performed.
--- include/linux/sched.h | 5 ++ mm/internal.h | 14 ++++ mm/page_alloc.c | 159 ++++++++++++++++++++++++++++++++++++++++++ mm/rmap.c | 2 +- 4 files changed, 179 insertions(+), 1 deletion(-)
include/linux/sched.h +5 -0
@@ -1400,6 +1400,11 @@ struct task_struct {struct callback_head cid_work;#endif +#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH ) +int luf_no_shootdown; +int luf_takeoff_started; +#endif +struct tlbflush_unmap_batch tlb_ubc;struct tlbflush_unmap_batch tlb_ubc_takeoff;
mm/internal.h +14 -0
@@ -1580,6 +1580,20 @@ static inline void accept_page(struct page *page) { }#endif /* CONFIG_UNACCEPTED_MEMORY */ +#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH ) +extern struct luf_batch luf_batch[]; +bool luf_takeoff_start (void ); +void luf_takeoff_end (void ); +bool luf_takeoff_no_shootdown (void ); +bool luf_takeoff_check (struct page *page); +bool luf_takeoff_check_and_fold (struct page *page); +#else +static inline bool luf_takeoff_start (void ) {return false; } +static inline void luf_takeoff_end (void ) {} +static inline bool luf_takeoff_no_shootdown (void ) {return true; } +static inline bool luf_takeoff_check (struct page *page) {return true; } +static inline bool luf_takeoff_check_and_fold (struct page *page) {return true; } +#endif /* pagewalk.c */ int walk_page_range_mm(struct mm_struct *mm, unsigned long start,
mm/page_alloc.c +159 -0
165 lines not on this page · raw message
mm/rmap.c +1 -1
@@ -693,7 +693,7 @@ void fold_batch(struct tlbflush_unmap_batch *dst,/* * Use 0th entry as accumulated batch. */ -static struct luf_batch luf_batch[NR_LUF_BATCH ]; +struct luf_batch luf_batch[NR_LUF_BATCH ];static void luf_batch_init (struct luf_batch *lb) {
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 11/25 mm: deliver luf_key to pcp or buddy on free after unmapping
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that needs to pass luf_key to pcp or buddy allocator on free after unmapping e.g. during page reclaim or page migration.
The luf_key will be used to track need of tlb shootdown and which cpus need to perform tlb flush, per page residing in pcp or buddy, and should be handed over properly when pages travel between pcp and buddy.
--- mm/internal.h | 4 +- mm/page_alloc.c | 116 ++++++++++++++++++++++++++++++++----------- mm/page_frag_cache.c | 6 +-- mm/page_isolation.c | 6 +++ mm/page_reporting.c | 6 +++ mm/slub.c | 2 +- mm/swap.c | 4 +- mm/vmscan.c | 8 +-- 8 files changed, 111 insertions(+), 41 deletions(-)
mm/internal.h +2 -2
@@ -746,8 +746,8 @@ struct page *__alloc_frozen_pages_noprof(gfp_t, unsigned int order, int nid, nodemask_t *);#define __alloc_frozen_pages (...) \ alloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__)) -void free_frozen_pages (struct page *page,unsigned int order); -void free_unref_folios (struct folio_batch *fbatch); +void free_frozen_pages (struct page *page,unsigned int order,unsigned short luf_key); +void free_unref_folios (struct folio_batch *fbatch,unsigned short luf_key);#ifdef CONFIG_NUMA struct page *alloc_frozen_pages_noprof (gfp_t ,unsigned int order);
mm/page_alloc.c +87 -29
319 lines not on this page · raw message
mm/page_frag_cache.c +3 -3
@@ -86,7 +86,7 @@ void __page_frag_cache_drain(struct page *page, unsigned int count)VM_BUG_ON_PAGE (page_ref_count (page) ==0 , page);if (page_ref_sub_and_test (page, count)) -free_frozen_pages (page,compound_order (page)); +free_frozen_pages (page,compound_order (page),0 ); }EXPORT_SYMBOL (__page_frag_cache_drain);
@@ -139,7 +139,7 @@ void *__page_frag_alloc_align(struct page_frag_cache *nc,if (unlikely (encoded_page_decode_pfmemalloc (encoded_page))) {free_frozen_pages (page, -encoded_page_decode_order (encoded_page)); +encoded_page_decode_order (encoded_page),0 ); goto refill; }
@@ -166,6 +166,6 @@ void page_frag_free(void *addr)struct page *page =virt_to_head_page (addr);if (unlikely (put_page_testzero (page))) -free_frozen_pages (page,compound_order (page)); +free_frozen_pages (page,compound_order (page),0 ); }EXPORT_SYMBOL (page_frag_free);
mm/page_isolation.c +6 -0
@@ -258,6 +258,12 @@ static void unset_migratetype_isolate(struct page *page, int migratetype)WARN_ON_ONCE (!move_freepages_block_isolate (zone, page, migratetype)); }else {set_pageblock_migratetype (page, migratetype); + +/* +* Do not clear the page's private to keep its luf_key +* unchanged. +*/ +__putback_isolated_page (page, order, migratetype); } zone->nr_isolate_pageblock--;
mm/page_reporting.c +6 -0
@@ -116,6 +116,12 @@ page_reporting_drain(struct page_reporting_dev_info *prdev,int mt =get_pageblock_migratetype (page);unsigned int order =get_order (sg->length); +/* +* Ensure private is zero before putting into the +* allocator. +*/ +set_page_private (page,0 ); +__putback_isolated_page (page, order, mt);/* If the pages were not reported due to error skip flagging */
mm/slub.c +1 -1
@@ -2652,7 +2652,7 @@ static void __free_slab(struct kmem_cache *s, struct slab *slab)__folio_clear_slab (folio);mm_account_reclaimed_pages (pages);unaccount_slab (slab, order, s); -free_frozen_pages (&folio->page, order); +free_frozen_pages (&folio->page, order,0 ); }static void rcu_free_slab (struct rcu_head *h)
mm/swap.c +2 -2
@@ -109,7 +109,7 @@ void __folio_put(struct folio *folio)page_cache_release (folio);folio_unqueue_deferred_split (folio);mem_cgroup_uncharge (folio); -free_frozen_pages (&folio->page,folio_order (folio)); +free_frozen_pages (&folio->page,folio_order (folio),0 ); }EXPORT_SYMBOL (__folio_put);
@@ -991,7 +991,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs) folios->nr = j;mem_cgroup_uncharge_folios (folios); -free_unref_folios (folios); +free_unref_folios (folios,0 ); }EXPORT_SYMBOL (folios_put_refs);
mm/vmscan.c +4 -4
@@ -1515,7 +1515,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (folio_batch_add (&free_folios, folio) ==0 ) {mem_cgroup_uncharge_folios (&free_folios);try_to_unmap_flush (); -free_unref_folios (&free_folios); +free_unref_folios (&free_folios,0 ); }continue ;
@@ -1584,7 +1584,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,mem_cgroup_uncharge_folios (&free_folios);try_to_unmap_flush (); -free_unref_folios (&free_folios); +free_unref_folios (&free_folios,0 );list_splice (&ret_folios, folio_list);count_vm_events (PGACTIVATE , pgactivate);
@@ -1908,7 +1908,7 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec,if (folio_batch_add (&free_folios, folio) ==0 ) {spin_unlock_irq (&lruvec->lru_lock);mem_cgroup_uncharge_folios (&free_folios); -free_unref_folios (&free_folios); +free_unref_folios (&free_folios,0 );spin_lock_irq (&lruvec->lru_lock); }
@@ -1930,7 +1930,7 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec,if (free_folios.nr) {spin_unlock_irq (&lruvec->lru_lock);mem_cgroup_uncharge_folios (&free_folios); -free_unref_folios (&free_folios); +free_unref_folios (&free_folios,0 );spin_lock_irq (&lruvec->lru_lock); }
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 12/25 mm: delimit critical sections to take off pages from pcp or buddy alloctor
In reply to Byungchul ParkNow that luf mechanism has been introduced, tlb shootdown might be necessary when luf'd pages exit from pcp or buddy allocator. Check if it's okay to take off pages and can perform for luf'd pages before use.
--- mm/compaction.c | 32 ++++++++++++++++-- mm/internal.h | 2 +- mm/page_alloc.c | 79 +++++++++++++++++++++++++++++++++++++++++++-- mm/page_isolation.c | 4 ++- mm/page_reporting.c | 20 +++++++++++- 5 files changed, 129 insertions(+), 8 deletions(-)
mm/compaction.c +29 -3
@@ -606,6 +606,7 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, page =pfn_to_page (blockpfn); +luf_takeoff_start ();/* Isolate free pages. */ for (; blockpfn < end_pfn; blockpfn += stride, page += stride) {int isolated;
@@ -654,9 +655,12 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, goto isolate_fail; } +if (!luf_takeoff_check (page)) + goto isolate_fail; +/* Found a free page, will break it into order-0 pages */ order =buddy_order (page); - isolated =__isolate_free_page (page, order); + isolated =__isolate_free_page (page, order, false);if (!isolated)break ;set_page_private (page, order);
@@ -684,6 +688,11 @@ static unsigned long isolate_freepages_block(struct compact_control *cc,if (locked)spin_unlock_irqrestore (&cc->zone->lock, flags); +/* +* Check and flush before using the pages taken off. +*/ +luf_takeoff_end (); +/* * Be careful to not go outside of the pageblock. */
@@ -1591,6 +1600,7 @@ static void fast_isolate_freepages(struct compact_control *cc)if (!area->nr_free)continue ; +luf_takeoff_start ();spin_lock_irqsave (&cc->zone->lock, flags); freelist = &area->free_list[MIGRATE_MOVABLE ];list_for_each_entry_reverse (freepage, freelist, buddy_list) {
@@ -1598,6 +1608,10 @@ static void fast_isolate_freepages(struct compact_control *cc) order_scanned++; nr_scanned++; + +if (!luf_takeoff_check (freepage)) + goto scan_next; + pfn =page_to_pfn (freepage);if (pfn >= highest)
@@ -1617,7 +1631,7 @@ static void fast_isolate_freepages(struct compact_control *cc)/* Shorten the scan if a candidate is found */ limit >>=1 ; } - +scan_next:if (order_scanned >= limit)break ; }
@@ -1635,7 +1649,7 @@ static void fast_isolate_freepages(struct compact_control *cc)/* Isolate the page if available */ if (page) { -if (__isolate_free_page (page, order)) { +if (__isolate_free_page (page, order, false)) {set_page_private (page, order); nr_isolated =1 << order; nr_scanned += nr_isolated -1 ;
@@ -1652,6 +1666,11 @@ static void fast_isolate_freepages(struct compact_control *cc)spin_unlock_irqrestore (&cc->zone->lock, flags); +/* +* Check and flush before using the pages taken off. +*/ +luf_takeoff_end (); +/* Skip fast search if enough freepages isolated */ if (cc->nr_freepages >= cc->nr_migratepages)break ;
@@ -2372,7 +2391,14 @@ static enum compact_result compact_finished(struct compact_control *cc) {int ret; +/* +* luf_takeoff_{start,end}() is required to identify whether +* this compaction context is tlb shootdownable for luf'd pages. +*/ +luf_takeoff_start (); ret =__compact_finished (cc); +luf_takeoff_end (); +trace_mm_compaction_finished (cc->zone, cc->order, ret);if (ret ==COMPACT_NO_SUITABLE_PAGE ) ret =COMPACT_CONTINUE ;
mm/internal.h +1 -1
@@ -664,7 +664,7 @@ static inline void clear_zone_contiguous(struct zone *zone) zone->contiguous = false; } -extern int __isolate_free_page (struct page *page,unsigned int order); +extern int __isolate_free_page (struct page *page,unsigned int order,bool willputback);extern void __putback_isolated_page (struct page *page,unsigned int order,int mt); extern void memblock_free_pages(struct page *page, unsigned long pfn,
mm/page_alloc.c +77 -2
246 lines not on this page · raw message
mm/page_isolation.c +3 -1
@@ -211,6 +211,7 @@ static void unset_migratetype_isolate(struct page *page, int migratetype)struct page *buddy; zone =page_zone (page); +luf_takeoff_start ();spin_lock_irqsave (&zone->lock, flags);if (!is_migrate_isolate_page (page)) goto out;
@@ -229,7 +230,7 @@ static void unset_migratetype_isolate(struct page *page, int migratetype) buddy =find_buddy_page_pfn (page,page_to_pfn (page), order,NULL );if (buddy && !is_migrate_isolate_page (buddy)) { - isolated_page = !!__isolate_free_page (page, order); + isolated_page = !!__isolate_free_page (page, order, true);/* * Isolating a free page in an isolated pageblock * is expected to always work as watermarks don't
@@ -269,6 +270,7 @@ static void unset_migratetype_isolate(struct page *page, int migratetype) zone->nr_isolate_pageblock--; out:spin_unlock_irqrestore (&zone->lock, flags); +luf_takeoff_end (zone); } static inline struct page *
mm/page_reporting.c +19 -1
@@ -167,6 +167,7 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,if (list_empty (list))return err; +luf_takeoff_start ();spin_lock_irq (&zone->lock);/*
@@ -191,6 +192,11 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,if (PageReported (page))continue ; +if (!luf_takeoff_check (page)) { +VM_WARN_ON (1 ); +continue ; + } +/* * If we fully consumed our budget then update our * state to indicate that we are requesting additional
@@ -204,7 +210,7 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,/* Attempt to pull page from list and place in scatterlist */ if (*offset) { -if (!__isolate_free_page (page, order)) { +if (!__isolate_free_page (page, order, false)) { next = page;break ; }
@@ -227,6 +233,11 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,/* release lock before waiting on report processing */ spin_unlock_irq (&zone->lock); +/* +* Check and flush before using the pages taken off. +*/ +luf_takeoff_end (); +/* begin processing pages in local list */ err = prdev->report (prdev, sgl,PAGE_REPORTING_CAPACITY );
@@ -236,6 +247,8 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,/* update budget to reflect call to report function */ budget--; +luf_takeoff_start (); +/* reacquire zone lock and resume processing */ spin_lock_irq (&zone->lock);
@@ -259,6 +272,11 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,spin_unlock_irq (&zone->lock); +/* +* Check and flush before using the pages taken off. +*/ +luf_takeoff_end (); +return err; }
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 13/25 mm: introduce pend_list in struct free_area to track luf'd pages
In reply to Byungchul Parkluf'd pages requires tlb shootdown on exiting from page allocator. For some page allocation request, it's okay to return luf'd page followed by tlb shootdown but it's not okay for e.g. irq context.
This patch splitted the list in free_area into two, 'free_list' for non-luf'd pages and 'pend_list' for luf'd pages so that the buddy allocator can work better with various conditions of context.
--- include/linux/mmzone.h | 3 ++ kernel/power/snapshot.c | 14 ++++++ kernel/vmcore_info.c | 2 + mm/compaction.c | 33 ++++++++++--- mm/internal.h | 17 ++++++- mm/mm_init.c | 2 + mm/page_alloc.c | 105 ++++++++++++++++++++++++++++++++++------ mm/page_reporting.c | 22 ++++++--- mm/vmstat.c | 15 ++++++ 9 files changed, 184 insertions(+), 29 deletions(-)
include/linux/mmzone.h +3 -0
@@ -116,6 +116,7 @@ extern int page_group_by_mobility_disabled; MIGRATETYPE_MASK)struct free_area {struct list_head free_list[MIGRATE_TYPES ]; +struct list_head pend_list[MIGRATE_TYPES ];unsigned long nr_free; };
@@ -1014,6 +1015,8 @@ struct zone {/* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS ];atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS ]; +/* Count pages that need tlb shootdown on allocation */ +atomic_long_t nr_luf_pages; }____cacheline_internodealigned_in_smp ;enum pgdat_flags {
kernel/power/snapshot.c +14 -0
@@ -1285,6 +1285,20 @@ static void mark_free_pages(struct zone *zone)swsusp_set_page_free (pfn_to_page (pfn + i)); } } + +list_for_each_entry (page, + &zone->free_area[order].pend_list[t], buddy_list) { +unsigned long i; + + pfn =page_to_pfn (page); +for (i =0 ; i < (1UL << order); i++) { +if (!--page_count) { +touch_nmi_watchdog (); + page_count =WD_PAGE_COUNT ; + } +swsusp_set_page_free (pfn_to_page (pfn + i)); + } + } }spin_unlock_irqrestore (&zone->lock, flags); }
kernel/vmcore_info.c +2 -0
@@ -188,11 +188,13 @@ static int __init crash_save_vmcoreinfo_init(void)VMCOREINFO_OFFSET (zone, vm_stat);VMCOREINFO_OFFSET (zone, spanned_pages);VMCOREINFO_OFFSET (free_area, free_list); +VMCOREINFO_OFFSET (free_area, pend_list);VMCOREINFO_OFFSET (list_head, next);VMCOREINFO_OFFSET (list_head, prev);VMCOREINFO_LENGTH (zone.free_area,NR_PAGE_ORDERS );log_buf_vmcoreinfo_setup ();VMCOREINFO_LENGTH (free_area.free_list,MIGRATE_TYPES ); +VMCOREINFO_LENGTH (free_area.pend_list,MIGRATE_TYPES );VMCOREINFO_NUMBER (NR_FREE_PAGES );VMCOREINFO_NUMBER (PG_lru);VMCOREINFO_NUMBER (PG_private);
mm/compaction.c +27 -6
@@ -1592,24 +1592,28 @@ static void fast_isolate_freepages(struct compact_control *cc) order = next_search_order(cc, order)) {struct free_area *area = &cc->zone->free_area[order];struct list_head *freelist; +struct list_head *high_pfn_list;struct page *freepage;unsigned long flags;unsigned int order_scanned =0 ;unsigned long high_pfn =0 ; +bool consider_pend = false; +bool can_shootdown;if (!area->nr_free)continue ; -luf_takeoff_start (); + can_shootdown =luf_takeoff_start ();spin_lock_irqsave (&cc->zone->lock, flags); freelist = &area->free_list[MIGRATE_MOVABLE ]; +retry:list_for_each_entry_reverse (freepage, freelist, buddy_list) {unsigned long pfn; order_scanned++; nr_scanned++; -if (!luf_takeoff_check (freepage)) +if (unlikely (consider_pend && !luf_takeoff_check (freepage))) goto scan_next; pfn =page_to_pfn (freepage);
@@ -1622,26 +1626,34 @@ static void fast_isolate_freepages(struct compact_control *cc) cc->fast_search_fail =0 ; cc->search_order = order; page = freepage; -break ; + goto done; }if (pfn >= min_pfn && pfn > high_pfn) { high_pfn = pfn; + high_pfn_list = freelist;/* Shorten the scan if a candidate is found */ limit >>=1 ; } scan_next:if (order_scanned >= limit) -break ; + goto done; } +if (!consider_pend && can_shootdown) { + consider_pend = true; + freelist = &area->pend_list[MIGRATE_MOVABLE ]; + goto retry; + } +done:/* Use a maximum candidate pfn if a preferred one was not found */ if (!page && high_pfn) { page =pfn_to_page (high_pfn);/* Update freepage for the list reorder below */ freepage = page; + freelist = high_pfn_list; }/* Reorder to so a future search skips recent pages */
@@ -2039,18 +2051,20 @@ static unsigned long fast_find_migrateblock(struct compact_control *cc)struct list_head *freelist;unsigned long flags;struct page *freepage; +bool consider_pend = false;if (!area->nr_free)continue ;spin_lock_irqsave (&cc->zone->lock, flags); freelist = &area->free_list[MIGRATE_MOVABLE ]; +retry:list_for_each_entry (freepage, freelist, buddy_list) {unsigned long free_pfn;if (nr_scanned++ >= limit) {move_freelist_tail (freelist, freepage); -break ; + goto done; } free_pfn =page_to_pfn (freepage);
@@ -2073,9 +2087,16 @@ static unsigned long fast_find_migrateblock(struct compact_control *cc) pfn = cc->zone->zone_start_pfn; cc->fast_search_fail =0 ; found_block = true; -break ; + goto done; } } + +if (!consider_pend) { + consider_pend = true; + freelist = &area->pend_list[MIGRATE_MOVABLE ]; + goto retry; + } +done:spin_unlock_irqrestore (&cc->zone->lock, flags); }
mm/internal.h +16 -1
@@ -849,11 +849,16 @@ void init_cma_reserved_pageblock(struct page *page);int find_suitable_fallback (struct free_area *area,unsigned int order,int migratetype,bool only_stealable,bool *can_steal); -static inline bool free_area_empty (struct free_area *area,int migratetype) +static inline bool free_list_empty (struct free_area *area,int migratetype) {return list_empty (&area->free_list[migratetype]); } +static inline bool free_area_empty (struct free_area *area,int migratetype) +{ +return list_empty (&area->free_list[migratetype]) && +list_empty (&area->pend_list[migratetype]); +}/* mm/util.c */ struct anon_vma *folio_anon_vma (const struct folio *folio);
@@ -1587,12 +1592,22 @@ void luf_takeoff_end(void);bool luf_takeoff_no_shootdown (void );bool luf_takeoff_check (struct page *page);bool luf_takeoff_check_and_fold (struct page *page); + +static inline bool non_luf_pages_ok (struct zone *zone) +{ +unsigned long nr_free =zone_page_state (zone,NR_FREE_PAGES ); +unsigned long min_wm =min_wmark_pages (zone); +unsigned long nr_luf_pages =atomic_long_read (&zone->nr_luf_pages); + +return nr_free - nr_luf_pages > min_wm; +}#else static inline bool luf_takeoff_start (void ) {return false; }static inline void luf_takeoff_end (void ) {}static inline bool luf_takeoff_no_shootdown (void ) {return true; }static inline bool luf_takeoff_check (struct page *page) {return true; }static inline bool luf_takeoff_check_and_fold (struct page *page) {return true; } +static inline bool non_luf_pages_ok (struct zone *zone) {return true; }#endif /* pagewalk.c */
mm/mm_init.c +2 -0
@@ -1399,12 +1399,14 @@ static void __meminit zone_init_free_lists(struct zone *zone)unsigned int order, t;for_each_migratetype_order (order, t) {INIT_LIST_HEAD (&zone->free_area[order].free_list[t]); +INIT_LIST_HEAD (&zone->free_area[order].pend_list[t]); zone->free_area[order].nr_free =0 ; }#ifdef CONFIG_UNACCEPTED_MEMORY INIT_LIST_HEAD (&zone->unaccepted_pages);#endif +atomic_long_set (&zone->nr_luf_pages,0 ); } void __meminit init_currently_empty_zone(struct zone *zone,
mm/page_alloc.c +90 -15
193 lines not on this page · raw message
mm/page_reporting.c +15 -7
@@ -159,15 +159,17 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,struct page *page, *next;long budget;int err =0 ; +bool consider_pend = false; +bool can_shootdown;/* * Perform early check, if free area is empty there is * nothing to process so we can skip this free_list. */ -if (list_empty (list)) +if (free_area_empty (area, mt))return err; -luf_takeoff_start (); + can_shootdown =luf_takeoff_start ();spin_lock_irq (&zone->lock);/*
@@ -185,14 +187,14 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,* should always be a power of 2. */ budget =DIV_ROUND_UP (area->nr_free,PAGE_REPORTING_CAPACITY *16 ); - +retry:/* loop through free list adding unreported pages to sg list */ list_for_each_entry_safe (page, next, list, lru) {/* We are going to skip over the reported pages. */ if (PageReported (page))continue ; -if (!luf_takeoff_check (page)) { +if (unlikely (consider_pend && !luf_takeoff_check (page))) {VM_WARN_ON (1 );continue ; }
@@ -205,14 +207,14 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,if (budget <0 ) {atomic_set (&prdev->state,PAGE_REPORTING_REQUESTED ); next = page; -break ; + goto done; }/* Attempt to pull page from list and place in scatterlist */ if (*offset) {if (!__isolate_free_page (page, order, false)) { next = page; -break ; + goto done; }/* Add page to scatter list */
@@ -263,9 +265,15 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,/* exit on error */ if (err) -break ; + goto done; } +if (!consider_pend && can_shootdown) { + consider_pend = true; + list = &area->pend_list[mt]; + goto retry; + } +done:/* Rotate any leftover pages to the head of the freelist */ if (!list_entry_is_head (next, list, lru) && !list_is_first (&next->lru, list))list_rotate_to_front (&next->lru, list);
mm/vmstat.c +15 -0
@@ -1581,6 +1581,21 @@ static void pagetypeinfo_showfree_print(struct seq_file *m,break ; } } +list_for_each (curr, &area->pend_list[mtype]) { +/* +* Cap the pend_list iteration because it might +* be really large and we are under a spinlock +* so a long time spent here could trigger a +* hard lockup detector. Anyway this is a +* debugging tool so knowing there is a handful +* of pages of this order should be more than +* sufficient. +*/ +if (++freecount >=100000 ) { + overflow = true; +break ; + } + }seq_printf (m,"%s%6lu " , overflow ?">" :"" , freecount);spin_unlock_irq (&zone->lock);cond_resched ();
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 14/25 mm/rmap: recognize read-only tlb entries during batched tlb flush
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that requires to recognize read-only tlb entries and handle them in a different way. The newly introduced API in this patch, fold_ubc(), will be used by luf mechanism.
--- include/linux/sched.h | 1 + mm/rmap.c | 16 ++++++++++++++-- 2 files changed, 15 insertions(+), 2 deletions(-)
include/linux/sched.h +1 -0
@@ -1407,6 +1407,7 @@ struct task_struct {struct tlbflush_unmap_batch tlb_ubc;struct tlbflush_unmap_batch tlb_ubc_takeoff; +struct tlbflush_unmap_batch tlb_ubc_ro;/* Cache last used pipe for splice(): */ struct pipe_inode_info *splice_pipe;
mm/rmap.c +14 -2
@@ -775,6 +775,7 @@ void fold_luf_batch(struct luf_batch *dst, struct luf_batch *src)void try_to_unmap_flush_takeoff (void ) {struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; +struct tlbflush_unmap_batch *tlb_ubc_ro = ¤t->tlb_ubc_ro;struct tlbflush_unmap_batch *tlb_ubc_takeoff = ¤t->tlb_ubc_takeoff;if (!tlb_ubc_takeoff->flush_required)
@@ -789,6 +790,9 @@ void try_to_unmap_flush_takeoff(void)if (arch_tlbbatch_done (&tlb_ubc->arch, &tlb_ubc_takeoff->arch))reset_batch (tlb_ubc); +if (arch_tlbbatch_done (&tlb_ubc_ro->arch, &tlb_ubc_takeoff->arch)) +reset_batch (tlb_ubc_ro); +reset_batch (tlb_ubc_takeoff); }
@@ -801,7 +805,9 @@ void try_to_unmap_flush_takeoff(void)void try_to_unmap_flush (void ) {struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; +struct tlbflush_unmap_batch *tlb_ubc_ro = ¤t->tlb_ubc_ro; +fold_batch (tlb_ubc, tlb_ubc_ro, true);if (!tlb_ubc->flush_required)return ;
@@ -813,8 +819,9 @@ void try_to_unmap_flush(void)void try_to_unmap_flush_dirty (void ) {struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; +struct tlbflush_unmap_batch *tlb_ubc_ro = ¤t->tlb_ubc_ro; -if (tlb_ubc->writable) +if (tlb_ubc->writable || tlb_ubc_ro->writable)try_to_unmap_flush (); }
@@ -831,13 +838,18 @@ void try_to_unmap_flush_dirty(void)static void set_tlb_ubc_flush_pending (struct mm_struct *mm,pte_t pteval,unsigned long uaddr) { -struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; +struct tlbflush_unmap_batch *tlb_ubc;int batch;bool writable =pte_dirty (pteval);if (!pte_accessible (mm, pteval))return ; +if (pte_write (pteval)) + tlb_ubc = ¤t->tlb_ubc; +else + tlb_ubc = ¤t->tlb_ubc_ro; +arch_tlbbatch_add_pending (&tlb_ubc->arch, mm, uaddr); tlb_ubc->flush_required = true;
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 15/25 fs, filemap: refactor to gather the scattered ->write_{begin,end}() calls
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that requires to hook when updating page cache that might have pages that have been mapped on any tasks so that tlb flush needed can be performed.
--- drivers/gpu/drm/i915/gem/i915_gem_shmem.c | 11 ++++------- fs/affs/file.c | 4 ++-- fs/buffer.c | 14 ++++++-------- fs/exfat/file.c | 5 ++--- fs/ext4/verity.c | 5 ++--- fs/f2fs/super.c | 5 ++--- fs/f2fs/verity.c | 5 ++--- fs/namei.c | 5 ++--- include/linux/fs.h | 18 ++++++++++++++++++ mm/filemap.c | 5 ++--- 10 files changed, 42 insertions(+), 35 deletions(-)
drivers/gpu/drm/i915/gem/i915_gem_shmem.c +4 -7
@@ -418,7 +418,6 @@ shmem_pwrite(struct drm_i915_gem_object *obj, const struct drm_i915_gem_pwrite *arg) {struct address_space *mapping = obj->base.filp->f_mapping; -const struct address_space_operations *aops = mapping->a_ops;char __user *user_data =u64_to_user_ptr (arg->data_ptr);u64 remain;loff_t pos;
@@ -477,7 +476,7 @@ shmem_pwrite(struct drm_i915_gem_object *obj,if (err)return err; - err = aops->write_begin (obj->base.filp, mapping, pos, len, + err =mapping_write_begin (obj->base.filp, mapping, pos, len, &folio, &data);if (err <0 )return err;
@@ -488,7 +487,7 @@ shmem_pwrite(struct drm_i915_gem_object *obj,pagefault_enable ();kunmap_local (vaddr); - err = aops->write_end (obj->base.filp, mapping, pos, len, + err =mapping_write_end (obj->base.filp, mapping, pos, len, len - unwritten, folio, data);if (err <0 )return err;
@@ -654,7 +653,6 @@ i915_gem_object_create_shmem_from_data(struct drm_i915_private *i915, {struct drm_i915_gem_object *obj;struct file *file; -const struct address_space_operations *aops;loff_t pos;int err;
@@ -666,21 +664,20 @@ i915_gem_object_create_shmem_from_data(struct drm_i915_private *i915,GEM_BUG_ON (obj->write_domain !=I915_GEM_DOMAIN_CPU ); file = obj->base.filp; - aops = file->f_mapping->a_ops; pos =0 ; do { unsigned int len = min_t(typeof(size), size, PAGE_SIZE); struct folio *folio; void *fsdata; - err = aops->write_begin(file, file->f_mapping, pos, len, + err = mapping_write_begin(file, file->f_mapping, pos, len, &folio, &fsdata); if (err < 0) goto fail; memcpy_to_folio(folio, offset_in_folio(folio, pos), data, len); - err = aops->write_end(file, file->f_mapping, pos, len, len, + err = mapping_write_end(file, file->f_mapping, pos, len, len, folio, fsdata); if (err < 0) goto fail;
fs/affs/file.c +2 -2
@@ -885,9 +885,9 @@ affs_truncate(struct inode *inode)loff_t isize = inode->i_size;int res; - res = mapping->a_ops->write_begin (NULL , mapping, isize,0 , &folio, &fsdata); + res =mapping_write_begin (NULL , mapping, isize,0 , &folio, &fsdata);if (!res) - res = mapping->a_ops->write_end (NULL , mapping, isize,0 ,0 , folio, fsdata); + res =mapping_write_end (NULL , mapping, isize,0 ,0 , folio, fsdata);else inode->i_size =AFFS_I (inode)->mmu_private;mark_inode_dirty (inode);
fs/buffer.c +6 -8
@@ -2456,7 +2456,6 @@ EXPORT_SYMBOL(block_read_full_folio);int generic_cont_expand_simple (struct inode *inode,loff_t size) {struct address_space *mapping = inode->i_mapping; -const struct address_space_operations *aops = mapping->a_ops;struct folio *folio;void *fsdata =NULL ;int err;
@@ -2465,11 +2464,11 @@ int generic_cont_expand_simple(struct inode *inode, loff_t size)if (err) goto out; - err = aops->write_begin (NULL , mapping, size,0 , &folio, &fsdata); + err =mapping_write_begin (NULL , mapping, size,0 , &folio, &fsdata);if (err) goto out; - err = aops->write_end (NULL , mapping, size,0 ,0 , folio, fsdata); + err =mapping_write_end (NULL , mapping, size,0 ,0 , folio, fsdata);BUG_ON (err >0 ); out:
@@ -2481,7 +2480,6 @@ static int cont_expand_zero(struct file *file, struct address_space *mapping, loff_t pos, loff_t *bytes) { struct inode *inode = mapping->host; -const struct address_space_operations *aops = mapping->a_ops;unsigned int blocksize =i_blocksize (inode);struct folio *folio;void *fsdata =NULL ;
@@ -2501,12 +2499,12 @@ static int cont_expand_zero(struct file *file, struct address_space *mapping, } len =PAGE_SIZE - zerofrom; - err = aops->write_begin (file, mapping, curpos, len, + err =mapping_write_begin (file, mapping, curpos, len, &folio, &fsdata);if (err) goto out;folio_zero_range (folio,offset_in_folio (folio, curpos), len); - err = aops->write_end (file, mapping, curpos, len, len, + err =mapping_write_end (file, mapping, curpos, len, len, folio, fsdata);if (err <0 ) goto out;
@@ -2534,12 +2532,12 @@ static int cont_expand_zero(struct file *file, struct address_space *mapping, } len = offset - zerofrom; - err = aops->write_begin (file, mapping, curpos, len, + err =mapping_write_begin (file, mapping, curpos, len, &folio, &fsdata);if (err) goto out;folio_zero_range (folio,offset_in_folio (folio, curpos), len); - err = aops->write_end (file, mapping, curpos, len, len, + err =mapping_write_end (file, mapping, curpos, len, len, folio, fsdata);if (err <0 ) goto out;
fs/exfat/file.c +2 -3
@@ -539,7 +539,6 @@ static int exfat_extend_valid_size(struct file *file, loff_t new_valid_size)struct inode *inode =file_inode (file);struct exfat_inode_info *ei =EXFAT_I (inode);struct address_space *mapping = inode->i_mapping; -const struct address_space_operations *ops = mapping->a_ops; pos = ei->valid_size;while (pos < new_valid_size) {
@@ -551,14 +550,14 @@ static int exfat_extend_valid_size(struct file *file, loff_t new_valid_size)if (pos + len > new_valid_size) len = new_valid_size - pos; - err = ops->write_begin (file, mapping, pos, len, &folio,NULL ); + err =mapping_write_begin (file, mapping, pos, len, &folio,NULL );if (err) goto out; off =offset_in_folio (folio, pos);folio_zero_new_buffers (folio, off, off + len); - err = ops->write_end (file, mapping, pos, len, len, folio,NULL ); + err =mapping_write_end (file, mapping, pos, len, len, folio,NULL );if (err <0 ) goto out; pos += len;
fs/ext4/verity.c +2 -3
@@ -68,7 +68,6 @@ static int pagecache_write(struct inode *inode, const void *buf, size_t count, loff_t pos) {struct address_space *mapping = inode->i_mapping; -const struct address_space_operations *aops = mapping->a_ops;if (pos + count > inode->i_sb->s_maxbytes)return -EFBIG ;
@@ -80,13 +79,13 @@ static int pagecache_write(struct inode *inode, const void *buf, size_t count,void *fsdata =NULL ;int res; - res = aops->write_begin (NULL , mapping, pos, n, &folio, &fsdata); + res =mapping_write_begin (NULL , mapping, pos, n, &folio, &fsdata);if (res)return res;memcpy_to_folio (folio,offset_in_folio (folio, pos), buf, n); - res = aops->write_end (NULL , mapping, pos, n, n, folio, fsdata); + res =mapping_write_end (NULL , mapping, pos, n, n, folio, fsdata);if (res <0 )return res;if (res != n)
fs/f2fs/super.c +2 -3
@@ -2710,7 +2710,6 @@ static ssize_t f2fs_quota_write(struct super_block *sb, int type, {struct inode *inode =sb_dqopt (sb)->files[type];struct address_space *mapping = inode->i_mapping; -const struct address_space_operations *a_ops = mapping->a_ops;int offset = off & (sb->s_blocksize -1 );size_t towrite = len;struct folio *folio;
@@ -2722,7 +2721,7 @@ static ssize_t f2fs_quota_write(struct super_block *sb, int type, tocopy = min_t(unsigned long, sb->s_blocksize - offset, towrite); retry: - err = a_ops->write_begin (NULL , mapping, off, tocopy, + err =mapping_write_begin (NULL , mapping, off, tocopy, &folio, &fsdata);if (unlikely (err)) {if (err == -ENOMEM ) {
@@ -2735,7 +2734,7 @@ static ssize_t f2fs_quota_write(struct super_block *sb, int type,memcpy_to_folio (folio,offset_in_folio (folio, off), data, tocopy); - a_ops->write_end (NULL , mapping, off, tocopy, tocopy, +mapping_write_end (NULL , mapping, off, tocopy, tocopy, folio, fsdata); offset =0 ; towrite -= tocopy;
fs/f2fs/verity.c +2 -3
@@ -72,7 +72,6 @@ static int pagecache_write(struct inode *inode, const void *buf, size_t count, loff_t pos) {struct address_space *mapping = inode->i_mapping; -const struct address_space_operations *aops = mapping->a_ops;if (pos + count >F2FS_BLK_TO_BYTES (max_file_blocks (inode)))return -EFBIG ;
@@ -84,13 +83,13 @@ static int pagecache_write(struct inode *inode, const void *buf, size_t count,void *fsdata =NULL ;int res; - res = aops->write_begin (NULL , mapping, pos, n, &folio, &fsdata); + res =mapping_write_begin (NULL , mapping, pos, n, &folio, &fsdata);if (res)return res;memcpy_to_folio (folio,offset_in_folio (folio, pos), buf, n); - res = aops->write_end (NULL , mapping, pos, n, n, folio, fsdata); + res =mapping_write_end (NULL , mapping, pos, n, n, folio, fsdata);if (res <0 )return res;if (res != n)
fs/namei.c +2 -3
@@ -5409,7 +5409,6 @@ EXPORT_SYMBOL(page_readlink);int page_symlink (struct inode *inode,const char *symname,int len) {struct address_space *mapping = inode->i_mapping; -const struct address_space_operations *aops = mapping->a_ops;bool nofs = !mapping_gfp_constraint (mapping, __GFP_FS);struct folio *folio;void *fsdata =NULL ;
@@ -5419,7 +5418,7 @@ int page_symlink(struct inode *inode, const char *symname, int len) retry:if (nofs) flags =memalloc_nofs_save (); - err = aops->write_begin (NULL , mapping,0 , len-1 , &folio, &fsdata); + err =mapping_write_begin (NULL , mapping,0 , len-1 , &folio, &fsdata);if (nofs)memalloc_nofs_restore (flags);if (err)
@@ -5427,7 +5426,7 @@ int page_symlink(struct inode *inode, const char *symname, int len)memcpy (folio_address (folio), symname, len -1 ); - err = aops->write_end (NULL , mapping,0 , len -1 , len -1 , + err =mapping_write_end (NULL , mapping,0 , len -1 , len -1 , folio, fsdata);if (err <0 ) goto fail;
include/linux/fs.h +18 -0
@@ -531,6 +531,24 @@ struct address_space {#define PAGECACHE_TAG_WRITEBACK XA_MARK_1#define PAGECACHE_TAG_TOWRITE XA_MARK_2 +static inline int mapping_write_begin (struct file *file, +struct address_space *mapping, +loff_t pos,unsigned len, +struct folio **foliop,void **fsdata) +{ +return mapping->a_ops->write_begin (file, mapping, pos, len, foliop, + fsdata); +} + +static inline int mapping_write_end (struct file *file, +struct address_space *mapping, +loff_t pos,unsigned len,unsigned copied, +struct folio *folio,void *fsdata) +{ +return mapping->a_ops->write_end (file, mapping, pos, len, copied, + folio, fsdata); +} +/* * Returns true if any of the pages in the mapping are marked with the tag. */
mm/filemap.c +2 -3
@@ -4152,7 +4152,6 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i)struct file *file = iocb->ki_filp;loff_t pos = iocb->ki_pos;struct address_space *mapping = file->f_mapping; -const struct address_space_operations *a_ops = mapping->a_ops;size_t chunk =mapping_max_folio_size (mapping);long status =0 ;ssize_t written =0 ;
@@ -4186,7 +4185,7 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i)break ; } - status = a_ops->write_begin (file, mapping, pos, bytes, + status =mapping_write_begin (file, mapping, pos, bytes, &folio, &fsdata);if (unlikely (status <0 ))break ;
@@ -4201,7 +4200,7 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i) copied =copy_folio_from_iter_atomic (folio, offset, bytes, i);flush_dcache_folio (folio); - status = a_ops->write_end (file, mapping, pos, bytes, copied, + status =mapping_write_end (file, mapping, pos, bytes, copied, folio, fsdata);if (unlikely (status != copied)) {iov_iter_revert (i, copied -max (status,0L ));
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 16/25 mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped
In reply to Byungchul ParkA new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read-only and were unmapped, as long as the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.
tlb flush can be defered when folios get unmapped as long as it guarantees to perform tlb flush needed, before the folios actually become used, of course, only if all the corresponding ptes don't have write permission. Otherwise, the system will get messed up.
To achieve that, for the folios that map only to non-writable tlb entries, prevent tlb flush during unmapping but perform it just before the folios actually become used, out of buddy or pcp.
However, we should cancel the pending by LUF and perform the deferred TLB flush right away when:
1. a writable pte is newly set through fault handler
2. a file is updated
3. kasan needs poisoning on free
4. the kernel wants to init pages on free
No matter what type of workload is used for performance evaluation, the result would be positive thanks to the unconditional reduction of tlb flushes, tlb misses and interrupts. For the test, I picked up one of the most popular and heavy workload, llama.cpp that is a LLM(Large Language Model) inference engine.
The result would depend on memory latency and how often reclaim runs, which implies tlb miss overhead and how many times unmapping happens. In my system, the result shows:
1. tlb shootdown interrupts are reduced about 97%.
2. The test program runtime is reduced about 4.5%.
The test environment and the test set are like:
Machine: bare metal, x86_64, Intel(R) Xeon(R) Gold 6430
CPU: 1 socket 64 core with hyper thread on Numa: 2 nodes (64 CPUs DRAM 42GB, no CPUs CXL expander 98GB) Config: swap off, numa balancing tiering on, demotion enabled
llama.cpp/main -m $(70G_model1) -p "who are you?" -s 1 -t 15 -n 20 & llama.cpp/main -m $(70G_model2) -p "who are you?" -s 1 -t 15 -n 20 & llama.cpp/main -m $(70G_model3) -p "who are you?" -s 1 -t 15 -n 20 & wait
where,
-t: nr of threads, -s: seed used to make the runtime stable, -n: nr of tokens that determines the runtime, -p: prompt to ask, -m: LLM model to use.
Run the test set 5 times successively with caches dropped every run via 'echo 3 > /proc/sys/vm/drop_caches'. Each inference prints its runtime at the end of each. The results are like:
1. Runtime from the output of llama.cpp
BEFORE
------
llama_print_timings: total time = 883450.54 ms / 24 tokens llama_print_timings: total time = 861665.91 ms / 24 tokens llama_print_timings: total time = 898079.02 ms / 24 tokens llama_print_timings: total time = 879897.69 ms / 24 tokens llama_print_timings: total time = 892360.75 ms / 24 tokens llama_print_timings: total time = 884587.85 ms / 24 tokens llama_print_timings: total time = 861023.19 ms / 24 tokens llama_print_timings: total time = 900022.18 ms / 24 tokens llama_print_timings: total time = 878771.88 ms / 24 tokens llama_print_timings: total time = 889027.98 ms / 24 tokens llama_print_timings: total time = 880783.90 ms / 24 tokens llama_print_timings: total time = 856475.29 ms / 24 tokens llama_print_timings: total time = 896842.21 ms / 24 tokens llama_print_timings: total time = 878883.53 ms / 24 tokens llama_print_timings: total time = 890122.10 ms / 24 tokens
AFTER
-----
llama_print_timings: total time = 871060.86 ms / 24 tokens llama_print_timings: total time = 825609.53 ms / 24 tokens llama_print_timings: total time = 836854.81 ms / 24 tokens llama_print_timings: total time = 843147.99 ms / 24 tokens llama_print_timings: total time = 831426.65 ms / 24 tokens llama_print_timings: total time = 873939.23 ms / 24 tokens llama_print_timings: total time = 826127.69 ms / 24 tokens llama_print_timings: total time = 835489.26 ms / 24 tokens llama_print_timings: total time = 842589.62 ms / 24 tokens llama_print_timings: total time = 833700.66 ms / 24 tokens llama_print_timings: total time = 875996.19 ms / 24 tokens llama_print_timings: total time = 826401.73 ms / 24 tokens llama_print_timings: total time = 839341.28 ms / 24 tokens llama_print_timings: total time = 841075.10 ms / 24 tokens llama_print_timings: total time = 835136.41 ms / 24 tokens
2. tlb shootdowns from 'cat /proc/interrupts'
BEFORE
------
TLB:
80911532 93691786 100296251 111062810 109769109 109862429 108968588 119175230 115779676 118377498 119325266 120300143 124514185 116697222 121068466 118031913 122660681 117494403 121819907 116960596 120936335 117217061 118630217 122322724 119595577 111693298 119232201 120030377 115334687 113179982 118808254 116353592 140987367 137095516 131724276 139742240 136501150 130428761 127585535 132483981 133430250 133756207 131786710 126365824 129812539 133850040 131742690 125142213 128572830 132234350 131945922 128417707 133355434 129972846 126331823 134050849 133991626 121129038 124637283 132830916 126875507 122322440 125776487 124340278 TLB shootdowns
AFTER
-----
TLB:
2121206 2615108 2983494 2911950 3055086 3092672
3204894 3346082 3286744 3307310 3357296 3315940
3428034 3112596 3143325 3185551 3186493 3322314
3330523 3339663 3156064 3272070 3296309 3198962
3332662 3315870 3234467 3353240 3281234 3300666
3345452 3173097 4009196 3932215 3898735 3726531
3717982 3671726 3728788 3724613 3799147 3691764
3620630 3684655 3666688 3393974 3448651 3487593
3446357 3618418 3671920 3712949 3575264 3715385
3641513 3630897 3691047 3630690 3504933 3662647
3629926 3443044 3832970 3548813 TLB shootdowns--- include/asm-generic/tlb.h | 5 ++ include/linux/fs.h | 12 +++- include/linux/mm_types.h | 6 ++ include/linux/sched.h | 9 +++ kernel/sched/core.c | 1 + mm/internal.h | 94 ++++++++++++++++++++++++- mm/memory.c | 15 ++++ mm/pgtable-generic.c | 2 + mm/rmap.c | 141 +++++++++++++++++++++++++++++++++++--- mm/truncate.c | 55 +++++++++++++-- mm/vmscan.c | 12 +++- 11 files changed, 333 insertions(+), 19 deletions(-)
include/asm-generic/tlb.h +5 -0
@@ -565,6 +565,11 @@ static inline void tlb_start_vma(struct mmu_gather *tlb, struct vm_area_struct *static inline void tlb_end_vma (struct mmu_gather *tlb,struct vm_area_struct *vma) { +/* +* Don't leave stale tlb entries for this vma. +*/ +luf_flush (0 ); +if (tlb->fullmm)return ;
include/linux/fs.h +11 -1
@@ -536,8 +536,18 @@ static inline int mapping_write_begin(struct file *file, loff_t pos, unsigned len, struct folio **foliop, void **fsdata) { - return mapping->a_ops->write_begin(file, mapping, pos, len, foliop, + int ret; + + ret = mapping->a_ops->write_begin (file, mapping, pos, len, foliop, fsdata); + +/* +* Ensure to clean stale tlb entries for this mapping. +*/ +if (!ret) +luf_flush (0 ); + +return ret; } static inline int mapping_write_end(struct file *file,
include/linux/mm_types.h +6 -0
@@ -1339,6 +1339,12 @@ extern void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm);extern void tlb_gather_mmu_fullmm (struct mmu_gather *tlb,struct mm_struct *mm);extern void tlb_finish_mmu (struct mmu_gather *tlb); +#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH ) +void luf_flush (unsigned short luf_key); +#else +static inline void luf_flush (unsigned short luf_key) {} +#endif +struct vm_fault ;/**
include/linux/sched.h +9 -0
@@ -1408,6 +1408,15 @@ struct task_struct {struct tlbflush_unmap_batch tlb_ubc;struct tlbflush_unmap_batch tlb_ubc_takeoff;struct tlbflush_unmap_batch tlb_ubc_ro; +struct tlbflush_unmap_batch tlb_ubc_luf; + +#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH ) +/* +* whether all the mappings of a folio during unmap are read-only +* so that luf can work on the folio +*/ +bool can_luf; +#endif /* Cache last used pipe for splice(): */ struct pipe_inode_info *splice_pipe;
kernel/sched/core.c +1 -0
@@ -5275,6 +5275,7 @@ static struct rq *finish_task_switch(struct task_struct *prev)if (mm) {membarrier_mm_sync_core_before_usermode (mm);mmdrop_lazy_tlb_sched (mm); +luf_flush (0 ); }if (unlikely (prev_state ==TASK_DEAD )) {
mm/internal.h +93 -1
106 lines not on this page · raw message
mm/memory.c +15 -0
@@ -6181,6 +6181,7 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,struct mm_struct *mm = vma->vm_mm;vm_fault_t ret;bool is_droppable; +bool flush = false;__set_current_state (TASK_RUNNING );
@@ -6206,6 +6207,14 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,lru_gen_enter_fault (vma); +/* +* Any potential cases that make pte writable even forcely +* should be considered. +*/ +if (vma->vm_flags & (VM_WRITE |VM_MAYWRITE ) || + flags &FAULT_FLAG_WRITE ) + flush = true; +if (unlikely (is_vm_hugetlb_page (vma))) ret =hugetlb_fault (vma->vm_mm, vma, address, flags);else
@@ -6237,6 +6246,12 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, out:mm_account_fault (mm, regs, address, flags, ret); +/* +* Ensure to clean stale tlb entries for this vma. +*/ +if (flush) +luf_flush (0 ); +return ret; }EXPORT_SYMBOL_GPL (handle_mm_fault);
mm/pgtable-generic.c +2 -0
@@ -99,6 +99,8 @@ pte_t ptep_clear_flush(struct vm_area_struct *vma, unsigned long address, pte =ptep_get_and_clear (mm, address, ptep);if (pte_accessible (mm, pte))flush_tlb_page (vma, address); +else +luf_flush (0 ); return pte; }#endif
mm/rmap.c +133 -8
276 lines not on this page · raw message
mm/truncate.c +48 -7
117 lines not on this page · raw message
mm/vmscan.c +10 -2
@@ -828,6 +828,8 @@ static int __remove_mapping(struct address_space *mapping, struct folio *folio,*/ long remove_mapping (struct address_space *mapping,struct folio *folio) { +long ret =0 ; +if (__remove_mapping (mapping, folio, false,NULL )) {/* * Unfreezing the refcount with 1 effectively
@@ -835,9 +837,15 @@ long remove_mapping(struct address_space *mapping, struct folio *folio)* atomic operation. */ folio_ref_unfreeze (folio,1 ); -return folio_nr_pages (folio); + ret =folio_nr_pages (folio); } -return 0 ; + +/* +* Ensure to clean stale tlb entries for this mapping. +*/ +luf_flush (0 ); + +return ret; }/**
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 17/25 x86/tlb, riscv/tlb, arm64/tlbflush, mm: remove cpus from tlb shootdown that already have been done
In reply to Byungchul Parkluf mechanism performs tlb shootdown for mappings that have been unmapped in lazy manner. However, it doesn't have to perform tlb shootdown to cpus that already have been done by others since the tlb shootdown was desired.
Since luf already introduced its own generation number used as a global timestamp, luf_ugen, it's possible to selectively pick cpus that have been done tlb flush required.
This patch introduced APIs that use the generation number to select and remove those cpus so that it can perform tlb shootdown with a smaller cpumask, for all the CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH archs, x86, riscv, and arm64.
--- arch/arm64/include/asm/tlbflush.h | 26 +++++++ arch/riscv/include/asm/tlbflush.h | 4 ++ arch/riscv/mm/tlbflush.c | 108 ++++++++++++++++++++++++++++++ arch/x86/include/asm/tlbflush.h | 4 ++ arch/x86/mm/tlb.c | 108 ++++++++++++++++++++++++++++++ include/linux/sched.h | 1 + mm/internal.h | 4 ++ mm/page_alloc.c | 32 +++++++-- mm/rmap.c | 46 ++++++++++++- 9 files changed, 327 insertions(+), 6 deletions(-)
arch/arm64/include/asm/tlbflush.h +26 -0
@@ -354,6 +354,32 @@ static inline void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)dsb (ish); } +static inline bool arch_tlbbatch_check_done (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen) +{ +/* +* Nothing is needed in this architecture. +*/ +return true; +} + +static inline bool arch_tlbbatch_diet (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen) +{ +/* +* Nothing is needed in this architecture. +*/ +return true; +} + +static inline void arch_tlbbatch_mark_ugen (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen) +{ +/* nothing to do */ +} + +static inline void arch_mm_mark_ugen (struct mm_struct *mm,unsigned long ugen) +{ +/* nothing to do */ +} +static inline void arch_tlbbatch_clear (struct arch_tlbflush_unmap_batch *batch) {/* nothing to do */
arch/riscv/include/asm/tlbflush.h +4 -0
@@ -65,6 +65,10 @@ void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch, unsigned long uaddr);void arch_flush_tlb_batched_pending (struct mm_struct *mm);void arch_tlbbatch_flush (struct arch_tlbflush_unmap_batch *batch); +bool arch_tlbbatch_check_done (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen); +bool arch_tlbbatch_diet (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen); +void arch_tlbbatch_mark_ugen (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen); +void arch_mm_mark_ugen (struct mm_struct *mm,unsigned long ugen);static inline void arch_tlbbatch_clear (struct arch_tlbflush_unmap_batch *batch) {
arch/riscv/mm/tlbflush.c +108 -0
111 lines not on this page · raw message
arch/x86/include/asm/tlbflush.h +4 -0
@@ -294,6 +294,10 @@ static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm) }extern void arch_tlbbatch_flush (struct arch_tlbflush_unmap_batch *batch); +extern bool arch_tlbbatch_check_done (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen); +extern bool arch_tlbbatch_diet (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen); +extern void arch_tlbbatch_mark_ugen (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen); +extern void arch_mm_mark_ugen (struct mm_struct *mm,unsigned long ugen);static inline void arch_tlbbatch_clear (struct arch_tlbflush_unmap_batch *batch) {
arch/x86/mm/tlb.c +108 -0
114 lines not on this page · raw message
include/linux/sched.h +1 -0
@@ -1403,6 +1403,7 @@ struct task_struct {#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH )int luf_no_shootdown;int luf_takeoff_started; +unsigned long luf_ugen;#endif struct tlbflush_unmap_batch tlb_ubc;
mm/internal.h +4 -0
@@ -1259,6 +1259,7 @@ void try_to_unmap_flush(void);void try_to_unmap_flush_dirty (void );void try_to_unmap_flush_takeoff (void );void flush_tlb_batched_pending (struct mm_struct *mm); +void reset_batch (struct tlbflush_unmap_batch *batch);void fold_batch (struct tlbflush_unmap_batch *dst,struct tlbflush_unmap_batch *src,bool reset);void fold_luf_batch (struct luf_batch *dst,struct luf_batch *src);#else
@@ -1274,6 +1275,9 @@ static inline void try_to_unmap_flush_takeoff(void)static inline void flush_tlb_batched_pending (struct mm_struct *mm) { } +static inline void reset_batch (struct tlbflush_unmap_batch *batch) +{ +}static inline void fold_batch (struct tlbflush_unmap_batch *dst,struct tlbflush_unmap_batch *src,bool reset) { }
mm/page_alloc.c +28 -4
@@ -668,9 +668,11 @@ bool luf_takeoff_start(void)*/ void luf_takeoff_end (void ) { +struct tlbflush_unmap_batch *tlb_ubc_takeoff = ¤t->tlb_ubc_takeoff;unsigned long flags;bool no_shootdown;bool outmost = false; +unsigned long cur_luf_ugen;local_irq_save (flags);VM_WARN_ON (!current->luf_takeoff_started);
@@ -697,10 +699,19 @@ void luf_takeoff_end(void)if (no_shootdown) goto out; + cur_luf_ugen = current->luf_ugen; + + current->luf_ugen =0 ; + +if (cur_luf_ugen &&arch_tlbbatch_diet (&tlb_ubc_takeoff->arch, cur_luf_ugen)) +reset_batch (tlb_ubc_takeoff); +try_to_unmap_flush_takeoff (); out: -if (outmost) +if (outmost) {VM_WARN_ON (current->luf_no_shootdown); +VM_WARN_ON (current->luf_ugen); + } }/*
@@ -757,6 +768,7 @@ bool luf_takeoff_check_and_fold(struct page *page)struct tlbflush_unmap_batch *tlb_ubc_takeoff = ¤t->tlb_ubc_takeoff;unsigned short luf_key =page_luf_key (page);struct luf_batch *lb; +unsigned long lb_ugen;unsigned long flags;/*
@@ -770,13 +782,25 @@ bool luf_takeoff_check_and_fold(struct page *page)if (!luf_key)return true; -if (current->luf_no_shootdown) -return false; - lb = &luf_batch[luf_key];read_lock_irqsave (&lb->lock, flags); + lb_ugen = lb->ugen; + +if (arch_tlbbatch_check_done (&lb->batch.arch, lb_ugen)) { +read_unlock_irqrestore (&lb->lock, flags); +return true; + } + +if (current->luf_no_shootdown) { +read_unlock_irqrestore (&lb->lock, flags); +return false; + } +fold_batch (tlb_ubc_takeoff, &lb->batch, false);read_unlock_irqrestore (&lb->lock, flags); + +if (!current->luf_ugen ||ugen_before (current->luf_ugen, lb_ugen)) + current->luf_ugen = lb_ugen; return true; }#endif
mm/rmap.c +44 -2
96 lines not on this page · raw message
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 18/25 mm/page_alloc: retry 3 times to take pcp pages on luf check failure
In reply to Byungchul Park--- mm/page_alloc.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-)
mm/page_alloc.c +20 -4
@@ -3306,6 +3306,12 @@ struct page *__rmqueue_pcplist(struct zone *zone, unsigned int order, {struct page *page; +/* +* give up taking page from pcp if it fails to take pcp page +* 3 times due to the tlb shootdownable issue. +*/ +int try_luf_pages =3 ; + do { if (list_empty(list)) { int batch = nr_pcp_alloc(pcp, zone, order);
@@ -3320,11 +3326,21 @@ struct page *__rmqueue_pcplist(struct zone *zone, unsigned int order,return NULL ; } - page = list_first_entry(list, struct page, pcp_list); -if (!luf_takeoff_check_and_fold (page)) +list_for_each_entry (page, list, pcp_list) { +if (luf_takeoff_check_and_fold (page)) { +list_del (&page->pcp_list); + pcp->count -=1 << order; +break ; + } +if (!--try_luf_pages) +return NULL ; + } + +/* +* If all the pages in the list fails... +*/ +if (list_entry_is_head (page, list, pcp_list))return NULL ; -list_del (&page->pcp_list); - pcp->count -=1 << order; }while (check_new_pages (page, order));return page;
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 19/25 mm: skip luf tlb flush for luf'd mm that already has been done
In reply to Byungchul ParkFault hander performs tlb flush pended by luf when a new pte becomes to have write permission, no matter whether tlb flush required has been performed or not.
By storing luf generation number, luf_ugen, in struct mm_struct, we can skip unnecessary tlb flush.
--- include/asm-generic/tlb.h | 2 +- include/linux/mm_types.h | 9 +++++ kernel/fork.c | 1 + kernel/sched/core.c | 2 +- mm/memory.c | 22 ++++++++++-- mm/pgtable-generic.c | 2 +- mm/rmap.c | 74 +++++++++++++++++++++++++++++++++++++-- 7 files changed, 104 insertions(+), 8 deletions(-)
include/asm-generic/tlb.h +1 -1
@@ -568,7 +568,7 @@ static inline void tlb_end_vma(struct mmu_gather *tlb, struct vm_area_struct *vm/* * Don't leave stale tlb entries for this vma. */ -luf_flush (0 ); +luf_flush_vma (vma);if (tlb->fullmm)return ;
include/linux/mm_types.h +9 -0
@@ -38,8 +38,10 @@ struct luf_batch {unsigned long ugen; rwlock_t lock; }; +void luf_batch_init (struct luf_batch *lb);#else struct luf_batch {}; +static inline void luf_batch_init (struct luf_batch *lb) {}#endif /*
@@ -1059,6 +1061,9 @@ struct mm_struct {* moving a PROT_NONE mapped page. */ atomic_t tlb_flush_pending; + +/* luf batch for this mm */ +struct luf_batch luf_batch;#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH /* See flush_tlb_batched_pending() */ atomic_t tlb_flush_batched;
@@ -1341,8 +1346,12 @@ extern void tlb_finish_mmu(struct mmu_gather *tlb);#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH )void luf_flush (unsigned short luf_key); +void luf_flush_mm (struct mm_struct *mm); +void luf_flush_vma (struct vm_area_struct *vma);#else static inline void luf_flush (unsigned short luf_key) {} +static inline void luf_flush_mm (struct mm_struct *mm) {} +static inline void luf_flush_vma (struct vm_area_struct *vma) {}#endif struct vm_fault ;
kernel/fork.c +1 -0
@@ -1280,6 +1280,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,memset (&mm->rss_stat,0 ,sizeof (mm->rss_stat));spin_lock_init (&mm->page_table_lock);spin_lock_init (&mm->arg_lock); +luf_batch_init (&mm->luf_batch);mm_init_cpumask (mm);mm_init_aio (mm);mm_init_owner (mm, p);
kernel/sched/core.c +1 -1
@@ -5275,7 +5275,7 @@ static struct rq *finish_task_switch(struct task_struct *prev)if (mm) {membarrier_mm_sync_core_before_usermode (mm);mmdrop_lazy_tlb_sched (mm); -luf_flush (0 ); +luf_flush_mm (mm); }if (unlikely (prev_state ==TASK_DEAD )) {
mm/memory.c +19 -3
@@ -6181,6 +6181,7 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,struct mm_struct *mm = vma->vm_mm;vm_fault_t ret;bool is_droppable; +struct address_space *mapping =NULL ;bool flush = false;__set_current_state (TASK_RUNNING );
@@ -6212,9 +6213,17 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,* should be considered. */ if (vma->vm_flags & (VM_WRITE |VM_MAYWRITE ) || - flags &FAULT_FLAG_WRITE ) + flags &FAULT_FLAG_WRITE ) { flush = true; +/* +* Doesn't care the !VM_SHARED cases because it won't +* update the pages that might be shared with others. +*/ +if (vma->vm_flags &VM_SHARED && vma->vm_file) + mapping = vma->vm_file->f_mapping; + } +if (unlikely (is_vm_hugetlb_page (vma))) ret =hugetlb_fault (vma->vm_mm, vma, address, flags);else
@@ -6249,8 +6258,15 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,/* * Ensure to clean stale tlb entries for this vma. */ -if (flush) -luf_flush (0 ); +if (flush) { +/* +* If it has a VM_SHARED mapping, all the mms involved +* should be luf_flush'ed. +*/ +if (mapping) +luf_flush (0 ); +luf_flush_mm (mm); + }return ret; }
mm/pgtable-generic.c +1 -1
@@ -100,7 +100,7 @@ pte_t ptep_clear_flush(struct vm_area_struct *vma, unsigned long address,if (pte_accessible (mm, pte))flush_tlb_page (vma, address);else -luf_flush (0 ); +luf_flush_vma (vma); return pte; }#endif
mm/rmap.c +72 -2
99 lines not on this page · raw message
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 20/25 mm, fs: skip tlb flushes for luf'd filemap that already has been done
In reply to Byungchul ParkFor luf'd filemap, tlb shootdown is performed when updating page cache, no matter whether tlb flushes required already has been done or not.
By storing luf meta data in struct address_space and updating the luf meta data properly, we can skip unnecessary tlb flush.
--- fs/inode.c | 1 + include/linux/fs.h | 4 ++- include/linux/mm_types.h | 2 ++ mm/memory.c | 4 +-- mm/rmap.c | 59 +++++++++++++++++++++++++--------------- mm/truncate.c | 14 +++++----- mm/vmscan.c | 2 +- 7 files changed, 53 insertions(+), 33 deletions(-)
fs/inode.c +1 -0
@@ -475,6 +475,7 @@ static void __address_space_init_once(struct address_space *mapping)init_rwsem (&mapping->i_mmap_rwsem);INIT_LIST_HEAD (&mapping->i_private_list);spin_lock_init (&mapping->i_private_lock); +luf_batch_init (&mapping->luf_batch); mapping->i_mmap =RB_ROOT_CACHED ; }
include/linux/fs.h +3 -1
@@ -498,6 +498,7 @@ extern const struct address_space_operations empty_aops;* @i_private_lock: For use by the owner of the address_space. * @i_private_list: For use by the owner of the address_space. * @i_private_data: For use by the owner of the address_space. +* @luf_batch: Data to track need of tlb flush by luf. */ struct address_space {struct inode *host;
@@ -519,6 +520,7 @@ struct address_space {struct list_head i_private_list;struct rw_semaphore i_mmap_rwsem;void * i_private_data; +struct luf_batch luf_batch; } __attribute__((aligned(sizeof(long)))) __randomize_layout;/* * On most architectures that alignment is already the case; but
@@ -545,7 +547,7 @@ static inline int mapping_write_begin(struct file *file,* Ensure to clean stale tlb entries for this mapping. */ if (!ret) -luf_flush (0 ); +luf_flush_mapping (mapping);return ret; }
include/linux/mm_types.h +2 -0
@@ -1348,10 +1348,12 @@ extern void tlb_finish_mmu(struct mmu_gather *tlb);void luf_flush (unsigned short luf_key);void luf_flush_mm (struct mm_struct *mm);void luf_flush_vma (struct vm_area_struct *vma); +void luf_flush_mapping (struct address_space *mapping);#else static inline void luf_flush (unsigned short luf_key) {}static inline void luf_flush_mm (struct mm_struct *mm) {}static inline void luf_flush_vma (struct vm_area_struct *vma) {} +static inline void luf_flush_mapping (struct address_space *mapping) {}#endif struct vm_fault ;
mm/memory.c +2 -2
@@ -6261,10 +6261,10 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,if (flush) {/* * If it has a VM_SHARED mapping, all the mms involved -* should be luf_flush'ed. +* in the struct address_space should be luf_flush'ed. */ if (mapping) -luf_flush (0 ); +luf_flush_mapping (mapping);luf_flush_mm (mm); }
mm/rmap.c +37 -22
131 lines not on this page · raw message
mm/truncate.c +7 -7
@@ -128,7 +128,7 @@ void folio_invalidate(struct folio *folio, size_t offset, size_t length)/* * Ensure to clean stale tlb entries for this mapping. */ -luf_flush (0 ); +luf_flush_mapping (folio->mapping); }EXPORT_SYMBOL_GPL (folio_invalidate);
@@ -169,7 +169,7 @@ int truncate_inode_folio(struct address_space *mapping, struct folio *folio)/* * Ensure to clean stale tlb entries for this mapping. */ -luf_flush (0 ); +luf_flush_mapping (mapping);return 0 ; }
@@ -219,7 +219,7 @@ bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end)/* * Ensure to clean stale tlb entries for this mapping. */ -luf_flush (0 ); +luf_flush_mapping (folio->mapping);if (!folio_test_large (folio))return true;
@@ -281,7 +281,7 @@ long mapping_evict_folio(struct address_space *mapping, struct folio *folio)/* * Ensure to clean stale tlb entries for this mapping. */ -luf_flush (0 ); +luf_flush_mapping (mapping);return ret; }
@@ -416,7 +416,7 @@ void truncate_inode_pages_range(struct address_space *mapping,/* * Ensure to clean stale tlb entries for this mapping. */ -luf_flush (0 ); +luf_flush_mapping (mapping); }EXPORT_SYMBOL (truncate_inode_pages_range);
@@ -536,7 +536,7 @@ unsigned long mapping_try_invalidate(struct address_space *mapping,/* * Ensure to clean stale tlb entries for this mapping. */ -luf_flush (0 ); +luf_flush_mapping (mapping);return count; }
@@ -706,7 +706,7 @@ int invalidate_inode_pages2_range(struct address_space *mapping,/* * Ensure to clean stale tlb entries for this mapping. */ -luf_flush (0 ); +luf_flush_mapping (mapping);return ret; }EXPORT_SYMBOL_GPL (invalidate_inode_pages2_range);
mm/vmscan.c +1 -1
@@ -843,7 +843,7 @@ long remove_mapping(struct address_space *mapping, struct folio *folio)/* * Ensure to clean stale tlb entries for this mapping. */ -luf_flush (0 ); +luf_flush_mapping (mapping);return ret; }
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 21/25 mm: perform luf tlb shootdown per zone in batched manner
In reply to Byungchul ParkEach luf page in buddy has its pending tlb shootdown information and performs the corresponding tlb shootdown on exit from buddy. However, every exit from buddy causes small but frequent IPIs. Even though total IPIs get reduced, unnecessary waits on conflict CPUs in IPI handler have been observed via perf profiling.
Thus, made it perfrom luf tlb shootdown per zone in batched manner when pages exit from buddy so as to avoid frequent IPIs.
--- include/linux/mm.h | 44 ++++- include/linux/mm_types.h | 19 +- include/linux/mmzone.h | 9 + include/linux/sched.h | 2 + mm/compaction.c | 10 +- mm/internal.h | 13 +- mm/mm_init.c | 5 + mm/page_alloc.c | 363 +++++++++++++++++++++++++++++++-------- mm/page_reporting.c | 9 +- mm/rmap.c | 6 +- 10 files changed, 383 insertions(+), 97 deletions(-)
include/linux/mm.h +41 -3
@@ -4155,12 +4155,16 @@ int arch_get_shadow_stack_status(struct task_struct *t, unsigned long __user *stint arch_set_shadow_stack_status (struct task_struct *t,unsigned long status);int arch_lock_shadow_stack_status (struct task_struct *t,unsigned long status); -#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH )/* * luf_ugen will start with 2 so that 1 can be regarded as a passed one. */ #define LUF_UGEN_INIT 2 +/* +* zone_ugen will start with 2 so that 1 can be regarded as done. +*/ +#define ZONE_UGEN_INIT 2 +#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH )static inline bool ugen_before (unsigned long a,unsigned long b) {/*
@@ -4171,7 +4175,11 @@ static inline bool ugen_before(unsigned long a, unsigned long b)static inline unsigned long next_ugen (unsigned long ugen) { -if (ugen +1 ) +/* +* Avoid zero even in unsigned short range so as to treat +* '(unsigned short)ugen == 0' as invalid. +*/ +if ((unsigned short )(ugen +1 ))return ugen +1 ;/* * Avoid invalid ugen, zero.
@@ -4181,7 +4189,11 @@ static inline unsigned long next_ugen(unsigned long ugen)static inline unsigned long prev_ugen (unsigned long ugen) { -if (ugen -1 ) +/* +* Avoid zero even in unsigned short range so as to treat +* '(unsigned short)ugen == 0' as invalid. +*/ +if ((unsigned short )(ugen -1 ))return ugen -1 ;/* * Avoid invalid ugen, zero.
@@ -4189,4 +4201,30 @@ static inline unsigned long prev_ugen(unsigned long ugen)return ugen -2 ; }#endif + +/* +* return the biggest ugen but it should be before the real zone_ugen. +*/ +static inline unsigned long page_zone_ugen (struct zone *zone,struct page *page) +{ +unsigned long zone_ugen = zone->zone_ugen; +unsigned short short_zone_ugen = page->zone_ugen; +unsigned long cand1, cand2; + +if (!short_zone_ugen) +return 0 ; + + cand1 = (zone_ugen & ~(unsigned long )USHRT_MAX ) | short_zone_ugen; + cand2 = cand1 -USHRT_MAX -1 ; + +if (!ugen_before (zone_ugen, cand1)) +return cand1; + +return cand2; +} + +static inline void set_page_zone_ugen (struct page *page,unsigned short zone_ugen) +{ + page->zone_ugen = zone_ugen; +}#endif /* _LINUX_MM_H */
include/linux/mm_types.h +14 -5
@@ -132,11 +132,20 @@ struct page {*/ unsigned short order; -/* -* For tracking need of tlb flush, -* by luf(lazy unmap flush). -*/ -unsigned short luf_key; +union { +/* +* For tracking need of +* tlb flush, by +* luf(lazy unmap flush). +*/ +unsigned short luf_key; + +/* +* Casted zone_ugen with +* unsigned short. +*/ +unsigned short zone_ugen; + }; }; }; };
include/linux/mmzone.h +9 -0
@@ -117,6 +117,7 @@ extern int page_group_by_mobility_disabled;struct free_area {struct list_head free_list[MIGRATE_TYPES ];struct list_head pend_list[MIGRATE_TYPES ]; +unsigned long pend_zone_ugen[MIGRATE_TYPES ];unsigned long nr_free; };
@@ -1017,6 +1018,14 @@ struct zone {atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS ];/* Count pages that need tlb shootdown on allocation */ atomic_long_t nr_luf_pages; +/* Generation number for that tlb shootdown has been done */ +unsigned long zone_ugen_done; +/* Generation number to control zone batched tlb shootdown */ +unsigned long zone_ugen; +/* Approximate latest luf_ugen that have ever entered */ +unsigned long luf_ugen; +/* Accumulated tlb batch for this zone */ +struct tlbflush_unmap_batch zone_batch; }____cacheline_internodealigned_in_smp ;enum pgdat_flags {
include/linux/sched.h +2 -0
@@ -1404,6 +1404,8 @@ struct task_struct {int luf_no_shootdown;int luf_takeoff_started;unsigned long luf_ugen; +unsigned long zone_ugen; +unsigned long wait_zone_ugen;#endif struct tlbflush_unmap_batch tlb_ubc;
mm/compaction.c +5 -5
@@ -655,7 +655,7 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, goto isolate_fail; } -if (!luf_takeoff_check (page)) +if (!luf_takeoff_check (cc->zone, page)) goto isolate_fail;/* Found a free page, will break it into order-0 pages */
@@ -691,7 +691,7 @@ static unsigned long isolate_freepages_block(struct compact_control *cc,/* * Check and flush before using the pages taken off. */ -luf_takeoff_end (); +luf_takeoff_end (cc->zone);/* * Be careful to not go outside of the pageblock.
@@ -1613,7 +1613,7 @@ static void fast_isolate_freepages(struct compact_control *cc) order_scanned++; nr_scanned++; -if (unlikely (consider_pend && !luf_takeoff_check (freepage))) +if (unlikely (consider_pend && !luf_takeoff_check (cc->zone, freepage))) goto scan_next; pfn =page_to_pfn (freepage);
@@ -1681,7 +1681,7 @@ static void fast_isolate_freepages(struct compact_control *cc)/* * Check and flush before using the pages taken off. */ -luf_takeoff_end (); +luf_takeoff_end (cc->zone);/* Skip fast search if enough freepages isolated */ if (cc->nr_freepages >= cc->nr_migratepages)
@@ -2418,7 +2418,7 @@ static enum compact_result compact_finished(struct compact_control *cc)*/ luf_takeoff_start (); ret =__compact_finished (cc); -luf_takeoff_end (); +luf_takeoff_end (cc->zone);trace_mm_compaction_finished (cc->zone, cc->order, ret);if (ret ==COMPACT_NO_SUITABLE_PAGE )
mm/internal.h +6 -7
@@ -1592,10 +1592,10 @@ static inline void accept_page(struct page *page)#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH )extern struct luf_batch luf_batch[];bool luf_takeoff_start (void ); -void luf_takeoff_end (void ); +void luf_takeoff_end (struct zone *zone);bool luf_takeoff_no_shootdown (void ); -bool luf_takeoff_check (struct page *page); -bool luf_takeoff_check_and_fold (struct page *page); +bool luf_takeoff_check (struct zone *zone,struct page *page); +bool luf_takeoff_check_and_fold (struct zone *zone,struct page *page);static inline bool non_luf_pages_ok (struct zone *zone) {
@@ -1605,7 +1605,6 @@ static inline bool non_luf_pages_ok(struct zone *zone)return nr_free - nr_luf_pages > min_wm; } -unsigned short fold_unmap_luf (void );/*
@@ -1693,10 +1692,10 @@ static inline bool can_luf_vma(struct vm_area_struct *vma) }#else /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ static inline bool luf_takeoff_start (void ) {return false; } -static inline void luf_takeoff_end (void ) {} +static inline void luf_takeoff_end (struct zone *zone) {}static inline bool luf_takeoff_no_shootdown (void ) {return true; } -static inline bool luf_takeoff_check (struct page *page) {return true; } -static inline bool luf_takeoff_check_and_fold (struct page *page) {return true; } +static inline bool luf_takeoff_check (struct zone *zone,struct page *page) {return true; } +static inline bool luf_takeoff_check_and_fold (struct zone *zone,struct page *page) {return true; }static inline bool non_luf_pages_ok (struct zone *zone) {return true; }static inline unsigned short fold_unmap_luf (void ) {return 0 ; }
mm/mm_init.c +5 -0
@@ -1400,6 +1400,7 @@ static void __meminit zone_init_free_lists(struct zone *zone)for_each_migratetype_order (order, t) {INIT_LIST_HEAD (&zone->free_area[order].free_list[t]);INIT_LIST_HEAD (&zone->free_area[order].pend_list[t]); + zone->free_area[order].pend_zone_ugen[t] =ZONE_UGEN_INIT ; zone->free_area[order].nr_free =0 ; }
@@ -1407,6 +1408,10 @@ static void __meminit zone_init_free_lists(struct zone *zone)INIT_LIST_HEAD (&zone->unaccepted_pages);#endif atomic_long_set (&zone->nr_luf_pages,0 ); + zone->zone_ugen_done =ZONE_UGEN_INIT -1 ; + zone->zone_ugen =ZONE_UGEN_INIT ; + zone->luf_ugen =LUF_UGEN_INIT -1 ; +reset_batch (&zone->zone_batch); } void __meminit init_currently_empty_zone(struct zone *zone,
mm/page_alloc.c +291 -72
669 lines not on this page · raw message
mm/page_reporting.c +5 -4
@@ -118,7 +118,8 @@ page_reporting_drain(struct page_reporting_dev_info *prdev,/* * Ensure private is zero before putting into the -* allocator. +* allocator. tlb shootdown has already been performed +* at isolation. */ set_page_private (page,0 );
@@ -194,7 +195,7 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,if (PageReported (page))continue ; -if (unlikely (consider_pend && !luf_takeoff_check (page))) { +if (unlikely (consider_pend && !luf_takeoff_check (zone, page))) {VM_WARN_ON (1 );continue ; }
@@ -238,7 +239,7 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,/* * Check and flush before using the pages taken off. */ -luf_takeoff_end (); +luf_takeoff_end (zone);/* begin processing pages in local list */ err = prdev->report (prdev, sgl,PAGE_REPORTING_CAPACITY );
@@ -283,7 +284,7 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,/* * Check and flush before using the pages taken off. */ -luf_takeoff_end (); +luf_takeoff_end (zone);return err; }
mm/rmap.c +5 -1
@@ -650,7 +650,11 @@ static unsigned long new_luf_ugen(void) {unsigned long ugen =atomic_long_inc_return (&luf_ugen); -if (!ugen) +/* +* Avoid zero even in unsigned short range so as to treat +* '(unsigned short)ugen == 0' as invalid. +*/ +if (!(unsigned short )ugen) ugen =atomic_long_inc_return (&luf_ugen);return ugen;
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 22/25 mm/page_alloc: not allow to tlb shootdown if !preemptable() && non_luf_pages_ok()
In reply to Byungchul ParkDo not perform tlb shootdown if the context is in preempt disable and there are already enough non luf pages, not to hurt preemptibility.
--- mm/compaction.c | 6 +++--- mm/internal.h | 5 +++-- mm/page_alloc.c | 27 +++++++++++++++------------ mm/page_isolation.c | 2 +- mm/page_reporting.c | 4 ++-- 5 files changed, 24 insertions(+), 20 deletions(-)
mm/compaction.c +3 -3
@@ -606,7 +606,7 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, page =pfn_to_page (blockpfn); -luf_takeoff_start (); +luf_takeoff_start (cc->zone);/* Isolate free pages. */ for (; blockpfn < end_pfn; blockpfn += stride, page += stride) {int isolated;
@@ -1603,7 +1603,7 @@ static void fast_isolate_freepages(struct compact_control *cc)if (!area->nr_free)continue ; - can_shootdown =luf_takeoff_start (); + can_shootdown =luf_takeoff_start (cc->zone);spin_lock_irqsave (&cc->zone->lock, flags); freelist = &area->free_list[MIGRATE_MOVABLE ]; retry:
@@ -2416,7 +2416,7 @@ static enum compact_result compact_finished(struct compact_control *cc)* luf_takeoff_{start,end}() is required to identify whether * this compaction context is tlb shootdownable for luf'd pages. */ -luf_takeoff_start (); +luf_takeoff_start (cc->zone); ret =__compact_finished (cc);luf_takeoff_end (cc->zone);
mm/internal.h +3 -2
@@ -1591,7 +1591,7 @@ static inline void accept_page(struct page *page)#endif /* CONFIG_UNACCEPTED_MEMORY */ #if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH )extern struct luf_batch luf_batch[]; -bool luf_takeoff_start (void ); +bool luf_takeoff_start (struct zone *zone);void luf_takeoff_end (struct zone *zone);bool luf_takeoff_no_shootdown (void );bool luf_takeoff_check (struct zone *zone,struct page *page);
@@ -1605,6 +1605,7 @@ static inline bool non_luf_pages_ok(struct zone *zone)return nr_free - nr_luf_pages > min_wm; } +unsigned short fold_unmap_luf (void );/*
@@ -1691,7 +1692,7 @@ static inline bool can_luf_vma(struct vm_area_struct *vma) return true; }#else /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ -static inline bool luf_takeoff_start (void ) {return false; } +static inline bool luf_takeoff_start (struct zone *zone) {return false; }static inline void luf_takeoff_end (struct zone *zone) {}static inline bool luf_takeoff_no_shootdown (void ) {return true; }static inline bool luf_takeoff_check (struct zone *zone,struct page *page) {return true; }
mm/page_alloc.c +15 -12
@@ -623,22 +623,25 @@ compaction_capture(struct capture_control *capc, struct page *page,#endif /* CONFIG_COMPACTION */ #if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH ) -static bool no_shootdown_context (void ) +static bool no_shootdown_context (struct zone *zone) {/* -* If it performs with irq disabled, that might cause a deadlock. -* Avoid tlb shootdown in this case. +* Tries to avoid tlb shootdown if !preemptible(). However, it +* should be allowed under heavy memory pressure. */ +if (zone &&non_luf_pages_ok (zone)) +return !(preemptible () &&in_task ()); +return !(!irqs_disabled () &&in_task ()); }/* * Can be called with zone lock released and irq enabled. */ -bool luf_takeoff_start (void ) +bool luf_takeoff_start (struct zone *zone) {unsigned long flags; -bool no_shootdown =no_shootdown_context (); +bool no_shootdown =no_shootdown_context (zone);local_irq_save (flags);
@@ -2591,7 +2594,7 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac,* luf_takeoff_{start,end}() is required for * get_page_from_free_area() to use luf_takeoff_check(). */ -luf_takeoff_start (); +luf_takeoff_start (zone);spin_lock_irqsave (&zone->lock, flags);for (order =0 ; order <NR_PAGE_ORDERS ; order++) {struct free_area *area = &(zone->free_area[order]);
@@ -2796,7 +2799,7 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order,unsigned long flags;int i; -luf_takeoff_start (); +luf_takeoff_start (zone);spin_lock_irqsave (&zone->lock, flags);for (i =0 ; i < count; ++i) { struct page *page = __rmqueue(zone, order, migratetype,
@@ -3422,7 +3425,7 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, do { page = NULL; - luf_takeoff_start(); + luf_takeoff_start(zone); spin_lock_irqsave(&zone->lock, flags); if (alloc_flags & ALLOC_HIGHATOMIC) page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);
@@ -3567,7 +3570,7 @@ static struct page *rmqueue_pcplist(struct zone *preferred_zone,struct page *page;unsigned long __maybe_unused UP_flags; -luf_takeoff_start (); +luf_takeoff_start (NULL );/* spin_trylock may fail due to a parallel drain or IRQ reentrancy. */ pcp_trylock_prepare (UP_flags); pcp =pcp_spin_trylock (zone->per_cpu_pageset);
@@ -5190,7 +5193,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,if (unlikely (!zone)) goto failed; -luf_takeoff_start (); +luf_takeoff_start (NULL );/* spin_trylock may fail due to a parallel drain or IRQ reentrancy. */ pcp_trylock_prepare (UP_flags); pcp =pcp_spin_trylock (zone->per_cpu_pageset);
@@ -7294,7 +7297,7 @@ unsigned long __offline_isolated_pages(unsigned long start_pfn,offline_mem_sections (pfn, end_pfn); zone =page_zone (pfn_to_page (pfn)); -luf_takeoff_start (); +luf_takeoff_start (zone);spin_lock_irqsave (&zone->lock, flags);while (pfn < end_pfn) { page =pfn_to_page (pfn);
@@ -7412,7 +7415,7 @@ bool take_page_off_buddy(struct page *page)unsigned int order;bool ret = false; -luf_takeoff_start (); +luf_takeoff_start (zone);spin_lock_irqsave (&zone->lock, flags);for (order =0 ; order <NR_PAGE_ORDERS ; order++) {struct page *page_head = page - (pfn & ((1 << order) -1 ));
mm/page_isolation.c +1 -1
@@ -211,7 +211,7 @@ static void unset_migratetype_isolate(struct page *page, int migratetype)struct page *buddy; zone =page_zone (page); -luf_takeoff_start (); +luf_takeoff_start (zone);spin_lock_irqsave (&zone->lock, flags);if (!is_migrate_isolate_page (page)) goto out;
mm/page_reporting.c +2 -2
@@ -170,7 +170,7 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,if (free_area_empty (area, mt))return err; - can_shootdown =luf_takeoff_start (); + can_shootdown =luf_takeoff_start (zone);spin_lock_irq (&zone->lock);/*
@@ -250,7 +250,7 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,/* update budget to reflect call to report function */ budget--; -luf_takeoff_start (); +luf_takeoff_start (zone);/* reacquire zone lock and resume processing */ spin_lock_irq (&zone->lock);
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 23/25 mm/migrate: apply luf mechanism to unmapping during migration
In reply to Byungchul ParkA new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read only and were unmapped, since the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.
Applied the mechanism to unmapping during migration.
--- include/linux/mm.h | 2 ++ include/linux/rmap.h | 2 +- mm/migrate.c | 66 ++++++++++++++++++++++++++++++++++---------- mm/rmap.c | 15 ++++++---- mm/swap.c | 2 +- 5 files changed, 64 insertions(+), 23 deletions(-)
include/linux/mm.h +2 -0
@@ -1489,6 +1489,8 @@ static inline void folio_put(struct folio *folio)__folio_put (folio); } +void page_cache_release (struct folio *folio); +/** * folio_put_refs - Reduce the reference count on a folio. * @folio: The folio.
include/linux/rmap.h +1 -1
@@ -660,7 +660,7 @@ static inline int folio_try_share_anon_rmap_pmd(struct folio *folio,int folio_referenced (struct folio *,int is_locked,struct mem_cgroup *memcg,unsigned long *vm_flags); -void try_to_migrate (struct folio *folio,enum ttu_flags flags); +bool try_to_migrate (struct folio *folio,enum ttu_flags flags);void try_to_unmap (struct folio *,enum ttu_flags flags); int make_device_exclusive_range(struct mm_struct *mm, unsigned long start,
mm/migrate.c +51 -15
166 lines not on this page · raw message
mm/rmap.c +9 -6
@@ -2750,8 +2750,9 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,* * Tries to remove all the page table entries which are mapping this folio and * replace them with special swap entries. Caller must hold the folio lock. +* Return true if all the mappings are read-only, otherwise false. */ -void try_to_migrate (struct folio *folio,enum ttu_flags flags) +bool try_to_migrate (struct folio *folio,enum ttu_flags flags) {struct rmap_walk_control rwc = { .rmap_one = try_to_migrate_one,
@@ -2769,11 +2770,11 @@ void try_to_migrate(struct folio *folio, enum ttu_flags flags)*/ if (WARN_ON_ONCE (flags & ~(TTU_RMAP_LOCKED |TTU_SPLIT_HUGE_PMD |TTU_SYNC |TTU_BATCH_FLUSH ))) -return ; +return false;if (folio_is_zone_device (folio) && (!folio_is_device_private (folio) && !folio_is_device_coherent (folio))) -return ; +return false;/* * During exec, a temporary VMA is setup and later moved.
@@ -2793,10 +2794,12 @@ void try_to_migrate(struct folio *folio, enum ttu_flags flags)else rmap_walk (folio, &rwc); -if (can_luf_test ()) +if (can_luf_test ()) {fold_batch (tlb_ubc_luf, tlb_ubc_ro, true); -else -fold_batch (tlb_ubc, tlb_ubc_ro, true); +return true; + } +fold_batch (tlb_ubc, tlb_ubc_ro, true); +return false; }#ifdef CONFIG_DEVICE_PRIVATE
mm/swap.c +1 -1
@@ -84,7 +84,7 @@ static void __page_cache_release(struct folio *folio, struct lruvec **lruvecp,* This path almost never happens for VM activity - pages are normally freed * in batches. But it gets used by networking - and for compound pages. */ -static void page_cache_release (struct folio *folio) +void page_cache_release (struct folio *folio) {struct lruvec *lruvec =NULL ;unsigned long flags;
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 24/25 mm/vmscan: apply luf mechanism to unmapping during folio reclaim
In reply to Byungchul ParkA new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read only and were unmapped, since the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.
Applied the mechanism to unmapping during folio reclaim.
--- include/linux/rmap.h | 5 +++-- mm/rmap.c | 11 +++++++---- mm/vmscan.c | 37 ++++++++++++++++++++++++++++++++----- 3 files changed, 42 insertions(+), 11 deletions(-)
include/linux/rmap.h +3 -2
@@ -661,7 +661,7 @@ int folio_referenced(struct folio *, int is_locked, struct mem_cgroup *memcg, unsigned long *vm_flags);bool try_to_migrate (struct folio *folio,enum ttu_flags flags); -void try_to_unmap (struct folio *,enum ttu_flags flags); +bool try_to_unmap (struct folio *,enum ttu_flags flags); int make_device_exclusive_range(struct mm_struct *mm, unsigned long start, unsigned long end, struct page **pages,
@@ -794,8 +794,9 @@ static inline int folio_referenced(struct folio *folio, int is_locked,return 0 ; } -static inline void try_to_unmap (struct folio *folio,enum ttu_flags flags) +static inline bool try_to_unmap (struct folio *folio,enum ttu_flags flags) { +return false; }static inline int folio_mkclean (struct folio *folio)
mm/rmap.c +7 -4
@@ -2386,10 +2386,11 @@ static int folio_not_mapped(struct folio *folio)* Tries to remove all the page table entries which are mapping this * folio. It is the caller's responsibility to check if the folio is * still mapped if needed (use TTU_SYNC to prevent accounting races). +* Return true if all the mappings are read-only, otherwise false. * * Context: Caller must hold the folio lock. */ -void try_to_unmap (struct folio *folio,enum ttu_flags flags) +bool try_to_unmap (struct folio *folio,enum ttu_flags flags) {struct rmap_walk_control rwc = { .rmap_one = try_to_unmap_one,
@@ -2408,10 +2409,12 @@ void try_to_unmap(struct folio *folio, enum ttu_flags flags)else rmap_walk (folio, &rwc); -if (can_luf_test ()) +if (can_luf_test ()) {fold_batch (tlb_ubc_luf, tlb_ubc_ro, true); -else -fold_batch (tlb_ubc, tlb_ubc_ro, true); +return true; + } +fold_batch (tlb_ubc, tlb_ubc_ro, true); +return false; }/*
mm/vmscan.c +32 -5
@@ -1092,14 +1092,17 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, struct reclaim_stat *stat, bool ignore_references) {struct folio_batch free_folios; +struct folio_batch free_folios_luf;LIST_HEAD (ret_folios);LIST_HEAD (demote_folios);unsigned int nr_reclaimed =0 , nr_demoted =0 ;unsigned int pgactivate =0 ;bool do_demote_pass;struct swap_iocb *plug =NULL ; +unsigned short luf_key;folio_batch_init (&free_folios); +folio_batch_init (&free_folios_luf);memset (stat,0 ,sizeof (*stat));cond_resched (); do_demote_pass =can_demote (pgdat->node_id, sc);
@@ -1111,6 +1114,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,enum folio_references references =FOLIOREF_RECLAIM ;bool dirty, writeback;unsigned int nr_pages; +bool can_luf = false;cond_resched ();
@@ -1344,7 +1348,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (folio_test_large (folio)) flags |=TTU_SYNC ; -try_to_unmap (folio, flags); + can_luf =try_to_unmap (folio, flags);if (folio_mapped (folio)) { stat->nr_unmap_fail += nr_pages; if (!was_swapbacked &&
@@ -1488,6 +1492,8 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,* leave it off the LRU). */ nr_reclaimed += nr_pages; +if (can_luf) +luf_flush (fold_unmap_luf ());continue ; } }
@@ -1520,6 +1526,19 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, nr_reclaimed += nr_pages;folio_unqueue_deferred_split (folio); + +if (can_luf) { +if (folio_batch_add (&free_folios_luf, folio) ==0 ) { +mem_cgroup_uncharge_folios (&free_folios); +mem_cgroup_uncharge_folios (&free_folios_luf); + luf_key =fold_unmap_luf (); +try_to_unmap_flush (); +free_unref_folios (&free_folios,0 ); +free_unref_folios (&free_folios_luf, luf_key); + } +continue ; + } +if (folio_batch_add (&free_folios, folio) ==0 ) {mem_cgroup_uncharge_folios (&free_folios);try_to_unmap_flush ();
@@ -1554,9 +1573,21 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,list_add (&folio->lru, &ret_folios);VM_BUG_ON_FOLIO (folio_test_lru (folio) ||folio_test_unevictable (folio), folio); +if (can_luf) +luf_flush (fold_unmap_luf ()); }/* 'folio_list' is always empty here */ +/* +* Finalize this turn before demote_folio_list(). +*/ +mem_cgroup_uncharge_folios (&free_folios); +mem_cgroup_uncharge_folios (&free_folios_luf); + luf_key =fold_unmap_luf (); +try_to_unmap_flush (); +free_unref_folios (&free_folios,0 ); +free_unref_folios (&free_folios_luf, luf_key); +/* Migrate folios selected for demotion */ nr_demoted =demote_folio_list (&demote_folios, pgdat); nr_reclaimed += nr_demoted;
@@ -1590,10 +1621,6 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, pgactivate = stat->nr_activate[0 ] + stat->nr_activate[1 ]; -mem_cgroup_uncharge_folios (&free_folios); -try_to_unmap_flush (); -free_unref_folios (&free_folios,0 ); -list_splice (&ret_folios, folio_list);count_vm_events (PGACTIVATE , pgactivate);
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 25/25 mm/luf: implement luf debug feature
In reply to Byungchul ParkWe need luf debug feature to detect when luf goes wrong by any chance. As a RFC, suggest a simple implementation to report problematic situations by luf.
--- arch/riscv/include/asm/tlbflush.h | 3 + arch/riscv/mm/tlbflush.c | 35 ++++- arch/x86/include/asm/pgtable.h | 10 ++ arch/x86/include/asm/tlbflush.h | 3 + arch/x86/mm/pgtable.c | 10 ++ arch/x86/mm/tlb.c | 35 ++++- include/linux/highmem-internal.h | 5 + include/linux/mm.h | 20 ++- include/linux/mm_types.h | 16 +-- include/linux/mm_types_task.h | 16 +++ include/linux/sched.h | 5 + mm/highmem.c | 1 + mm/memory.c | 12 ++ mm/page_alloc.c | 34 ++++- mm/page_ext.c | 3 + mm/rmap.c | 229 ++++++++++++++++++++++++++++++ 16 files changed, 418 insertions(+), 19 deletions(-)
arch/riscv/include/asm/tlbflush.h +3 -0
@@ -69,6 +69,9 @@ bool arch_tlbbatch_check_done(struct arch_tlbflush_unmap_batch *batch, unsignedbool arch_tlbbatch_diet (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen);void arch_tlbbatch_mark_ugen (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen);void arch_mm_mark_ugen (struct mm_struct *mm,unsigned long ugen); +#ifdef CONFIG_LUF_DEBUG +extern void print_lufd_arch (void ); +#endif static inline void arch_tlbbatch_clear (struct arch_tlbflush_unmap_batch *batch) {
arch/riscv/mm/tlbflush.c +34 -1
@@ -216,6 +216,25 @@ static int __init luf_init_arch(void) }early_initcall (luf_init_arch); +#ifdef CONFIG_LUF_DEBUG +static DEFINE_SPINLOCK (luf_debug_lock); +#define lufd_lock (f) spin_lock_irqsave(&luf_debug_lock, (f)) +#define lufd_unlock (f) spin_unlock_irqrestore(&luf_debug_lock, (f)) + +void print_lufd_arch (void ) +{ +int cpu; + +pr_cont ("LUFD ARCH:" ); +for_each_cpu (cpu, cpu_possible_mask) +pr_cont (" %lu" ,atomic_long_read (per_cpu_ptr (&ugen_done, cpu))); +pr_cont ("\n" ); +} +#else +#define lufd_lock (f) do { (void)(f); } while(0) +#define lufd_unlock (f) do { (void)(f); } while(0) +#endif +/* * batch will not be updated. */
@@ -223,17 +242,22 @@ bool arch_tlbbatch_check_done(struct arch_tlbflush_unmap_batch *batch, unsigned long ugen) {int cpu; +unsigned long flags;if (!ugen) goto out; +lufd_lock (flags);for_each_cpu (cpu, &batch->cpumask) {unsigned long done; done =atomic_long_read (per_cpu_ptr (&ugen_done, cpu)); -if (ugen_before (done, ugen)) +if (ugen_before (done, ugen)) { +lufd_unlock (flags);return false; + } } +lufd_unlock (flags);return true; out:return cpumask_empty (&batch->cpumask);
@@ -243,10 +267,12 @@ bool arch_tlbbatch_diet(struct arch_tlbflush_unmap_batch *batch, unsigned long ugen) {int cpu; +unsigned long flags;if (!ugen) goto out; +lufd_lock (flags);for_each_cpu (cpu, &batch->cpumask) {unsigned long done;
@@ -254,6 +280,7 @@ bool arch_tlbbatch_diet(struct arch_tlbflush_unmap_batch *batch,if (!ugen_before (done, ugen))cpumask_clear_cpu (cpu, &batch->cpumask); } +lufd_unlock (flags); out:return cpumask_empty (&batch->cpumask); }
@@ -262,10 +289,12 @@ void arch_tlbbatch_mark_ugen(struct arch_tlbflush_unmap_batch *batch, unsigned long ugen) {int cpu; +unsigned long flags;if (!ugen)return ; +lufd_lock (flags);for_each_cpu (cpu, &batch->cpumask) {atomic_long_t *done =per_cpu_ptr (&ugen_done, cpu);unsigned long old =atomic_long_read (done);
@@ -283,15 +312,18 @@ void arch_tlbbatch_mark_ugen(struct arch_tlbflush_unmap_batch *batch,*/ atomic_long_cmpxchg (done, old, ugen); } +lufd_unlock (flags); }void arch_mm_mark_ugen (struct mm_struct *mm,unsigned long ugen) {int cpu; +unsigned long flags;if (!ugen)return ; +lufd_lock (flags);for_each_cpu (cpu,mm_cpumask (mm)) {atomic_long_t *done =per_cpu_ptr (&ugen_done, cpu);unsigned long old =atomic_long_read (done);
@@ -309,4 +341,5 @@ void arch_mm_mark_ugen(struct mm_struct *mm, unsigned long ugen)*/ atomic_long_cmpxchg (done, old, ugen); } +lufd_unlock (flags); }
arch/x86/include/asm/pgtable.h +10 -0
@@ -695,12 +695,22 @@ static inline pud_t pud_mkyoung(pud_t pud)return pud_set_flags (pud, _PAGE_ACCESSED); } +#ifdef CONFIG_LUF_DEBUG +pud_t pud_mkwrite (pud_t pud); +static inline pud_t __pud_mkwrite (pud_t pud) +{ + pud =pud_set_flags (pud, _PAGE_RW); + +return pud_clear_saveddirty (pud); +} +#else static inline pud_t pud_mkwrite (pud_t pud) { pud =pud_set_flags (pud, _PAGE_RW);return pud_clear_saveddirty (pud); } +#endif #ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY static inline int pte_soft_dirty (pte_t pte)
arch/x86/include/asm/tlbflush.h +3 -0
@@ -298,6 +298,9 @@ extern bool arch_tlbbatch_check_done(struct arch_tlbflush_unmap_batch *batch, unextern bool arch_tlbbatch_diet (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen);extern void arch_tlbbatch_mark_ugen (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen);extern void arch_mm_mark_ugen (struct mm_struct *mm,unsigned long ugen); +#ifdef CONFIG_LUF_DEBUG +extern void print_lufd_arch (void ); +#endif static inline void arch_tlbbatch_clear (struct arch_tlbflush_unmap_batch *batch) {
arch/x86/mm/pgtable.c +10 -0
@@ -904,6 +904,7 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) { +lufd_check_pages (pte_page (pte),0 );if (vma->vm_flags &VM_SHADOW_STACK )return pte_mkwrite_shstk (pte);
@@ -914,6 +915,7 @@ pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)pmd_t pmd_mkwrite (pmd_t pmd,struct vm_area_struct *vma) { +lufd_check_pages (pmd_page (pmd),PMD_ORDER );if (vma->vm_flags &VM_SHADOW_STACK )return pmd_mkwrite_shstk (pmd);
@@ -922,6 +924,14 @@ pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)return pmd_clear_saveddirty (pmd); } +#ifdef CONFIG_LUF_DEBUG +pud_t pud_mkwrite (pud_t pud) +{ +lufd_check_pages (pud_page (pud),PUD_ORDER ); +return __pud_mkwrite (pud); +} +#endif +void arch_check_zapped_pte (struct vm_area_struct *vma,pte_t pte) {/*
arch/x86/mm/tlb.c +34 -1
@@ -1283,6 +1283,25 @@ static int __init luf_init_arch(void) }early_initcall (luf_init_arch); +#ifdef CONFIG_LUF_DEBUG +static DEFINE_SPINLOCK (luf_debug_lock); +#define lufd_lock (f) spin_lock_irqsave(&luf_debug_lock, (f)) +#define lufd_unlock (f) spin_unlock_irqrestore(&luf_debug_lock, (f)) + +void print_lufd_arch (void ) +{ +int cpu; + +pr_cont ("LUFD ARCH:" ); +for_each_cpu (cpu, cpu_possible_mask) +pr_cont (" %lu" ,atomic_long_read (per_cpu_ptr (&ugen_done, cpu))); +pr_cont ("\n" ); +} +#else +#define lufd_lock (f) do { (void)(f); } while(0) +#define lufd_unlock (f) do { (void)(f); } while(0) +#endif +/* * batch will not be updated. */
@@ -1290,17 +1309,22 @@ bool arch_tlbbatch_check_done(struct arch_tlbflush_unmap_batch *batch, unsigned long ugen) {int cpu; +unsigned long flags;if (!ugen) goto out; +lufd_lock (flags);for_each_cpu (cpu, &batch->cpumask) {unsigned long done; done =atomic_long_read (per_cpu_ptr (&ugen_done, cpu)); -if (ugen_before (done, ugen)) +if (ugen_before (done, ugen)) { +lufd_unlock (flags);return false; + } } +lufd_unlock (flags);return true; out:return cpumask_empty (&batch->cpumask);
@@ -1310,10 +1334,12 @@ bool arch_tlbbatch_diet(struct arch_tlbflush_unmap_batch *batch, unsigned long ugen) {int cpu; +unsigned long flags;if (!ugen) goto out; +lufd_lock (flags);for_each_cpu (cpu, &batch->cpumask) {unsigned long done;
@@ -1321,6 +1347,7 @@ bool arch_tlbbatch_diet(struct arch_tlbflush_unmap_batch *batch,if (!ugen_before (done, ugen))cpumask_clear_cpu (cpu, &batch->cpumask); } +lufd_unlock (flags); out:return cpumask_empty (&batch->cpumask); }
@@ -1329,10 +1356,12 @@ void arch_tlbbatch_mark_ugen(struct arch_tlbflush_unmap_batch *batch, unsigned long ugen) {int cpu; +unsigned long flags;if (!ugen)return ; +lufd_lock (flags);for_each_cpu (cpu, &batch->cpumask) {atomic_long_t *done =per_cpu_ptr (&ugen_done, cpu);unsigned long old =atomic_long_read (done);
@@ -1350,15 +1379,18 @@ void arch_tlbbatch_mark_ugen(struct arch_tlbflush_unmap_batch *batch,*/ atomic_long_cmpxchg (done, old, ugen); } +lufd_unlock (flags); }void arch_mm_mark_ugen (struct mm_struct *mm,unsigned long ugen) {int cpu; +unsigned long flags;if (!ugen)return ; +lufd_lock (flags);for_each_cpu (cpu,mm_cpumask (mm)) {atomic_long_t *done =per_cpu_ptr (&ugen_done, cpu);unsigned long old =atomic_long_read (done);
@@ -1376,6 +1408,7 @@ void arch_mm_mark_ugen(struct mm_struct *mm, unsigned long ugen)*/ atomic_long_cmpxchg (done, old, ugen); } +lufd_unlock (flags); }void arch_tlbbatch_flush (struct arch_tlbflush_unmap_batch *batch)
include/linux/highmem-internal.h +5 -0
@@ -41,6 +41,7 @@ static inline void *kmap(struct page *page) {void *addr; +lufd_check_pages (page,0 );might_sleep ();if (!PageHighMem (page)) addr =page_address (page);
@@ -161,6 +162,7 @@ static inline struct page *kmap_to_page(void *addr)static inline void *kmap (struct page *page) { +lufd_check_pages (page,0 );might_sleep ();return page_address (page); }
@@ -177,11 +179,13 @@ static inline void kunmap(struct page *page)static inline void *kmap_local_page (struct page *page) { +lufd_check_pages (page,0 );return page_address (page); }static inline void *kmap_local_folio (struct folio *folio,size_t offset) { +lufd_check_folio (folio);return page_address (&folio->page) + offset; }
@@ -204,6 +208,7 @@ static inline void __kunmap_local(const void *addr)static inline void *kmap_atomic (struct page *page) { +lufd_check_pages (page,0 );if (IS_ENABLED (CONFIG_PREEMPT_RT ))migrate_disable ();else
include/linux/mm.h +19 -1
@@ -45,6 +45,24 @@ extern int sysctl_page_lock_unfairness;void mm_core_init (void );void init_mm_internals (void ); +#ifdef CONFIG_LUF_DEBUG +void lufd_check_folio (struct folio *f); +void lufd_check_pages (const struct page *p,unsigned int order); +void lufd_check_zone_pages (struct zone *zone,struct page *page,unsigned int order); +void lufd_check_queued_pages (void ); +void lufd_queue_page_for_check (struct page *page,int order); +void lufd_mark_folio (struct folio *f,unsigned short luf_key); +void lufd_mark_pages (struct page *p,unsigned int order,unsigned short luf_key); +#else +static inline void lufd_check_folio (struct folio *f) {} +static inline void lufd_check_pages (const struct page *p,unsigned int order) {} +static inline void lufd_check_zone_pages (struct zone *zone,struct page *page,unsigned int order) {} +static inline void lufd_check_queued_pages (void ) {} +static inline void lufd_queue_page_for_check (struct page *page,int order) {} +static inline void lufd_mark_folio (struct folio *f,unsigned short luf_key) {} +static inline void lufd_mark_pages (struct page *p,unsigned int order,unsigned short luf_key) {} +#endif +#ifndef CONFIG_NUMA /* Don't use mapnrs, do it properly */ extern unsigned long max_mapnr;
@@ -114,7 +132,7 @@ extern int mmap_rnd_compat_bits __read_mostly;#endif #ifndef page_to_virt -#define page_to_virt (x) __va(PFN_PHYS(page_to_pfn(x))) +#define page_to_virt (x) ({ lufd_check_pages(x, 0); __va(PFN_PHYS(page_to_pfn(x)));})#endif #ifndef lm_alias
include/linux/mm_types.h +4 -12
@@ -22,6 +22,10 @@#include <asm/mmu.h> +#ifdef CONFIG_LUF_DEBUG +extern struct page_ext_operations luf_debug_ops; +#endif +#ifndef AT_VECTOR_SIZE_ARCH #define AT_VECTOR_SIZE_ARCH 0#endif
@@ -32,18 +36,6 @@struct address_space ;struct mem_cgroup ; -#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH -struct luf_batch { -struct tlbflush_unmap_batch batch; -unsigned long ugen; -rwlock_t lock; -}; -void luf_batch_init (struct luf_batch *lb); -#else -struct luf_batch {}; -static inline void luf_batch_init (struct luf_batch *lb) {} -#endif -/* * Each physical page in the system has a struct page associated with * it to keep track of whatever it is we are using the page for at the
include/linux/mm_types_task.h +16 -0
@@ -10,6 +10,7 @@#include <linux/align.h> #include <linux/types.h> +#include <linux/spinlock_types.h> #include <asm/page.h>
@@ -88,4 +89,19 @@ struct tlbflush_unmap_batch { #endif }; +#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH +struct luf_batch { +struct tlbflush_unmap_batch batch; +unsigned long ugen; +rwlock_t lock; +}; +void luf_batch_init (struct luf_batch *lb); +#else +struct luf_batch {}; +static inline void luf_batch_init (struct luf_batch *lb) {} +#endif + +#if defined(CONFIG_LUF_DEBUG ) +#define NR_LUFD_PAGES 512 +#endif #endif /* _LINUX_MM_TYPES_TASK_H */
include/linux/sched.h +5 -0
@@ -1406,6 +1406,11 @@ struct task_struct {unsigned long luf_ugen;unsigned long zone_ugen;unsigned long wait_zone_ugen; +#if defined(CONFIG_LUF_DEBUG ) +struct page *lufd_pages[NR_LUFD_PAGES ]; +int lufd_pages_order[NR_LUFD_PAGES ]; +int lufd_pages_nr; +#endif #endif struct tlbflush_unmap_batch tlb_ubc;
mm/highmem.c +1 -0
@@ -576,6 +576,7 @@ void *__kmap_local_page_prot(struct page *page, pgprot_t prot) {void *kmap; +lufd_check_pages (page,0 );/* * To broaden the usage of the actual kmap_local() machinery always map * pages when debugging is enabled and the architecture has no problems
mm/memory.c +12 -0
@@ -6224,6 +6224,18 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, mapping = vma->vm_file->f_mapping; } +#ifdef CONFIG_LUF_DEBUG +if (luf_flush) { +/* +* If it has a VM_SHARED mapping, all the mms involved +* in the struct address_space should be luf_flush'ed. +*/ +if (mapping) +luf_flush_mapping (mapping); +luf_flush_mm (mm); + } +#endif +if (unlikely (is_vm_hugetlb_page (vma))) ret =hugetlb_fault (vma->vm_mm, vma, address, flags);else
mm/page_alloc.c +30 -4
@@ -758,6 +758,8 @@ void luf_takeoff_end(struct zone *zone)VM_WARN_ON (current->zone_ugen);VM_WARN_ON (current->wait_zone_ugen); } + +lufd_check_queued_pages (); }/*
@@ -853,8 +855,10 @@ bool luf_takeoff_check_and_fold(struct zone *zone, struct page *page)struct luf_batch *lb;unsigned long lb_ugen; -if (!luf_key) +if (!luf_key) { +lufd_check_pages (page,buddy_order (page));return true; + } lb = &luf_batch[luf_key];read_lock_irqsave (&lb->lock, flags);
@@ -875,12 +879,15 @@ bool luf_takeoff_check_and_fold(struct zone *zone, struct page *page)if (!current->luf_ugen ||ugen_before (current->luf_ugen, lb_ugen)) current->luf_ugen = lb_ugen; +lufd_queue_page_for_check (page,buddy_order (page));return true; } zone_ugen =page_zone_ugen (zone, page); -if (!zone_ugen) +if (!zone_ugen) { +lufd_check_pages (page,buddy_order (page));return true; + }/* * Should not be zero since zone-zone_ugen has been updated in
@@ -888,17 +895,23 @@ bool luf_takeoff_check_and_fold(struct zone *zone, struct page *page)*/ VM_WARN_ON (!zone->zone_ugen); -if (!ugen_before (READ_ONCE (zone->zone_ugen_done), zone_ugen)) +if (!ugen_before (READ_ONCE (zone->zone_ugen_done), zone_ugen)) { +lufd_check_pages (page,buddy_order (page));return true; + }if (current->luf_no_shootdown)return false; +lufd_check_zone_pages (zone, page,buddy_order (page)); +/* * zone batched flush has been already set. */ -if (current->zone_ugen) +if (current->zone_ugen) { +lufd_queue_page_for_check (page,buddy_order (page));return true; + }/* * Others are already performing tlb shootdown for us. All we
@@ -933,6 +946,7 @@ bool luf_takeoff_check_and_fold(struct zone *zone, struct page *page)atomic_long_set (&zone->nr_luf_pages,0 ); fold_batch(tlb_ubc_takeoff, &zone->zone_batch, true); } +lufd_queue_page_for_check (page,buddy_order (page));return true; }#endif
@@ -1238,6 +1252,11 @@ static inline void __free_one_page(struct page *page, }else zone_ugen =page_zone_ugen (zone, page); +if (!zone_ugen) +lufd_check_pages (page, order); +else +lufd_check_zone_pages (zone, page, order); +while (order <MAX_PAGE_ORDER ) {int buddy_mt = migratetype;unsigned long buddy_zone_ugen;
@@ -1299,6 +1318,10 @@ static inline void __free_one_page(struct page *page,set_page_zone_ugen (page, zone_ugen); pfn = combined_pfn; order++; +if (!zone_ugen) +lufd_check_pages (page, order); +else +lufd_check_zone_pages (zone, page, order); } done_merging:
@@ -3168,6 +3191,8 @@ void free_frozen_pages(struct page *page, unsigned int order,unsigned long pfn =page_to_pfn (page);int migratetype; +lufd_mark_pages (page, order, luf_key); +if (!pcp_allowed_order (order)) {__free_pages_ok (page, order,FPI_NONE , luf_key);return ;
@@ -3220,6 +3245,7 @@ void free_unref_folios(struct folio_batch *folios, unsigned short luf_key)unsigned long pfn =folio_pfn (folio);unsigned int order =folio_order (folio); +lufd_mark_folio (folio, luf_key);if (!free_pages_prepare (&folio->page, order))continue ;/*
mm/page_ext.c +3 -0
@@ -89,6 +89,9 @@ static struct page_ext_operations *page_ext_ops[] __initdata = {#ifdef CONFIG_PAGE_TABLE_CHECK &page_table_check_ops,#endif +#ifdef CONFIG_LUF_DEBUG + &luf_debug_ops, +#endif };unsigned long page_ext_size;
mm/rmap.c +229 -0
235 lines not on this page · raw message
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC v12 rebased on mm-unstable as of Feb 21, 2025
In reply to Byungchul ParkQuoted 25 lines · Byungchul Park
Quoted 6 lines · Vlastimil Babka
Quoted 5 lines · Dave Hansen
Quoted 3 lines · Byungchul Park
I'm posting the latest version so that anyone can try luf mechanism if wanted by any chance. However, I tagged RFC again because there are still issues that should be resolved to merge to mainline:
Quoted 6 lines
Quoted 4 lines
I don't see anything fundamentally different here from the last 11 versions. I think the entire approach is dangerous and basically makes things impossible to debug. It's not clear that some of the failure scenarios that I've brought up in the past have actually been fixed.
Yes, and it's still an invasive change to the buddy allocator.
Didn't want.. but admit.
Quoted 3 lines
IIRC at Plumbers the opinion in the audience was that there might be ways to improve the batching on unmap to reduce the flushes without such an invasive and potentially dangerous change? Has that been investigated?
Sure. I tried like, by holding those pages not freed until either no one accesses the interesting pages or memory pressure is high. However, unfortunately it was super hard to fix performance degradation by the number of page reclaim increased due to the unfreed pages.
Quoted 4 lines
Quoted 2 lines
Also "Rebase on akpm/mm.git mm-unstable(5a7056135b) as of Nov 22, 2024." is very outdated at this point?
Sorry for that. I will rebase and share.
This is the same patch set but rebased on akpm/mm.git mm-unstable(f7ed46277aa) as of Feb 21, 2025.
Byungchul
Quoted 22 lines
Byungchul
Quoted 21 lines
Thanks,
VlastimilQuoted 17 lines
What I've said here still stands:
Quoted 9 lines
https://lore.kernel.org/all/[email protected]/
I think tglx would call all of this "tinkering". The approach to this series is to "fix" narrow, specific cases that reviewers point out, make it compile, then send it out again, hoping someone will apply it.
So, for me, until the approach to this series changes: NAK, for x86. Andrew, please don't take this series. Or, if you do, please drop the patch enabling it on x86.
I think I'd also like to stop being cc'd on this. If LUF is merged into mainline and proven to work on arm64 or riscv for a year, I'd be happy to take another look at enabling it on x86. I think that's just about the only thing that would make me reconsider.
Byungchul Park
reviewerRFC PATCH v12 1/25 x86/tlb: add APIs manipulating tlb batch's arch data
A new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read-only and were unmapped, since the contents of the folios wouldn't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.
This is a preparation for the mechanism that needs to recognize read-only tlb entries by separating tlb batch arch data into two, one is for read-only entries and the other is for writable ones, and merging those two when needed.
It also optimizes tlb shootdown by skipping CPUs that have already performed tlb flush needed since. To support it, added APIs manipulating arch data for x86.
--- arch/x86/include/asm/tlbflush.h | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
arch/x86/include/asm/tlbflush.h +24 -0
@@ -5,6 +5,7 @@#include <linux/mm_types.h> #include <linux/mmu_notifier.h> #include <linux/sched.h> +#include <linux/cpumask.h> #include <asm/processor.h> #include <asm/cpufeature.h>
@@ -293,6 +294,29 @@ static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm)extern void arch_tlbbatch_flush (struct arch_tlbflush_unmap_batch *batch); +static inline void arch_tlbbatch_clear (struct arch_tlbflush_unmap_batch *batch) +{ +cpumask_clear (&batch->cpumask); +} + +static inline void arch_tlbbatch_fold (struct arch_tlbflush_unmap_batch *bdst, +struct arch_tlbflush_unmap_batch *bsrc) +{ +cpumask_or (&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + +static inline bool arch_tlbbatch_need_fold (struct arch_tlbflush_unmap_batch *batch, +struct mm_struct *mm) +{ +return !cpumask_subset (mm_cpumask (mm), &batch->cpumask); +} + +static inline bool arch_tlbbatch_done (struct arch_tlbflush_unmap_batch *bdst, +struct arch_tlbflush_unmap_batch *bsrc) +{ +return !cpumask_andnot (&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} +static inline bool pte_flags_need_flush (unsigned long oldflags,unsigned long newflags,bool ignore_access)
base-commit: f7ed46277aaa8f848f18959ff68469f5186ba87c
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 2/25 arm64/tlbflush: add APIs manipulating tlb batch's arch data
A new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read only and were unmapped, since the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.
This is a preparation for the mechanism that requires to manipulate tlb batch's arch data. Even though arm64 does nothing for tlb things, arch with CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH should provide the APIs.
--- arch/arm64/include/asm/tlbflush.h | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
arch/arm64/include/asm/tlbflush.h +27 -0
@@ -347,6 +347,33 @@ static inline void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)dsb (ish); } +static inline void arch_tlbbatch_clear (struct arch_tlbflush_unmap_batch *batch) +{ +/* nothing to do */ +} + +static inline void arch_tlbbatch_fold (struct arch_tlbflush_unmap_batch *bdst, +struct arch_tlbflush_unmap_batch *bsrc) +{ +/* nothing to do */ +} + +static inline bool arch_tlbbatch_need_fold (struct arch_tlbflush_unmap_batch *batch, +struct mm_struct *mm) +{ +/* +* Nothing is needed in this architecture. +*/ +return false; +} + +static inline bool arch_tlbbatch_done (struct arch_tlbflush_unmap_batch *bdst, +struct arch_tlbflush_unmap_batch *bsrc) +{ +/* Kernel can consider tlb batch always has been done. */ +return true; +} +/* * This is meant to avoid soft lock-ups on large TLB flushing ranges and not * necessarily a performance improvement.
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 3/25 riscv/tlb: add APIs manipulating tlb batch's arch data
In reply to Byungchul ParkA new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read only and were unmapped, since the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.
This is a preparation for the mechanism that needs to recognize read-only tlb entries by separating tlb batch arch data into two, one is for read-only entries and the other is for writable ones, and merging those two when needed.
It also optimizes tlb shootdown by skipping CPUs that have already performed tlb flush needed since. To support it, added APIs manipulating arch data for riscv.
--- arch/riscv/include/asm/tlbflush.h | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)
arch/riscv/include/asm/tlbflush.h +28 -0
@@ -8,6 +8,7 @@#define _ASM_RISCV_TLBFLUSH_H#include <linux/mm_types.h> +#include <linux/cpumask.h> #include <asm/smp.h> #include <asm/errata_list.h>
@@ -64,6 +65,33 @@ void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch,void arch_flush_tlb_batched_pending (struct mm_struct *mm);void arch_tlbbatch_flush (struct arch_tlbflush_unmap_batch *batch); +static inline void arch_tlbbatch_clear (struct arch_tlbflush_unmap_batch *batch) +{ +cpumask_clear (&batch->cpumask); + +} + +static inline void arch_tlbbatch_fold (struct arch_tlbflush_unmap_batch *bdst, +struct arch_tlbflush_unmap_batch *bsrc) +{ +cpumask_or (&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); + +} + +static inline bool arch_tlbbatch_need_fold (struct arch_tlbflush_unmap_batch *batch, +struct mm_struct *mm) +{ +return !cpumask_subset (mm_cpumask (mm), &batch->cpumask); + +} + +static inline bool arch_tlbbatch_done (struct arch_tlbflush_unmap_batch *bdst, +struct arch_tlbflush_unmap_batch *bsrc) +{ +return !cpumask_andnot (&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); + +} +extern unsigned long tlb_flush_all_threshold;#else /* CONFIG_MMU */ #define local_flush_tlb_all () do { } while (0)
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 4/25 x86/tlb, riscv/tlb, mm/rmap: separate arch_tlbbatch_clear() out of arch_tlbbatch_flush()
In reply to Byungchul ParkA new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read only and were unmapped, since the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.
This is a preparation for the mechanism that requires to avoid redundant tlb flush by manipulating tlb batch's arch data. To achieve that, we need to separate the part clearing the tlb batch's arch data out of arch_tlbbatch_flush().
--- arch/riscv/mm/tlbflush.c | 1 - arch/x86/mm/tlb.c | 2 -- mm/rmap.c | 1 + 3 files changed, 1 insertion(+), 3 deletions(-)
arch/riscv/mm/tlbflush.c +0 -1
@@ -200,5 +200,4 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) {__flush_tlb_range (&batch->cpumask,FLUSH_TLB_NO_ASID ,0 ,FLUSH_TLB_MAX_SIZE ,PAGE_SIZE ); -cpumask_clear (&batch->cpumask); }
arch/x86/mm/tlb.c +0 -2
@@ -1292,8 +1292,6 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)local_irq_enable (); } -cpumask_clear (&batch->cpumask); -put_flush_tlb_info ();put_cpu (); }
mm/rmap.c +1 -0
@@ -648,6 +648,7 @@ void try_to_unmap_flush(void)return ;arch_tlbbatch_flush (&tlb_ubc->arch); +arch_tlbbatch_clear (&tlb_ubc->arch); tlb_ubc->flush_required = false; tlb_ubc->writable = false; }
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 5/25 mm/buddy: make room for a new variable, luf_key, in struct page
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that tracks need of tlb flush for each page residing in buddy.
Since the private field in struct page is used only to store page order in buddy, ranging from 0 to MAX_PAGE_ORDER, that can be covered with unsigned short. So splitted it into two smaller ones, order and luf_key, so that the both can be used in buddy at the same time.
--- include/linux/mm_types.h | 42 +++++++++++++++++++++++++++++++++------- mm/internal.h | 4 ++-- mm/page_alloc.c | 2 +- 3 files changed, 38 insertions(+), 10 deletions(-)
include/linux/mm_types.h +35 -7
@@ -107,13 +107,27 @@ struct page {pgoff_t index;/* Our offset within mapping. */ unsigned long share;/* share count for fsdax */ }; -/** -* @private: Mapping-private opaque data. -* Usually used for buffer_heads if PagePrivate. -* Used for swp_entry_t if swapcache flag set. -* Indicates order in the buddy system if PageBuddy. -*/ -unsigned long private; +union { +/** +* @private: Mapping-private opaque data. +* Usually used for buffer_heads if PagePrivate. +* Used for swp_entry_t if swapcache flag set. +* Indicates order in the buddy system if PageBuddy. +*/ +unsigned long private; +struct { +/* +* Indicates order in the buddy system if PageBuddy. +*/ +unsigned short order; + +/* +* For tracking need of tlb flush, +* by luf(lazy unmap flush). +*/ +unsigned short luf_key; + }; + }; };struct {/* page_pool used by netstack */ /**
@@ -577,6 +591,20 @@ static inline void set_page_private(struct page *page, unsigned long private) page->private = private; } +#define page_buddy_order (page) ((page)->order) + +static inline void set_page_buddy_order (struct page *page,unsigned int order) +{ + page->order = (unsigned short )order; +} + +#define page_luf_key (page) ((page)->luf_key) + +static inline void set_page_luf_key (struct page *page,unsigned short luf_key) +{ + page->luf_key = luf_key; +} +static inline void *folio_get_private (struct folio *folio) {return folio->private;
mm/internal.h +2 -2
@@ -543,7 +543,7 @@ struct alloc_context {static inline unsigned int buddy_order (struct page *page) {/* PageBuddy() must be checked by the caller */ -return page_private (page); +return page_buddy_order (page); }/*
@@ -557,7 +557,7 @@ static inline unsigned int buddy_order(struct page *page)* times, potentially observing different values in the tests and the actual * use of the result. */ -#define buddy_order_unsafe (page) READ_ONCE(page_private(page)) +#define buddy_order_unsafe (page) READ_ONCE(page_buddy_order(page))/* * This function checks whether a page is free && is the buddy
mm/page_alloc.c +1 -1
@@ -576,7 +576,7 @@ void prep_compound_page(struct page *page, unsigned int order)static inline void set_buddy_order (struct page *page,unsigned int order) { -set_page_private (page, order); +set_page_buddy_order (page, order);__SetPageBuddy (page); }
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 6/25 mm: move should_skip_kasan_poison() to mm/internal.h
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that needs to use should_skip_kasan_poison() function in mm/internal.h.
--- mm/internal.h | 47 +++++++++++++++++++++++++++++++++++++++++++++++ mm/page_alloc.c | 47 ----------------------------------------------- 2 files changed, 47 insertions(+), 47 deletions(-)
mm/internal.h +47 -0
@@ -1067,8 +1067,55 @@ static inline void vunmap_range_noflush(unsigned long start, unsigned long end)DECLARE_STATIC_KEY_TRUE (deferred_pages);bool __init deferred_grow_zone (struct zone *zone,unsigned int order); + +static inline bool deferred_pages_enabled (void ) +{ +return static_branch_unlikely (&deferred_pages); +} +#else +static inline bool deferred_pages_enabled (void ) +{ +return false; +}#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ +/* +* Skip KASAN memory poisoning when either: +* +* 1. For generic KASAN: deferred memory initialization has not yet completed. +* Tag-based KASAN modes skip pages freed via deferred memory initialization +* using page tags instead (see below). +* 2. For tag-based KASAN modes: the page has a match-all KASAN tag, indicating +* that error detection is disabled for accesses via the page address. +* +* Pages will have match-all tags in the following circumstances: +* +* 1. Pages are being initialized for the first time, including during deferred +* memory init; see the call to page_kasan_tag_reset in __init_single_page. +* 2. The allocation was not unpoisoned due to __GFP_SKIP_KASAN, with the +* exception of pages unpoisoned by kasan_unpoison_vmalloc. +* 3. The allocation was excluded from being checked due to sampling, +* see the call to kasan_unpoison_pages. +* +* Poisoning pages during deferred memory init will greatly lengthen the +* process and cause problem in large memory systems as the deferred pages +* initialization is done with interrupt disabled. +* +* Assuming that there will be no reference to those newly initialized +* pages before they are ever allocated, this should have no effect on +* KASAN memory tracking as the poison will be properly inserted at page +* allocation time. The only corner case is when pages are allocated by +* on-demand allocation and then freed again before the deferred pages +* initialization is done, but this is not likely to happen. +*/ +static inline bool should_skip_kasan_poison (struct page *page) +{ +if (IS_ENABLED (CONFIG_KASAN_GENERIC )) +return deferred_pages_enabled (); + +return page_kasan_tag (page) ==KASAN_TAG_KERNEL ; +} +enum mminit_level {MMINIT_WARNING ,MMINIT_VERIFY ,
mm/page_alloc.c +0 -47
65 lines not on this page · raw message
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 7/25 mm: introduce luf_ugen to be used as a global timestamp
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that needs to evaluate the temporal sequence of events to determine whether tlb flush required has been done on each CPU.
To achieve that, this patch introduced a generation number, luf_ugen, and a few APIs manipulating the number. It's worth noting the number is designed to wraparound so care must be taken when using it.
--- include/linux/mm.h | 34 ++++++++++++++++++++++++++++++++++ mm/rmap.c | 22 ++++++++++++++++++++++ 2 files changed, 56 insertions(+)
include/linux/mm.h +34 -0
@@ -4240,4 +4240,38 @@ int arch_get_shadow_stack_status(struct task_struct *t, unsigned long __user *stint arch_set_shadow_stack_status (struct task_struct *t,unsigned long status);int arch_lock_shadow_stack_status (struct task_struct *t,unsigned long status); +#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH ) +/* +* luf_ugen will start with 2 so that 1 can be regarded as a passed one. +*/ +#define LUF_UGEN_INIT 2 + +static inline bool ugen_before (unsigned long a,unsigned long b) +{ +/* +* Consider wraparound. +*/ +return (long )(a - b) <0 ; +} + +static inline unsigned long next_ugen (unsigned long ugen) +{ +if (ugen +1 ) +return ugen +1 ; +/* +* Avoid invalid ugen, zero. +*/ +return ugen +2 ; +} + +static inline unsigned long prev_ugen (unsigned long ugen) +{ +if (ugen -1 ) +return ugen -1 ; +/* +* Avoid invalid ugen, zero. +*/ +return ugen -2 ; +} +#endif #endif /* _LINUX_MM_H */
mm/rmap.c +22 -0
@@ -634,6 +634,28 @@ struct anon_vma *folio_lock_anon_vma_read(const struct folio *folio, }#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH + +/* +* This generation number is primarily used as a global timestamp to +* determine whether tlb flush required has been done on each CPU. The +* function, ugen_before(), should be used to evaluate the temporal +* sequence of events because the number is designed to wraparound. +*/ +static atomic_long_t __maybe_unused luf_ugen =ATOMIC_LONG_INIT (LUF_UGEN_INIT ); + +/* +* Don't return invalid luf_ugen, zero. +*/ +static unsigned long __maybe_unused new_luf_ugen (void ) +{ +unsigned long ugen =atomic_long_inc_return (&luf_ugen); + +if (!ugen) + ugen =atomic_long_inc_return (&luf_ugen); + +return ugen; +} +/* * Flush TLB entries for recently unmapped pages from remote CPUs. It is * important if a PTE was dirty when it was unmapped that it's flushed
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 8/25 mm: introduce luf_batch to be used as hash table to store luf meta data
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that needs to keep luf meta data per page while staying in pcp or buddy allocator. The meta data includes cpumask for tlb shootdown and luf's request generation number.
Since struct page doesn't have enough room to store luf meta data, this patch introduces a hash table to store them and makes each page keep its hash key instead.
Since all the pages in pcp or buddy share the hash table, confliction is inevitable so care must be taken when reading or updating its entry.
--- include/linux/mm_types.h | 10 ++++ mm/internal.h | 8 +++ mm/rmap.c | 122 +++++++++++++++++++++++++++++++++++++-- 3 files changed, 136 insertions(+), 4 deletions(-)
include/linux/mm_types.h +10 -0
@@ -33,6 +33,16 @@struct address_space ;struct mem_cgroup ; +#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH +struct luf_batch { +struct tlbflush_unmap_batch batch; +unsigned long ugen; +rwlock_t lock; +}; +#else +struct luf_batch {}; +#endif +/* * Each physical page in the system has a struct page associated with * it to keep track of whatever it is we are using the page for at the
mm/internal.h +8 -0
@@ -1270,6 +1270,8 @@ extern struct workqueue_struct *mm_percpu_wq;void try_to_unmap_flush (void );void try_to_unmap_flush_dirty (void );void flush_tlb_batched_pending (struct mm_struct *mm); +void fold_batch (struct tlbflush_unmap_batch *dst,struct tlbflush_unmap_batch *src,bool reset); +void fold_luf_batch (struct luf_batch *dst,struct luf_batch *src);#else static inline void try_to_unmap_flush (void ) {
@@ -1280,6 +1282,12 @@ static inline void try_to_unmap_flush_dirty(void)static inline void flush_tlb_batched_pending (struct mm_struct *mm) { } +static inline void fold_batch (struct tlbflush_unmap_batch *dst,struct tlbflush_unmap_batch *src,bool reset) +{ +} +static inline void fold_luf_batch (struct luf_batch *dst,struct luf_batch *src) +{ +}#endif /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ extern const struct trace_print_flags pageflag_names[];
mm/rmap.c +118 -4
140 lines not on this page · raw message
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 9/25 mm: introduce API to perform tlb shootdown on exit from page allocator
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that performs tlb shootdown required on exit from page allocator.
This patch introduced a new API rather than making use of existing try_to_unmap_flush() to avoid repeated and redundant tlb shootdown due to frequent page allocations during a session of batched unmap flush.
--- include/linux/sched.h | 1 + mm/internal.h | 4 ++++ mm/rmap.c | 20 ++++++++++++++++++++ 3 files changed, 25 insertions(+)
include/linux/sched.h +1 -0
@@ -1401,6 +1401,7 @@ struct task_struct {#endif struct tlbflush_unmap_batch tlb_ubc; +struct tlbflush_unmap_batch tlb_ubc_takeoff;/* Cache last used pipe for splice(): */ struct pipe_inode_info *splice_pipe;
mm/internal.h +4 -0
@@ -1269,6 +1269,7 @@ extern struct workqueue_struct *mm_percpu_wq;#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH void try_to_unmap_flush (void );void try_to_unmap_flush_dirty (void ); +void try_to_unmap_flush_takeoff (void );void flush_tlb_batched_pending (struct mm_struct *mm);void fold_batch (struct tlbflush_unmap_batch *dst,struct tlbflush_unmap_batch *src,bool reset);void fold_luf_batch (struct luf_batch *dst,struct luf_batch *src);
@@ -1279,6 +1280,9 @@ static inline void try_to_unmap_flush(void)static inline void try_to_unmap_flush_dirty (void ) { } +static inline void try_to_unmap_flush_takeoff (void ) +{ +}static inline void flush_tlb_batched_pending (struct mm_struct *mm) { }
mm/rmap.c +20 -0
@@ -772,6 +772,26 @@ void fold_luf_batch(struct luf_batch *dst, struct luf_batch *src)read_unlock_irqrestore (&src->lock, flags); } +void try_to_unmap_flush_takeoff (void ) +{ +struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; +struct tlbflush_unmap_batch *tlb_ubc_takeoff = ¤t->tlb_ubc_takeoff; + +if (!tlb_ubc_takeoff->flush_required) +return ; + +arch_tlbbatch_flush (&tlb_ubc_takeoff->arch); + +/* +* Now that tlb shootdown of tlb_ubc_takeoff has been performed, +* it's good chance to shrink tlb_ubc if possible. +*/ +if (arch_tlbbatch_done (&tlb_ubc->arch, &tlb_ubc_takeoff->arch)) +reset_batch (tlb_ubc); + +reset_batch (tlb_ubc_takeoff); +} +/* * Flush TLB entries for recently unmapped pages from remote CPUs. It is * important if a PTE was dirty when it was unmapped that it's flushed
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 10/25 mm: introduce APIs to check if the page allocation is tlb shootdownable
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that should indentify if tlb shootdown can be performed on page allocation.
In a context with irq disabled or non-task, tlb shootdown cannot be performed because of deadlock issue. Thus, page allocator should work being aware of whether tlb shootdown can be performed on returning page.
This patch introduced APIs that pcp or buddy page allocator can use to delimit the critical sections taking off pages and indentify whether tlb shootdown can be performed.
--- include/linux/sched.h | 5 ++ mm/internal.h | 14 ++++ mm/page_alloc.c | 159 ++++++++++++++++++++++++++++++++++++++++++ mm/rmap.c | 2 +- 4 files changed, 179 insertions(+), 1 deletion(-)
include/linux/sched.h +5 -0
@@ -1400,6 +1400,11 @@ struct task_struct {struct callback_head cid_work;#endif +#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH ) +int luf_no_shootdown; +int luf_takeoff_started; +#endif +struct tlbflush_unmap_batch tlb_ubc;struct tlbflush_unmap_batch tlb_ubc_takeoff;
mm/internal.h +14 -0
@@ -1598,6 +1598,20 @@ static inline void accept_page(struct page *page) { }#endif /* CONFIG_UNACCEPTED_MEMORY */ +#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH ) +extern struct luf_batch luf_batch[]; +bool luf_takeoff_start (void ); +void luf_takeoff_end (void ); +bool luf_takeoff_no_shootdown (void ); +bool luf_takeoff_check (struct page *page); +bool luf_takeoff_check_and_fold (struct page *page); +#else +static inline bool luf_takeoff_start (void ) {return false; } +static inline void luf_takeoff_end (void ) {} +static inline bool luf_takeoff_no_shootdown (void ) {return true; } +static inline bool luf_takeoff_check (struct page *page) {return true; } +static inline bool luf_takeoff_check_and_fold (struct page *page) {return true; } +#endif /* pagewalk.c */ int walk_page_range_mm(struct mm_struct *mm, unsigned long start,
mm/page_alloc.c +159 -0
165 lines not on this page · raw message
mm/rmap.c +1 -1
@@ -693,7 +693,7 @@ void fold_batch(struct tlbflush_unmap_batch *dst,/* * Use 0th entry as accumulated batch. */ -static struct luf_batch luf_batch[NR_LUF_BATCH ]; +struct luf_batch luf_batch[NR_LUF_BATCH ];static void luf_batch_init (struct luf_batch *lb) {
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 11/25 mm: deliver luf_key to pcp or buddy on free after unmapping
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that needs to pass luf_key to pcp or buddy allocator on free after unmapping e.g. during page reclaim or page migration.
The luf_key will be used to track need of tlb shootdown and which cpus need to perform tlb flush, per page residing in pcp or buddy, and should be handed over properly when pages travel between pcp and buddy.
--- mm/internal.h | 4 +- mm/page_alloc.c | 116 ++++++++++++++++++++++++++++++++----------- mm/page_frag_cache.c | 6 +-- mm/page_isolation.c | 6 +++ mm/page_reporting.c | 6 +++ mm/slub.c | 2 +- mm/swap.c | 4 +- mm/vmscan.c | 8 +-- 8 files changed, 111 insertions(+), 41 deletions(-)
mm/internal.h +2 -2
@@ -746,8 +746,8 @@ struct page *__alloc_frozen_pages_noprof(gfp_t, unsigned int order, int nid, nodemask_t *);#define __alloc_frozen_pages (...) \ alloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__)) -void free_frozen_pages (struct page *page,unsigned int order); -void free_unref_folios (struct folio_batch *fbatch); +void free_frozen_pages (struct page *page,unsigned int order,unsigned short luf_key); +void free_unref_folios (struct folio_batch *fbatch,unsigned short luf_key);#ifdef CONFIG_NUMA struct page *alloc_frozen_pages_noprof (gfp_t ,unsigned int order);
mm/page_alloc.c +87 -29
319 lines not on this page · raw message
mm/page_frag_cache.c +3 -3
@@ -86,7 +86,7 @@ void __page_frag_cache_drain(struct page *page, unsigned int count)VM_BUG_ON_PAGE (page_ref_count (page) ==0 , page);if (page_ref_sub_and_test (page, count)) -free_frozen_pages (page,compound_order (page)); +free_frozen_pages (page,compound_order (page),0 ); }EXPORT_SYMBOL (__page_frag_cache_drain);
@@ -139,7 +139,7 @@ void *__page_frag_alloc_align(struct page_frag_cache *nc,if (unlikely (encoded_page_decode_pfmemalloc (encoded_page))) {free_frozen_pages (page, -encoded_page_decode_order (encoded_page)); +encoded_page_decode_order (encoded_page),0 ); goto refill; }
@@ -166,6 +166,6 @@ void page_frag_free(void *addr)struct page *page =virt_to_head_page (addr);if (unlikely (put_page_testzero (page))) -free_frozen_pages (page,compound_order (page)); +free_frozen_pages (page,compound_order (page),0 ); }EXPORT_SYMBOL (page_frag_free);
mm/page_isolation.c +6 -0
@@ -265,6 +265,12 @@ static void unset_migratetype_isolate(struct page *page, int migratetype)WARN_ON_ONCE (!move_freepages_block_isolate (zone, page, migratetype)); }else {set_pageblock_migratetype (page, migratetype); + +/* +* Do not clear the page's private to keep its luf_key +* unchanged. +*/ +__putback_isolated_page (page, order, migratetype); } zone->nr_isolate_pageblock--;
mm/page_reporting.c +6 -0
@@ -116,6 +116,12 @@ page_reporting_drain(struct page_reporting_dev_info *prdev,int mt =get_pageblock_migratetype (page);unsigned int order =get_order (sg->length); +/* +* Ensure private is zero before putting into the +* allocator. +*/ +set_page_private (page,0 ); +__putback_isolated_page (page, order, mt);/* If the pages were not reported due to error skip flagging */
mm/slub.c +1 -1
@@ -2665,7 +2665,7 @@ static void __free_slab(struct kmem_cache *s, struct slab *slab)__folio_clear_slab (folio);mm_account_reclaimed_pages (pages);unaccount_slab (slab, order, s); -free_frozen_pages (&folio->page, order); +free_frozen_pages (&folio->page, order,0 ); }static void rcu_free_slab (struct rcu_head *h)
mm/swap.c +2 -2
@@ -109,7 +109,7 @@ void __folio_put(struct folio *folio)page_cache_release (folio);folio_unqueue_deferred_split (folio);mem_cgroup_uncharge (folio); -free_frozen_pages (&folio->page,folio_order (folio)); +free_frozen_pages (&folio->page,folio_order (folio),0 ); }EXPORT_SYMBOL (__folio_put);
@@ -989,7 +989,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs) folios->nr = j;mem_cgroup_uncharge_folios (folios); -free_unref_folios (folios); +free_unref_folios (folios,0 ); }EXPORT_SYMBOL (folios_put_refs);
mm/vmscan.c +4 -4
@@ -1525,7 +1525,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (folio_batch_add (&free_folios, folio) ==0 ) {mem_cgroup_uncharge_folios (&free_folios);try_to_unmap_flush (); -free_unref_folios (&free_folios); +free_unref_folios (&free_folios,0 ); }continue ;
@@ -1594,7 +1594,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,mem_cgroup_uncharge_folios (&free_folios);try_to_unmap_flush (); -free_unref_folios (&free_folios); +free_unref_folios (&free_folios,0 );list_splice (&ret_folios, folio_list);count_vm_events (PGACTIVATE , pgactivate);
@@ -1918,7 +1918,7 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec,if (folio_batch_add (&free_folios, folio) ==0 ) {spin_unlock_irq (&lruvec->lru_lock);mem_cgroup_uncharge_folios (&free_folios); -free_unref_folios (&free_folios); +free_unref_folios (&free_folios,0 );spin_lock_irq (&lruvec->lru_lock); }
@@ -1940,7 +1940,7 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec,if (free_folios.nr) {spin_unlock_irq (&lruvec->lru_lock);mem_cgroup_uncharge_folios (&free_folios); -free_unref_folios (&free_folios); +free_unref_folios (&free_folios,0 );spin_lock_irq (&lruvec->lru_lock); }
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 12/25 mm: delimit critical sections to take off pages from pcp or buddy alloctor
In reply to Byungchul ParkNow that luf mechanism has been introduced, tlb shootdown might be necessary when luf'd pages exit from pcp or buddy allocator. Check if it's okay to take off pages and can perform for luf'd pages before use.
--- mm/compaction.c | 32 ++++++++++++++++-- mm/internal.h | 2 +- mm/page_alloc.c | 79 +++++++++++++++++++++++++++++++++++++++++++-- mm/page_isolation.c | 4 ++- mm/page_reporting.c | 20 +++++++++++- 5 files changed, 129 insertions(+), 8 deletions(-)
mm/compaction.c +29 -3
@@ -606,6 +606,7 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, page =pfn_to_page (blockpfn); +luf_takeoff_start ();/* Isolate free pages. */ for (; blockpfn < end_pfn; blockpfn += stride, page += stride) {int isolated;
@@ -654,9 +655,12 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, goto isolate_fail; } +if (!luf_takeoff_check (page)) + goto isolate_fail; +/* Found a free page, will break it into order-0 pages */ order =buddy_order (page); - isolated =__isolate_free_page (page, order); + isolated =__isolate_free_page (page, order, false);if (!isolated)break ;set_page_private (page, order);
@@ -684,6 +688,11 @@ static unsigned long isolate_freepages_block(struct compact_control *cc,if (locked)spin_unlock_irqrestore (&cc->zone->lock, flags); +/* +* Check and flush before using the pages taken off. +*/ +luf_takeoff_end (); +/* * Be careful to not go outside of the pageblock. */
@@ -1591,6 +1600,7 @@ static void fast_isolate_freepages(struct compact_control *cc)if (!area->nr_free)continue ; +luf_takeoff_start ();spin_lock_irqsave (&cc->zone->lock, flags); freelist = &area->free_list[MIGRATE_MOVABLE ];list_for_each_entry_reverse (freepage, freelist, buddy_list) {
@@ -1598,6 +1608,10 @@ static void fast_isolate_freepages(struct compact_control *cc) order_scanned++; nr_scanned++; + +if (!luf_takeoff_check (freepage)) + goto scan_next; + pfn =page_to_pfn (freepage);if (pfn >= highest)
@@ -1617,7 +1631,7 @@ static void fast_isolate_freepages(struct compact_control *cc)/* Shorten the scan if a candidate is found */ limit >>=1 ; } - +scan_next:if (order_scanned >= limit)break ; }
@@ -1635,7 +1649,7 @@ static void fast_isolate_freepages(struct compact_control *cc)/* Isolate the page if available */ if (page) { -if (__isolate_free_page (page, order)) { +if (__isolate_free_page (page, order, false)) {set_page_private (page, order); nr_isolated =1 << order; nr_scanned += nr_isolated -1 ;
@@ -1652,6 +1666,11 @@ static void fast_isolate_freepages(struct compact_control *cc)spin_unlock_irqrestore (&cc->zone->lock, flags); +/* +* Check and flush before using the pages taken off. +*/ +luf_takeoff_end (); +/* Skip fast search if enough freepages isolated */ if (cc->nr_freepages >= cc->nr_migratepages)break ;
@@ -2373,7 +2392,14 @@ static enum compact_result compact_finished(struct compact_control *cc) {int ret; +/* +* luf_takeoff_{start,end}() is required to identify whether +* this compaction context is tlb shootdownable for luf'd pages. +*/ +luf_takeoff_start (); ret =__compact_finished (cc); +luf_takeoff_end (); +trace_mm_compaction_finished (cc->zone, cc->order, ret);if (ret ==COMPACT_NO_SUITABLE_PAGE ) ret =COMPACT_CONTINUE ;
mm/internal.h +1 -1
@@ -666,7 +666,7 @@ static inline void clear_zone_contiguous(struct zone *zone) zone->contiguous = false; } -extern int __isolate_free_page (struct page *page,unsigned int order); +extern int __isolate_free_page (struct page *page,unsigned int order,bool willputback);extern void __putback_isolated_page (struct page *page,unsigned int order,int mt); extern void memblock_free_pages(struct page *page, unsigned long pfn,
mm/page_alloc.c +77 -2
246 lines not on this page · raw message
mm/page_isolation.c +3 -1
@@ -218,6 +218,7 @@ static void unset_migratetype_isolate(struct page *page, int migratetype)struct page *buddy; zone =page_zone (page); +luf_takeoff_start ();spin_lock_irqsave (&zone->lock, flags);if (!is_migrate_isolate_page (page)) goto out;
@@ -236,7 +237,7 @@ static void unset_migratetype_isolate(struct page *page, int migratetype) buddy =find_buddy_page_pfn (page,page_to_pfn (page), order,NULL );if (buddy && !is_migrate_isolate_page (buddy)) { - isolated_page = !!__isolate_free_page (page, order); + isolated_page = !!__isolate_free_page (page, order, true);/* * Isolating a free page in an isolated pageblock * is expected to always work as watermarks don't
@@ -276,6 +277,7 @@ static void unset_migratetype_isolate(struct page *page, int migratetype) zone->nr_isolate_pageblock--; out:spin_unlock_irqrestore (&zone->lock, flags); +luf_takeoff_end (zone); } static inline struct page *
mm/page_reporting.c +19 -1
@@ -167,6 +167,7 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,if (list_empty (list))return err; +luf_takeoff_start ();spin_lock_irq (&zone->lock);/*
@@ -191,6 +192,11 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,if (PageReported (page))continue ; +if (!luf_takeoff_check (page)) { +VM_WARN_ON (1 ); +continue ; + } +/* * If we fully consumed our budget then update our * state to indicate that we are requesting additional
@@ -204,7 +210,7 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,/* Attempt to pull page from list and place in scatterlist */ if (*offset) { -if (!__isolate_free_page (page, order)) { +if (!__isolate_free_page (page, order, false)) { next = page;break ; }
@@ -227,6 +233,11 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,/* release lock before waiting on report processing */ spin_unlock_irq (&zone->lock); +/* +* Check and flush before using the pages taken off. +*/ +luf_takeoff_end (); +/* begin processing pages in local list */ err = prdev->report (prdev, sgl,PAGE_REPORTING_CAPACITY );
@@ -236,6 +247,8 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,/* update budget to reflect call to report function */ budget--; +luf_takeoff_start (); +/* reacquire zone lock and resume processing */ spin_lock_irq (&zone->lock);
@@ -259,6 +272,11 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,spin_unlock_irq (&zone->lock); +/* +* Check and flush before using the pages taken off. +*/ +luf_takeoff_end (); +return err; }
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 13/25 mm: introduce pend_list in struct free_area to track luf'd pages
In reply to Byungchul Parkluf'd pages requires tlb shootdown on exiting from page allocator. For some page allocation request, it's okay to return luf'd page followed by tlb shootdown but it's not okay for e.g. irq context.
This patch splitted the list in free_area into two, 'free_list' for non-luf'd pages and 'pend_list' for luf'd pages so that the buddy allocator can work better with various conditions of context.
--- include/linux/mmzone.h | 3 ++ kernel/power/snapshot.c | 14 ++++++ kernel/vmcore_info.c | 2 + mm/compaction.c | 33 ++++++++++--- mm/internal.h | 17 ++++++- mm/mm_init.c | 2 + mm/page_alloc.c | 105 ++++++++++++++++++++++++++++++++++------ mm/page_reporting.c | 22 ++++++--- mm/vmstat.c | 15 ++++++ 9 files changed, 184 insertions(+), 29 deletions(-)
include/linux/mmzone.h +3 -0
@@ -116,6 +116,7 @@ extern int page_group_by_mobility_disabled; MIGRATETYPE_MASK)struct free_area {struct list_head free_list[MIGRATE_TYPES ]; +struct list_head pend_list[MIGRATE_TYPES ];unsigned long nr_free; };
@@ -1014,6 +1015,8 @@ struct zone {/* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS ];atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS ]; +/* Count pages that need tlb shootdown on allocation */ +atomic_long_t nr_luf_pages; }____cacheline_internodealigned_in_smp ;enum pgdat_flags {
kernel/power/snapshot.c +14 -0
@@ -1285,6 +1285,20 @@ static void mark_free_pages(struct zone *zone)swsusp_set_page_free (pfn_to_page (pfn + i)); } } + +list_for_each_entry (page, + &zone->free_area[order].pend_list[t], buddy_list) { +unsigned long i; + + pfn =page_to_pfn (page); +for (i =0 ; i < (1UL << order); i++) { +if (!--page_count) { +touch_nmi_watchdog (); + page_count =WD_PAGE_COUNT ; + } +swsusp_set_page_free (pfn_to_page (pfn + i)); + } + } }spin_unlock_irqrestore (&zone->lock, flags); }
kernel/vmcore_info.c +2 -0
@@ -188,11 +188,13 @@ static int __init crash_save_vmcoreinfo_init(void)VMCOREINFO_OFFSET (zone, vm_stat);VMCOREINFO_OFFSET (zone, spanned_pages);VMCOREINFO_OFFSET (free_area, free_list); +VMCOREINFO_OFFSET (free_area, pend_list);VMCOREINFO_OFFSET (list_head, next);VMCOREINFO_OFFSET (list_head, prev);VMCOREINFO_LENGTH (zone.free_area,NR_PAGE_ORDERS );log_buf_vmcoreinfo_setup ();VMCOREINFO_LENGTH (free_area.free_list,MIGRATE_TYPES ); +VMCOREINFO_LENGTH (free_area.pend_list,MIGRATE_TYPES );VMCOREINFO_NUMBER (NR_FREE_PAGES );VMCOREINFO_NUMBER (PG_lru);VMCOREINFO_NUMBER (PG_private);
mm/compaction.c +27 -6
@@ -1592,24 +1592,28 @@ static void fast_isolate_freepages(struct compact_control *cc) order = next_search_order(cc, order)) {struct free_area *area = &cc->zone->free_area[order];struct list_head *freelist; +struct list_head *high_pfn_list;struct page *freepage;unsigned long flags;unsigned int order_scanned =0 ;unsigned long high_pfn =0 ; +bool consider_pend = false; +bool can_shootdown;if (!area->nr_free)continue ; -luf_takeoff_start (); + can_shootdown =luf_takeoff_start ();spin_lock_irqsave (&cc->zone->lock, flags); freelist = &area->free_list[MIGRATE_MOVABLE ]; +retry:list_for_each_entry_reverse (freepage, freelist, buddy_list) {unsigned long pfn; order_scanned++; nr_scanned++; -if (!luf_takeoff_check (freepage)) +if (unlikely (consider_pend && !luf_takeoff_check (freepage))) goto scan_next; pfn =page_to_pfn (freepage);
@@ -1622,26 +1626,34 @@ static void fast_isolate_freepages(struct compact_control *cc) cc->fast_search_fail =0 ; cc->search_order = order; page = freepage; -break ; + goto done; }if (pfn >= min_pfn && pfn > high_pfn) { high_pfn = pfn; + high_pfn_list = freelist;/* Shorten the scan if a candidate is found */ limit >>=1 ; } scan_next:if (order_scanned >= limit) -break ; + goto done; } +if (!consider_pend && can_shootdown) { + consider_pend = true; + freelist = &area->pend_list[MIGRATE_MOVABLE ]; + goto retry; + } +done:/* Use a maximum candidate pfn if a preferred one was not found */ if (!page && high_pfn) { page =pfn_to_page (high_pfn);/* Update freepage for the list reorder below */ freepage = page; + freelist = high_pfn_list; }/* Reorder to so a future search skips recent pages */
@@ -2040,18 +2052,20 @@ static unsigned long fast_find_migrateblock(struct compact_control *cc)struct list_head *freelist;unsigned long flags;struct page *freepage; +bool consider_pend = false;if (!area->nr_free)continue ;spin_lock_irqsave (&cc->zone->lock, flags); freelist = &area->free_list[MIGRATE_MOVABLE ]; +retry:list_for_each_entry (freepage, freelist, buddy_list) {unsigned long free_pfn;if (nr_scanned++ >= limit) {move_freelist_tail (freelist, freepage); -break ; + goto done; } free_pfn =page_to_pfn (freepage);
@@ -2074,9 +2088,16 @@ static unsigned long fast_find_migrateblock(struct compact_control *cc) pfn = cc->zone->zone_start_pfn; cc->fast_search_fail =0 ; found_block = true; -break ; + goto done; } } + +if (!consider_pend) { + consider_pend = true; + freelist = &area->pend_list[MIGRATE_MOVABLE ]; + goto retry; + } +done:spin_unlock_irqrestore (&cc->zone->lock, flags); }
mm/internal.h +16 -1
@@ -865,11 +865,16 @@ static inline void init_cma_pageblock(struct page *page)int find_suitable_fallback (struct free_area *area,unsigned int order,int migratetype,bool only_stealable,bool *can_steal); -static inline bool free_area_empty (struct free_area *area,int migratetype) +static inline bool free_list_empty (struct free_area *area,int migratetype) {return list_empty (&area->free_list[migratetype]); } +static inline bool free_area_empty (struct free_area *area,int migratetype) +{ +return list_empty (&area->free_list[migratetype]) && +list_empty (&area->pend_list[migratetype]); +}/* mm/util.c */ struct anon_vma *folio_anon_vma (const struct folio *folio);
@@ -1605,12 +1610,22 @@ void luf_takeoff_end(void);bool luf_takeoff_no_shootdown (void );bool luf_takeoff_check (struct page *page);bool luf_takeoff_check_and_fold (struct page *page); + +static inline bool non_luf_pages_ok (struct zone *zone) +{ +unsigned long nr_free =zone_page_state (zone,NR_FREE_PAGES ); +unsigned long min_wm =min_wmark_pages (zone); +unsigned long nr_luf_pages =atomic_long_read (&zone->nr_luf_pages); + +return nr_free - nr_luf_pages > min_wm; +}#else static inline bool luf_takeoff_start (void ) {return false; }static inline void luf_takeoff_end (void ) {}static inline bool luf_takeoff_no_shootdown (void ) {return true; }static inline bool luf_takeoff_check (struct page *page) {return true; }static inline bool luf_takeoff_check_and_fold (struct page *page) {return true; } +static inline bool non_luf_pages_ok (struct zone *zone) {return true; }#endif /* pagewalk.c */
mm/mm_init.c +2 -0
@@ -1421,12 +1421,14 @@ static void __meminit zone_init_free_lists(struct zone *zone)unsigned int order, t;for_each_migratetype_order (order, t) {INIT_LIST_HEAD (&zone->free_area[order].free_list[t]); +INIT_LIST_HEAD (&zone->free_area[order].pend_list[t]); zone->free_area[order].nr_free =0 ; }#ifdef CONFIG_UNACCEPTED_MEMORY INIT_LIST_HEAD (&zone->unaccepted_pages);#endif +atomic_long_set (&zone->nr_luf_pages,0 ); } void __meminit init_currently_empty_zone(struct zone *zone,
mm/page_alloc.c +90 -15
193 lines not on this page · raw message
mm/page_reporting.c +15 -7
@@ -159,15 +159,17 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,struct page *page, *next;long budget;int err =0 ; +bool consider_pend = false; +bool can_shootdown;/* * Perform early check, if free area is empty there is * nothing to process so we can skip this free_list. */ -if (list_empty (list)) +if (free_area_empty (area, mt))return err; -luf_takeoff_start (); + can_shootdown =luf_takeoff_start ();spin_lock_irq (&zone->lock);/*
@@ -185,14 +187,14 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,* should always be a power of 2. */ budget =DIV_ROUND_UP (area->nr_free,PAGE_REPORTING_CAPACITY *16 ); - +retry:/* loop through free list adding unreported pages to sg list */ list_for_each_entry_safe (page, next, list, lru) {/* We are going to skip over the reported pages. */ if (PageReported (page))continue ; -if (!luf_takeoff_check (page)) { +if (unlikely (consider_pend && !luf_takeoff_check (page))) {VM_WARN_ON (1 );continue ; }
@@ -205,14 +207,14 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,if (budget <0 ) {atomic_set (&prdev->state,PAGE_REPORTING_REQUESTED ); next = page; -break ; + goto done; }/* Attempt to pull page from list and place in scatterlist */ if (*offset) {if (!__isolate_free_page (page, order, false)) { next = page; -break ; + goto done; }/* Add page to scatter list */
@@ -263,9 +265,15 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,/* exit on error */ if (err) -break ; + goto done; } +if (!consider_pend && can_shootdown) { + consider_pend = true; + list = &area->pend_list[mt]; + goto retry; + } +done:/* Rotate any leftover pages to the head of the freelist */ if (!list_entry_is_head (next, list, lru) && !list_is_first (&next->lru, list))list_rotate_to_front (&next->lru, list);
mm/vmstat.c +15 -0
@@ -1581,6 +1581,21 @@ static void pagetypeinfo_showfree_print(struct seq_file *m,break ; } } +list_for_each (curr, &area->pend_list[mtype]) { +/* +* Cap the pend_list iteration because it might +* be really large and we are under a spinlock +* so a long time spent here could trigger a +* hard lockup detector. Anyway this is a +* debugging tool so knowing there is a handful +* of pages of this order should be more than +* sufficient. +*/ +if (++freecount >=100000 ) { + overflow = true; +break ; + } + }seq_printf (m,"%s%6lu " , overflow ?">" :"" , freecount);spin_unlock_irq (&zone->lock);cond_resched ();
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 14/25 mm/rmap: recognize read-only tlb entries during batched tlb flush
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that requires to recognize read-only tlb entries and handle them in a different way. The newly introduced API in this patch, fold_ubc(), will be used by luf mechanism.
--- include/linux/sched.h | 1 + mm/rmap.c | 16 ++++++++++++++-- 2 files changed, 15 insertions(+), 2 deletions(-)
include/linux/sched.h +1 -0
@@ -1407,6 +1407,7 @@ struct task_struct {struct tlbflush_unmap_batch tlb_ubc;struct tlbflush_unmap_batch tlb_ubc_takeoff; +struct tlbflush_unmap_batch tlb_ubc_ro;/* Cache last used pipe for splice(): */ struct pipe_inode_info *splice_pipe;
mm/rmap.c +14 -2
@@ -775,6 +775,7 @@ void fold_luf_batch(struct luf_batch *dst, struct luf_batch *src)void try_to_unmap_flush_takeoff (void ) {struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; +struct tlbflush_unmap_batch *tlb_ubc_ro = ¤t->tlb_ubc_ro;struct tlbflush_unmap_batch *tlb_ubc_takeoff = ¤t->tlb_ubc_takeoff;if (!tlb_ubc_takeoff->flush_required)
@@ -789,6 +790,9 @@ void try_to_unmap_flush_takeoff(void)if (arch_tlbbatch_done (&tlb_ubc->arch, &tlb_ubc_takeoff->arch))reset_batch (tlb_ubc); +if (arch_tlbbatch_done (&tlb_ubc_ro->arch, &tlb_ubc_takeoff->arch)) +reset_batch (tlb_ubc_ro); +reset_batch (tlb_ubc_takeoff); }
@@ -801,7 +805,9 @@ void try_to_unmap_flush_takeoff(void)void try_to_unmap_flush (void ) {struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; +struct tlbflush_unmap_batch *tlb_ubc_ro = ¤t->tlb_ubc_ro; +fold_batch (tlb_ubc, tlb_ubc_ro, true);if (!tlb_ubc->flush_required)return ;
@@ -813,8 +819,9 @@ void try_to_unmap_flush(void)void try_to_unmap_flush_dirty (void ) {struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; +struct tlbflush_unmap_batch *tlb_ubc_ro = ¤t->tlb_ubc_ro; -if (tlb_ubc->writable) +if (tlb_ubc->writable || tlb_ubc_ro->writable)try_to_unmap_flush (); }
@@ -831,13 +838,18 @@ void try_to_unmap_flush_dirty(void)static void set_tlb_ubc_flush_pending (struct mm_struct *mm,pte_t pteval,unsigned long start,unsigned long end) { -struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; +struct tlbflush_unmap_batch *tlb_ubc;int batch;bool writable =pte_dirty (pteval);if (!pte_accessible (mm, pteval))return ; +if (pte_write (pteval)) + tlb_ubc = ¤t->tlb_ubc; +else + tlb_ubc = ¤t->tlb_ubc_ro; +arch_tlbbatch_add_pending (&tlb_ubc->arch, mm, start, end); tlb_ubc->flush_required = true;
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 15/25 fs, filemap: refactor to gather the scattered ->write_{begin,end}() calls
In reply to Byungchul ParkFunctionally, no change. This is a preparation for luf mechanism that requires to hook when updating page cache that might have pages that have been mapped on any tasks so that tlb flush needed can be performed.
--- drivers/gpu/drm/i915/gem/i915_gem_shmem.c | 11 ++++------- fs/affs/file.c | 4 ++-- fs/buffer.c | 14 ++++++-------- fs/exfat/file.c | 5 ++--- fs/ext4/verity.c | 5 ++--- fs/f2fs/super.c | 5 ++--- fs/f2fs/verity.c | 5 ++--- fs/namei.c | 5 ++--- include/linux/fs.h | 18 ++++++++++++++++++ mm/filemap.c | 5 ++--- 10 files changed, 42 insertions(+), 35 deletions(-)
drivers/gpu/drm/i915/gem/i915_gem_shmem.c +4 -7
@@ -418,7 +418,6 @@ shmem_pwrite(struct drm_i915_gem_object *obj, const struct drm_i915_gem_pwrite *arg) {struct address_space *mapping = obj->base.filp->f_mapping; -const struct address_space_operations *aops = mapping->a_ops;char __user *user_data =u64_to_user_ptr (arg->data_ptr);u64 remain;loff_t pos;
@@ -477,7 +476,7 @@ shmem_pwrite(struct drm_i915_gem_object *obj,if (err)return err; - err = aops->write_begin (obj->base.filp, mapping, pos, len, + err =mapping_write_begin (obj->base.filp, mapping, pos, len, &folio, &data);if (err <0 )return err;
@@ -488,7 +487,7 @@ shmem_pwrite(struct drm_i915_gem_object *obj,pagefault_enable ();kunmap_local (vaddr); - err = aops->write_end (obj->base.filp, mapping, pos, len, + err =mapping_write_end (obj->base.filp, mapping, pos, len, len - unwritten, folio, data);if (err <0 )return err;
@@ -654,7 +653,6 @@ i915_gem_object_create_shmem_from_data(struct drm_i915_private *i915, {struct drm_i915_gem_object *obj;struct file *file; -const struct address_space_operations *aops;loff_t pos;int err;
@@ -666,21 +664,20 @@ i915_gem_object_create_shmem_from_data(struct drm_i915_private *i915,GEM_BUG_ON (obj->write_domain !=I915_GEM_DOMAIN_CPU ); file = obj->base.filp; - aops = file->f_mapping->a_ops; pos =0 ; do { unsigned int len = min_t(typeof(size), size, PAGE_SIZE); struct folio *folio; void *fsdata; - err = aops->write_begin(file, file->f_mapping, pos, len, + err = mapping_write_begin(file, file->f_mapping, pos, len, &folio, &fsdata); if (err < 0) goto fail; memcpy_to_folio(folio, offset_in_folio(folio, pos), data, len); - err = aops->write_end(file, file->f_mapping, pos, len, len, + err = mapping_write_end(file, file->f_mapping, pos, len, len, folio, fsdata); if (err < 0) goto fail;
fs/affs/file.c +2 -2
@@ -885,9 +885,9 @@ affs_truncate(struct inode *inode)loff_t isize = inode->i_size;int res; - res = mapping->a_ops->write_begin (NULL , mapping, isize,0 , &folio, &fsdata); + res =mapping_write_begin (NULL , mapping, isize,0 , &folio, &fsdata);if (!res) - res = mapping->a_ops->write_end (NULL , mapping, isize,0 ,0 , folio, fsdata); + res =mapping_write_end (NULL , mapping, isize,0 ,0 , folio, fsdata);else inode->i_size =AFFS_I (inode)->mmu_private;mark_inode_dirty (inode);
fs/buffer.c +6 -8
@@ -2457,7 +2457,6 @@ EXPORT_SYMBOL(block_read_full_folio);int generic_cont_expand_simple (struct inode *inode,loff_t size) {struct address_space *mapping = inode->i_mapping; -const struct address_space_operations *aops = mapping->a_ops;struct folio *folio;void *fsdata =NULL ;int err;
@@ -2466,11 +2465,11 @@ int generic_cont_expand_simple(struct inode *inode, loff_t size)if (err) goto out; - err = aops->write_begin (NULL , mapping, size,0 , &folio, &fsdata); + err =mapping_write_begin (NULL , mapping, size,0 , &folio, &fsdata);if (err) goto out; - err = aops->write_end (NULL , mapping, size,0 ,0 , folio, fsdata); + err =mapping_write_end (NULL , mapping, size,0 ,0 , folio, fsdata);BUG_ON (err >0 ); out:
@@ -2482,7 +2481,6 @@ static int cont_expand_zero(struct file *file, struct address_space *mapping, loff_t pos, loff_t *bytes) { struct inode *inode = mapping->host; -const struct address_space_operations *aops = mapping->a_ops;unsigned int blocksize =i_blocksize (inode);struct folio *folio;void *fsdata =NULL ;
@@ -2502,12 +2500,12 @@ static int cont_expand_zero(struct file *file, struct address_space *mapping, } len =PAGE_SIZE - zerofrom; - err = aops->write_begin (file, mapping, curpos, len, + err =mapping_write_begin (file, mapping, curpos, len, &folio, &fsdata);if (err) goto out;folio_zero_range (folio,offset_in_folio (folio, curpos), len); - err = aops->write_end (file, mapping, curpos, len, len, + err =mapping_write_end (file, mapping, curpos, len, len, folio, fsdata);if (err <0 ) goto out;
@@ -2535,12 +2533,12 @@ static int cont_expand_zero(struct file *file, struct address_space *mapping, } len = offset - zerofrom; - err = aops->write_begin (file, mapping, curpos, len, + err =mapping_write_begin (file, mapping, curpos, len, &folio, &fsdata);if (err) goto out;folio_zero_range (folio,offset_in_folio (folio, curpos), len); - err = aops->write_end (file, mapping, curpos, len, len, + err =mapping_write_end (file, mapping, curpos, len, len, folio, fsdata);if (err <0 ) goto out;
fs/exfat/file.c +2 -3
@@ -539,7 +539,6 @@ static int exfat_extend_valid_size(struct file *file, loff_t new_valid_size)struct inode *inode =file_inode (file);struct exfat_inode_info *ei =EXFAT_I (inode);struct address_space *mapping = inode->i_mapping; -const struct address_space_operations *ops = mapping->a_ops; pos = ei->valid_size;while (pos < new_valid_size) {
@@ -551,14 +550,14 @@ static int exfat_extend_valid_size(struct file *file, loff_t new_valid_size)if (pos + len > new_valid_size) len = new_valid_size - pos; - err = ops->write_begin (file, mapping, pos, len, &folio,NULL ); + err =mapping_write_begin (file, mapping, pos, len, &folio,NULL );if (err) goto out; off =offset_in_folio (folio, pos);folio_zero_new_buffers (folio, off, off + len); - err = ops->write_end (file, mapping, pos, len, len, folio,NULL ); + err =mapping_write_end (file, mapping, pos, len, len, folio,NULL );if (err <0 ) goto out; pos += len;
fs/ext4/verity.c +2 -3
@@ -68,7 +68,6 @@ static int pagecache_write(struct inode *inode, const void *buf, size_t count, loff_t pos) {struct address_space *mapping = inode->i_mapping; -const struct address_space_operations *aops = mapping->a_ops;if (pos + count > inode->i_sb->s_maxbytes)return -EFBIG ;
@@ -80,13 +79,13 @@ static int pagecache_write(struct inode *inode, const void *buf, size_t count,void *fsdata =NULL ;int res; - res = aops->write_begin (NULL , mapping, pos, n, &folio, &fsdata); + res =mapping_write_begin (NULL , mapping, pos, n, &folio, &fsdata);if (res)return res;memcpy_to_folio (folio,offset_in_folio (folio, pos), buf, n); - res = aops->write_end (NULL , mapping, pos, n, n, folio, fsdata); + res =mapping_write_end (NULL , mapping, pos, n, n, folio, fsdata);if (res <0 )return res;if (res != n)
fs/f2fs/super.c +2 -3
@@ -2710,7 +2710,6 @@ static ssize_t f2fs_quota_write(struct super_block *sb, int type, {struct inode *inode =sb_dqopt (sb)->files[type];struct address_space *mapping = inode->i_mapping; -const struct address_space_operations *a_ops = mapping->a_ops;int offset = off & (sb->s_blocksize -1 );size_t towrite = len;struct folio *folio;
@@ -2722,7 +2721,7 @@ static ssize_t f2fs_quota_write(struct super_block *sb, int type, tocopy = min_t(unsigned long, sb->s_blocksize - offset, towrite); retry: - err = a_ops->write_begin (NULL , mapping, off, tocopy, + err =mapping_write_begin (NULL , mapping, off, tocopy, &folio, &fsdata);if (unlikely (err)) {if (err == -ENOMEM ) {
@@ -2735,7 +2734,7 @@ static ssize_t f2fs_quota_write(struct super_block *sb, int type,memcpy_to_folio (folio,offset_in_folio (folio, off), data, tocopy); - a_ops->write_end (NULL , mapping, off, tocopy, tocopy, +mapping_write_end (NULL , mapping, off, tocopy, tocopy, folio, fsdata); offset =0 ; towrite -= tocopy;
fs/f2fs/verity.c +2 -3
@@ -72,7 +72,6 @@ static int pagecache_write(struct inode *inode, const void *buf, size_t count, loff_t pos) {struct address_space *mapping = inode->i_mapping; -const struct address_space_operations *aops = mapping->a_ops;if (pos + count >F2FS_BLK_TO_BYTES (max_file_blocks (inode)))return -EFBIG ;
@@ -84,13 +83,13 @@ static int pagecache_write(struct inode *inode, const void *buf, size_t count,void *fsdata =NULL ;int res; - res = aops->write_begin (NULL , mapping, pos, n, &folio, &fsdata); + res =mapping_write_begin (NULL , mapping, pos, n, &folio, &fsdata);if (res)return res;memcpy_to_folio (folio,offset_in_folio (folio, pos), buf, n); - res = aops->write_end (NULL , mapping, pos, n, n, folio, fsdata); + res =mapping_write_end (NULL , mapping, pos, n, n, folio, fsdata);if (res <0 )return res;if (res != n)
fs/namei.c +2 -3
@@ -5409,7 +5409,6 @@ EXPORT_SYMBOL(page_readlink);int page_symlink (struct inode *inode,const char *symname,int len) {struct address_space *mapping = inode->i_mapping; -const struct address_space_operations *aops = mapping->a_ops;bool nofs = !mapping_gfp_constraint (mapping, __GFP_FS);struct folio *folio;void *fsdata =NULL ;
@@ -5419,7 +5418,7 @@ int page_symlink(struct inode *inode, const char *symname, int len) retry:if (nofs) flags =memalloc_nofs_save (); - err = aops->write_begin (NULL , mapping,0 , len-1 , &folio, &fsdata); + err =mapping_write_begin (NULL , mapping,0 , len-1 , &folio, &fsdata);if (nofs)memalloc_nofs_restore (flags);if (err)
@@ -5427,7 +5426,7 @@ int page_symlink(struct inode *inode, const char *symname, int len)memcpy (folio_address (folio), symname, len -1 ); - err = aops->write_end (NULL , mapping,0 , len -1 , len -1 , + err =mapping_write_end (NULL , mapping,0 , len -1 , len -1 , folio, fsdata);if (err <0 ) goto fail;
include/linux/fs.h +18 -0
@@ -531,6 +531,24 @@ struct address_space {#define PAGECACHE_TAG_WRITEBACK XA_MARK_1#define PAGECACHE_TAG_TOWRITE XA_MARK_2 +static inline int mapping_write_begin (struct file *file, +struct address_space *mapping, +loff_t pos,unsigned len, +struct folio **foliop,void **fsdata) +{ +return mapping->a_ops->write_begin (file, mapping, pos, len, foliop, + fsdata); +} + +static inline int mapping_write_end (struct file *file, +struct address_space *mapping, +loff_t pos,unsigned len,unsigned copied, +struct folio *folio,void *fsdata) +{ +return mapping->a_ops->write_end (file, mapping, pos, len, copied, + folio, fsdata); +} +/* * Returns true if any of the pages in the mapping are marked with the tag. */
mm/filemap.c +2 -3
@@ -4141,7 +4141,6 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i)struct file *file = iocb->ki_filp;loff_t pos = iocb->ki_pos;struct address_space *mapping = file->f_mapping; -const struct address_space_operations *a_ops = mapping->a_ops;size_t chunk =mapping_max_folio_size (mapping);long status =0 ;ssize_t written =0 ;
@@ -4175,7 +4174,7 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i)break ; } - status = a_ops->write_begin (file, mapping, pos, bytes, + status =mapping_write_begin (file, mapping, pos, bytes, &folio, &fsdata);if (unlikely (status <0 ))break ;
@@ -4190,7 +4189,7 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i) copied =copy_folio_from_iter_atomic (folio, offset, bytes, i);flush_dcache_folio (folio); - status = a_ops->write_end (file, mapping, pos, bytes, copied, + status =mapping_write_end (file, mapping, pos, bytes, copied, folio, fsdata);if (unlikely (status != copied)) {iov_iter_revert (i, copied -max (status,0L ));
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 16/25 mm: implement LUF(Lazy Unmap Flush) defering tlb flush when folios get unmapped
In reply to Byungchul ParkA new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read-only and were unmapped, as long as the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries.
tlb flush can be defered when folios get unmapped as long as it guarantees to perform tlb flush needed, before the folios actually become used, of course, only if all the corresponding ptes don't have write permission. Otherwise, the system will get messed up.
To achieve that, for the folios that map only to non-writable tlb entries, prevent tlb flush during unmapping but perform it just before the folios actually become used, out of buddy or pcp.
However, we should cancel the pending by LUF and perform the deferred TLB flush right away when:
1. a writable pte is newly set through fault handler
2. a file is updated
3. kasan needs poisoning on free
4. the kernel wants to init pages on free
No matter what type of workload is used for performance evaluation, the result would be positive thanks to the unconditional reduction of tlb flushes, tlb misses and interrupts. For the test, I picked up one of the most popular and heavy workload, llama.cpp that is a LLM(Large Language Model) inference engine.
The result would depend on memory latency and how often reclaim runs, which implies tlb miss overhead and how many times unmapping happens. In my system, the result shows:
1. tlb shootdown interrupts are reduced about 97%.
2. The test program runtime is reduced about 4.5%.
The test environment and the test set are like:
Machine: bare metal, x86_64, Intel(R) Xeon(R) Gold 6430
CPU: 1 socket 64 core with hyper thread on Numa: 2 nodes (64 CPUs DRAM 42GB, no CPUs CXL expander 98GB) Config: swap off, numa balancing tiering on, demotion enabled
llama.cpp/main -m $(70G_model1) -p "who are you?" -s 1 -t 15 -n 20 & llama.cpp/main -m $(70G_model2) -p "who are you?" -s 1 -t 15 -n 20 & llama.cpp/main -m $(70G_model3) -p "who are you?" -s 1 -t 15 -n 20 & wait
where,
-t: nr of threads, -s: seed used to make the runtime stable, -n: nr of tokens that determines the runtime, -p: prompt to ask, -m: LLM model to use.
Run the test set 5 times successively with caches dropped every run via 'echo 3 > /proc/sys/vm/drop_caches'. Each inference prints its runtime at the end of each. The results are like:
1. Runtime from the output of llama.cpp
BEFORE
------
llama_print_timings: total time = 883450.54 ms / 24 tokens llama_print_timings: total time = 861665.91 ms / 24 tokens llama_print_timings: total time = 898079.02 ms / 24 tokens llama_print_timings: total time = 879897.69 ms / 24 tokens llama_print_timings: total time = 892360.75 ms / 24 tokens llama_print_timings: total time = 884587.85 ms / 24 tokens llama_print_timings: total time = 861023.19 ms / 24 tokens llama_print_timings: total time = 900022.18 ms / 24 tokens llama_print_timings: total time = 878771.88 ms / 24 tokens llama_print_timings: total time = 889027.98 ms / 24 tokens llama_print_timings: total time = 880783.90 ms / 24 tokens llama_print_timings: total time = 856475.29 ms / 24 tokens llama_print_timings: total time = 896842.21 ms / 24 tokens llama_print_timings: total time = 878883.53 ms / 24 tokens llama_print_timings: total time = 890122.10 ms / 24 tokens
AFTER
-----
llama_print_timings: total time = 871060.86 ms / 24 tokens llama_print_timings: total time = 825609.53 ms / 24 tokens llama_print_timings: total time = 836854.81 ms / 24 tokens llama_print_timings: total time = 843147.99 ms / 24 tokens llama_print_timings: total time = 831426.65 ms / 24 tokens llama_print_timings: total time = 873939.23 ms / 24 tokens llama_print_timings: total time = 826127.69 ms / 24 tokens llama_print_timings: total time = 835489.26 ms / 24 tokens llama_print_timings: total time = 842589.62 ms / 24 tokens llama_print_timings: total time = 833700.66 ms / 24 tokens llama_print_timings: total time = 875996.19 ms / 24 tokens llama_print_timings: total time = 826401.73 ms / 24 tokens llama_print_timings: total time = 839341.28 ms / 24 tokens llama_print_timings: total time = 841075.10 ms / 24 tokens llama_print_timings: total time = 835136.41 ms / 24 tokens
2. tlb shootdowns from 'cat /proc/interrupts'
BEFORE
------
TLB:
80911532 93691786 100296251 111062810 109769109 109862429 108968588 119175230 115779676 118377498 119325266 120300143 124514185 116697222 121068466 118031913 122660681 117494403 121819907 116960596 120936335 117217061 118630217 122322724 119595577 111693298 119232201 120030377 115334687 113179982 118808254 116353592 140987367 137095516 131724276 139742240 136501150 130428761 127585535 132483981 133430250 133756207 131786710 126365824 129812539 133850040 131742690 125142213 128572830 132234350 131945922 128417707 133355434 129972846 126331823 134050849 133991626 121129038 124637283 132830916 126875507 122322440 125776487 124340278 TLB shootdowns
AFTER
-----
TLB:
2121206 2615108 2983494 2911950 3055086 3092672
3204894 3346082 3286744 3307310 3357296 3315940
3428034 3112596 3143325 3185551 3186493 3322314
3330523 3339663 3156064 3272070 3296309 3198962
3332662 3315870 3234467 3353240 3281234 3300666
3345452 3173097 4009196 3932215 3898735 3726531
3717982 3671726 3728788 3724613 3799147 3691764
3620630 3684655 3666688 3393974 3448651 3487593
3446357 3618418 3671920 3712949 3575264 3715385
3641513 3630897 3691047 3630690 3504933 3662647
3629926 3443044 3832970 3548813 TLB shootdowns--- include/asm-generic/tlb.h | 5 ++ include/linux/fs.h | 12 +++- include/linux/mm_types.h | 6 ++ include/linux/sched.h | 9 +++ kernel/sched/core.c | 1 + mm/internal.h | 94 ++++++++++++++++++++++++- mm/memory.c | 15 ++++ mm/pgtable-generic.c | 2 + mm/rmap.c | 141 +++++++++++++++++++++++++++++++++++--- mm/truncate.c | 55 +++++++++++++-- mm/vmscan.c | 12 +++- 11 files changed, 333 insertions(+), 19 deletions(-)
include/asm-generic/tlb.h +5 -0
@@ -567,6 +567,11 @@ static inline void tlb_start_vma(struct mmu_gather *tlb, struct vm_area_struct *static inline void tlb_end_vma (struct mmu_gather *tlb,struct vm_area_struct *vma) { +/* +* Don't leave stale tlb entries for this vma. +*/ +luf_flush (0 ); +if (tlb->fullmm ||IS_ENABLED (CONFIG_MMU_GATHER_MERGE_VMAS ))return ;
include/linux/fs.h +11 -1
@@ -536,8 +536,18 @@ static inline int mapping_write_begin(struct file *file, loff_t pos, unsigned len, struct folio **foliop, void **fsdata) { - return mapping->a_ops->write_begin(file, mapping, pos, len, foliop, + int ret; + + ret = mapping->a_ops->write_begin (file, mapping, pos, len, foliop, fsdata); + +/* +* Ensure to clean stale tlb entries for this mapping. +*/ +if (!ret) +luf_flush (0 ); + +return ret; } static inline int mapping_write_end(struct file *file,
include/linux/mm_types.h +6 -0
@@ -1353,6 +1353,12 @@ extern void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm);extern void tlb_gather_mmu_fullmm (struct mmu_gather *tlb,struct mm_struct *mm);extern void tlb_finish_mmu (struct mmu_gather *tlb); +#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH ) +void luf_flush (unsigned short luf_key); +#else +static inline void luf_flush (unsigned short luf_key) {} +#endif +struct vm_fault ;/**
include/linux/sched.h +9 -0
@@ -1408,6 +1408,15 @@ struct task_struct {struct tlbflush_unmap_batch tlb_ubc;struct tlbflush_unmap_batch tlb_ubc_takeoff;struct tlbflush_unmap_batch tlb_ubc_ro; +struct tlbflush_unmap_batch tlb_ubc_luf; + +#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH ) +/* +* whether all the mappings of a folio during unmap are read-only +* so that luf can work on the folio +*/ +bool can_luf; +#endif /* Cache last used pipe for splice(): */ struct pipe_inode_info *splice_pipe;
kernel/sched/core.c +1 -0
@@ -5275,6 +5275,7 @@ static struct rq *finish_task_switch(struct task_struct *prev)if (mm) {membarrier_mm_sync_core_before_usermode (mm);mmdrop_lazy_tlb_sched (mm); +luf_flush (0 ); }if (unlikely (prev_state ==TASK_DEAD )) {
mm/internal.h +93 -1
106 lines not on this page · raw message
mm/memory.c +15 -0
@@ -6216,6 +6216,7 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,struct mm_struct *mm = vma->vm_mm;vm_fault_t ret;bool is_droppable; +bool flush = false;__set_current_state (TASK_RUNNING );
@@ -6241,6 +6242,14 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,lru_gen_enter_fault (vma); +/* +* Any potential cases that make pte writable even forcely +* should be considered. +*/ +if (vma->vm_flags & (VM_WRITE |VM_MAYWRITE ) || + flags &FAULT_FLAG_WRITE ) + flush = true; +if (unlikely (is_vm_hugetlb_page (vma))) ret =hugetlb_fault (vma->vm_mm, vma, address, flags);else
@@ -6272,6 +6281,12 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, out:mm_account_fault (mm, regs, address, flags, ret); +/* +* Ensure to clean stale tlb entries for this vma. +*/ +if (flush) +luf_flush (0 ); +return ret; }EXPORT_SYMBOL_GPL (handle_mm_fault);
mm/pgtable-generic.c +2 -0
@@ -99,6 +99,8 @@ pte_t ptep_clear_flush(struct vm_area_struct *vma, unsigned long address, pte =ptep_get_and_clear (mm, address, ptep);if (pte_accessible (mm, pte))flush_tlb_page (vma, address); +else +luf_flush (0 ); return pte; }#endif
mm/rmap.c +133 -8
276 lines not on this page · raw message
mm/truncate.c +48 -7
117 lines not on this page · raw message
mm/vmscan.c +10 -2
@@ -838,6 +838,8 @@ static int __remove_mapping(struct address_space *mapping, struct folio *folio,*/ long remove_mapping (struct address_space *mapping,struct folio *folio) { +long ret =0 ; +if (__remove_mapping (mapping, folio, false,NULL )) {/* * Unfreezing the refcount with 1 effectively
@@ -845,9 +847,15 @@ long remove_mapping(struct address_space *mapping, struct folio *folio)* atomic operation. */ folio_ref_unfreeze (folio,1 ); -return folio_nr_pages (folio); + ret =folio_nr_pages (folio); } -return 0 ; + +/* +* Ensure to clean stale tlb entries for this mapping. +*/ +luf_flush (0 ); + +return ret; }/**
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 17/25 x86/tlb, riscv/tlb, arm64/tlbflush, mm: remove cpus from tlb shootdown that already have been done
In reply to Byungchul Parkluf mechanism performs tlb shootdown for mappings that have been unmapped in lazy manner. However, it doesn't have to perform tlb shootdown to cpus that already have been done by others since the tlb shootdown was desired.
Since luf already introduced its own generation number used as a global timestamp, luf_ugen, it's possible to selectively pick cpus that have been done tlb flush required.
This patch introduced APIs that use the generation number to select and remove those cpus so that it can perform tlb shootdown with a smaller cpumask, for all the CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH archs, x86, riscv, and arm64.
--- arch/arm64/include/asm/tlbflush.h | 26 +++++++ arch/riscv/include/asm/tlbflush.h | 4 ++ arch/riscv/mm/tlbflush.c | 108 ++++++++++++++++++++++++++++++ arch/x86/include/asm/tlbflush.h | 4 ++ arch/x86/mm/tlb.c | 108 ++++++++++++++++++++++++++++++ include/linux/sched.h | 1 + mm/internal.h | 4 ++ mm/page_alloc.c | 32 +++++++-- mm/rmap.c | 46 ++++++++++++- 9 files changed, 327 insertions(+), 6 deletions(-)
arch/arm64/include/asm/tlbflush.h +26 -0
@@ -347,6 +347,32 @@ static inline void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)dsb (ish); } +static inline bool arch_tlbbatch_check_done (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen) +{ +/* +* Nothing is needed in this architecture. +*/ +return true; +} + +static inline bool arch_tlbbatch_diet (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen) +{ +/* +* Nothing is needed in this architecture. +*/ +return true; +} + +static inline void arch_tlbbatch_mark_ugen (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen) +{ +/* nothing to do */ +} + +static inline void arch_mm_mark_ugen (struct mm_struct *mm,unsigned long ugen) +{ +/* nothing to do */ +} +static inline void arch_tlbbatch_clear (struct arch_tlbflush_unmap_batch *batch) {/* nothing to do */
arch/riscv/include/asm/tlbflush.h +4 -0
@@ -64,6 +64,10 @@ void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch, struct mm_struct *mm, unsigned long start, unsigned long end);void arch_flush_tlb_batched_pending (struct mm_struct *mm);void arch_tlbbatch_flush (struct arch_tlbflush_unmap_batch *batch); +bool arch_tlbbatch_check_done (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen); +bool arch_tlbbatch_diet (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen); +void arch_tlbbatch_mark_ugen (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen); +void arch_mm_mark_ugen (struct mm_struct *mm,unsigned long ugen);static inline void arch_tlbbatch_clear (struct arch_tlbflush_unmap_batch *batch) {
arch/riscv/mm/tlbflush.c +108 -0
111 lines not on this page · raw message
arch/x86/include/asm/tlbflush.h +4 -0
@@ -293,6 +293,10 @@ static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm) }extern void arch_tlbbatch_flush (struct arch_tlbflush_unmap_batch *batch); +extern bool arch_tlbbatch_check_done (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen); +extern bool arch_tlbbatch_diet (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen); +extern void arch_tlbbatch_mark_ugen (struct arch_tlbflush_unmap_batch *batch,unsigned long ugen); +extern void arch_mm_mark_ugen (struct mm_struct *mm,unsigned long ugen);static inline void arch_tlbbatch_clear (struct arch_tlbflush_unmap_batch *batch) {
arch/x86/mm/tlb.c +108 -0
114 lines not on this page · raw message
include/linux/sched.h +1 -0
@@ -1403,6 +1403,7 @@ struct task_struct {#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH )int luf_no_shootdown;int luf_takeoff_started; +unsigned long luf_ugen;#endif struct tlbflush_unmap_batch tlb_ubc;
mm/internal.h +4 -0
@@ -1276,6 +1276,7 @@ void try_to_unmap_flush(void);void try_to_unmap_flush_dirty (void );void try_to_unmap_flush_takeoff (void );void flush_tlb_batched_pending (struct mm_struct *mm); +void reset_batch (struct tlbflush_unmap_batch *batch);void fold_batch (struct tlbflush_unmap_batch *dst,struct tlbflush_unmap_batch *src,bool reset);void fold_luf_batch (struct luf_batch *dst,struct luf_batch *src);#else
@@ -1291,6 +1292,9 @@ static inline void try_to_unmap_flush_takeoff(void)static inline void flush_tlb_batched_pending (struct mm_struct *mm) { } +static inline void reset_batch (struct tlbflush_unmap_batch *batch) +{ +}static inline void fold_batch (struct tlbflush_unmap_batch *dst,struct tlbflush_unmap_batch *src,bool reset) { }
mm/page_alloc.c +28 -4
@@ -668,9 +668,11 @@ bool luf_takeoff_start(void)*/ void luf_takeoff_end (void ) { +struct tlbflush_unmap_batch *tlb_ubc_takeoff = ¤t->tlb_ubc_takeoff;unsigned long flags;bool no_shootdown;bool outmost = false; +unsigned long cur_luf_ugen;local_irq_save (flags);VM_WARN_ON (!current->luf_takeoff_started);
@@ -697,10 +699,19 @@ void luf_takeoff_end(void)if (no_shootdown) goto out; + cur_luf_ugen = current->luf_ugen; + + current->luf_ugen =0 ; + +if (cur_luf_ugen &&arch_tlbbatch_diet (&tlb_ubc_takeoff->arch, cur_luf_ugen)) +reset_batch (tlb_ubc_takeoff); +try_to_unmap_flush_takeoff (); out: -if (outmost) +if (outmost) {VM_WARN_ON (current->luf_no_shootdown); +VM_WARN_ON (current->luf_ugen); + } }/*
@@ -757,6 +768,7 @@ bool luf_takeoff_check_and_fold(struct page *page)struct tlbflush_unmap_batch *tlb_ubc_takeoff = ¤t->tlb_ubc_takeoff;unsigned short luf_key =page_luf_key (page);struct luf_batch *lb; +unsigned long lb_ugen;unsigned long flags;/*
@@ -770,13 +782,25 @@ bool luf_takeoff_check_and_fold(struct page *page)if (!luf_key)return true; -if (current->luf_no_shootdown) -return false; - lb = &luf_batch[luf_key];read_lock_irqsave (&lb->lock, flags); + lb_ugen = lb->ugen; + +if (arch_tlbbatch_check_done (&lb->batch.arch, lb_ugen)) { +read_unlock_irqrestore (&lb->lock, flags); +return true; + } + +if (current->luf_no_shootdown) { +read_unlock_irqrestore (&lb->lock, flags); +return false; + } +fold_batch (tlb_ubc_takeoff, &lb->batch, false);read_unlock_irqrestore (&lb->lock, flags); + +if (!current->luf_ugen ||ugen_before (current->luf_ugen, lb_ugen)) + current->luf_ugen = lb_ugen; return true; }#endif
mm/rmap.c +44 -2
96 lines not on this page · raw message
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 18/25 mm/page_alloc: retry 3 times to take pcp pages on luf check failure
In reply to Byungchul Park--- mm/page_alloc.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-)
mm/page_alloc.c +20 -4
@@ -3384,6 +3384,12 @@ struct page *__rmqueue_pcplist(struct zone *zone, unsigned int order, {struct page *page; +/* +* give up taking page from pcp if it fails to take pcp page +* 3 times due to the tlb shootdownable issue. +*/ +int try_luf_pages =3 ; + do { if (list_empty(list)) { int batch = nr_pcp_alloc(pcp, zone, order);
@@ -3398,11 +3404,21 @@ struct page *__rmqueue_pcplist(struct zone *zone, unsigned int order,return NULL ; } - page = list_first_entry(list, struct page, pcp_list); -if (!luf_takeoff_check_and_fold (page)) +list_for_each_entry (page, list, pcp_list) { +if (luf_takeoff_check_and_fold (page)) { +list_del (&page->pcp_list); + pcp->count -=1 << order; +break ; + } +if (!--try_luf_pages) +return NULL ; + } + +/* +* If all the pages in the list fails... +*/ +if (list_entry_is_head (page, list, pcp_list))return NULL ; -list_del (&page->pcp_list); - pcp->count -=1 << order; }while (check_new_pages (page, order));return page;
Signature · 1 line
2.17.1
Byungchul Park
reviewerRFC PATCH v12 19/25 mm: skip luf tlb flush for luf'd mm that already has been done
In reply to Byungchul ParkFault hander performs tlb flush pended by luf when a new pte becomes to have write permission, no matter whether tlb flush required has been performed or not.
By storing luf generation number, luf_ugen, in struct mm_struct, we can skip unnecessary tlb flush.
--- include/asm-generic/tlb.h | 2 +- include/linux/mm_types.h | 9 +++++ kernel/fork.c | 1 + kernel/sched/core.c | 2 +- mm/memory.c | 22 ++++++++++-- mm/pgtable-generic.c | 2 +- mm/rmap.c | 74 +++++++++++++++++++++++++++++++++++++-- 7 files changed, 104 insertions(+), 8 deletions(-)
include/asm-generic/tlb.h +1 -1
@@ -570,7 +570,7 @@ static inline void tlb_end_vma(struct mmu_gather *tlb, struct vm_area_struct *vm/* * Don't leave stale tlb entries for this vma. */ -luf_flush (0 ); +luf_flush_vma (vma);if (tlb->fullmm ||IS_ENABLED (CONFIG_MMU_GATHER_MERGE_VMAS ))return ;
include/linux/mm_types.h +9 -0
@@ -39,8 +39,10 @@ struct luf_batch {unsigned long ugen; rwlock_t lock; }; +void luf_batch_init (struct luf_batch *lb);#else struct luf_batch {}; +static inline void luf_batch_init (struct luf_batch *lb) {}#endif /*
@@ -1073,6 +1075,9 @@ struct mm_struct {* moving a PROT_NONE mapped page. */ atomic_t tlb_flush_pending; + +/* luf batch for this mm */ +struct luf_batch luf_batch;#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH /* See flush_tlb_batched_pending() */ atomic_t tlb_flush_batched;
@@ -1355,8 +1360,12 @@ extern void tlb_finish_mmu(struct mmu_gather *tlb);#if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH )void luf_flush (unsigned short luf_key); +void luf_flush_mm (struct mm_struct *mm); +void luf_flush_vma (struct vm_area_struct *vma);#else static inline void luf_flush (unsigned short luf_key) {} +static inline void luf_flush_mm (struct mm_struct *mm) {} +static inline void luf_flush_vma (struct vm_area_struct *vma) {}#endif struct vm_fault ;
kernel/fork.c +1 -0
@@ -1265,6 +1265,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,memset (&mm->rss_stat,0 ,sizeof (mm->rss_stat));spin_lock_init (&mm->page_table_lock);spin_lock_init (&mm->arg_lock); +luf_batch_init (&mm->luf_batch);mm_init_cpumask (mm);mm_init_aio (mm);mm_init_owner (mm, p);
kernel/sched/core.c +1 -1
@@ -5275,7 +5275,7 @@ static struct rq *finish_task_switch(struct task_struct *prev)if (mm) {membarrier_mm_sync_core_before_usermode (mm);mmdrop_lazy_tlb_sched (mm); -luf_flush (0 ); +luf_flush_mm (mm); }if (unlikely (prev_state ==TASK_DEAD )) {
mm/memory.c +19 -3
@@ -6216,6 +6216,7 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,struct mm_struct *mm = vma->vm_mm;vm_fault_t ret;bool is_droppable; +struct address_space *mapping =NULL ;bool flush = false;__set_current_state (TASK_RUNNING );
@@ -6247,9 +6248,17 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,* should be considered. */ if (vma->vm_flags & (VM_WRITE |VM_MAYWRITE ) || - flags &FAULT_FLAG_WRITE ) + flags &FAULT_FLAG_WRITE ) { flush = true; +/* +* Doesn't care the !VM_SHARED cases because it won't +* update the pages that might be shared with others. +*/ +if (vma->vm_flags &VM_SHARED && vma->vm_file) + mapping = vma->vm_file->f_mapping; + } +if (unlikely (is_vm_hugetlb_page (vma))) ret =hugetlb_fault (vma->vm_mm, vma, address, flags);else
@@ -6284,8 +6293,15 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,/* * Ensure to clean stale tlb entries for this vma. */ -if (flush) -luf_flush (0 ); +if (flush) { +/* +* If it has a VM_SHARED mapping, all the mms involved +* should be luf_flush'ed. +*/ +if (mapping) +luf_flush (0 ); +luf_flush_mm (mm); + }return ret; }
mm/pgtable-generic.c +1 -1
@@ -100,7 +100,7 @@ pte_t ptep_clear_flush(struct vm_area_struct *vma, unsigned long address,if (pte_accessible (mm, pte))flush_tlb_page (vma, address);else -luf_flush (0 ); +luf_flush_vma (vma); return pte; }#endif
mm/rmap.c +72 -2
99 lines not on this page · raw message
Signature · 1 line
2.17.1