Skip to content

AMD broadcast TLB invalidation

Rik van Riel linux-mm lkml 15 versions – Applied

Status from this message: The following commit has been merged into the x86/mm branch of tip:

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v14 Latest13 patches 48 repliesApplied
  2. v1314 patches 28 repliesSuperseded
  3. v1216 patches 9 repliesSuperseded
Earlier versions · 12
VersionPostedPatchesRepliesStatus
v1112 patches36 repliesApplied
v1012 patches13 repliesSuperseded
v912 patches34 repliesSuperseded
v812 patches5 repliesSuperseded
v712 patches3 repliesSuperseded
v612 patches23 repliesSuperseded
v512 patches23 repliesSuperseded
v4 posting 2/212 patches23 repliesSuperseded
v4 RFC posting 1/20 of 10 patches0 repliesSuperseded
v312 patches76 repliesSuperseded
v2 RFC11 patches9 repliesSuperseded
v1 RFC10 patches17 repliesSuperseded

Changes in v14 from the cover letter

- code & comment cleanups (Boris)
- drop "noinvlpgb" commandline option (Boris)
- fix !CONFIG_X86_BROADCAST_TLB_FLUSH compile anywhere in the series

Patches in v14 Applied

#Subject and filesSizeRepliesApplied
1/13x86/mm: consolidate full flush threshold decisionarch/x86/mm/tlb.c+19 -2211 replies
2/13x86/mm: get INVLPGB count max from CPUIDarch/x86/Kconfig.cpu arch/x86/include/asm/cpufeatures.h arch/x86/include/asm/tlbflush.h +1 more+14 -02 replies
3/13x86/mm: add INVLPGB support codearch/x86/include/asm/disabled-features.h arch/x86/include/asm/tlb.h+105 -110 replies
4/13x86/mm: use INVLPGB for kernel TLB flushesarch/x86/mm/tlb.c+30 -22 replies
5/13x86/mm: use INVLPGB in flush_tlb_allarch/x86/mm/tlb.c+9 -14 replies
6/13x86/mm: use broadcast TLB flushing for page reclaim TLB flushingarch/x86/mm/tlb.c+3 -11 reply
7/13x86/mm: add global ASID allocation helper functionsarch/x86/include/asm/mmu.h arch/x86/include/asm/mmu_context.h arch/x86/include/asm/tlbflush.h +1 more+199 -31 reply
8/13x86/mm: global ASID context switch & TLB flush handlingarch/x86/include/asm/tlbflush.h arch/x86/mm/tlb.c+88 -71 reply
9/13x86/mm: global ASID process exit helpersarch/x86/include/asm/mmu_context.h+12 -03 replies
10/13x86/mm: enable broadcast TLB invalidation for multi-threaded processesarch/x86/include/asm/tlbflush.h arch/x86/mm/tlb.c+115 -11 reply
11/13x86/mm: do targeted broadcast flushing from tlbbatch codearch/x86/include/asm/tlb.h arch/x86/include/asm/tlbflush.h arch/x86/mm/tlb.c+107 -188 replies
12/13x86/mm: enable AMD translation cache extensionsarch/x86/include/asm/msr-index.h arch/x86/kernel/cpu/amd.c tools/arch/x86/include/asm/msr-index.h+8 -00 replies
13/13x86/mm: only invalidate final translations with INVLPGBarch/x86/include/asm/tlb.h arch/x86/mm/tlb.c+15 -82 replies

Reviews trailers per patch

5 of 13 patches have a Reviewed-by or Acked-by.

PatchManali ShuklaBrendan JackmanMichael KelleyDave HansenBorislav Petkov (AMD)Nadav Amit
1/13T-b A-b R-b
2/13T-b T-b T-b A-b
3/13T-b T-b T-b A-b
4/13T-b T-b T-b A-b
5/13T-b T-b T-b
6/13T-b T-b T-b
7/13T-b T-b T-b
8/13
9/13
10/13T-b T-b T-b R-b R-b
11/13T-b T-b T-b
12/13T-b T-b T-b
13/13T-b T-b T-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 6 reviewers.

Read the v14 thread

Cover letter [PATCH v14 00/13] AMD broadcast TLB invalidation

Add support for broadcast TLB invalidation using AMD's INVLPGB instruction.

This allows the kernel to invalidate TLB entries on remote CPUs without needing to send IPIs, without having to wait for remote CPUs to handle those interrupts, and with less interruption to what was running on those CPUs.

Because x86 PCID space is limited, and there are some very large systems out there, broadcast TLB invalidation is only used for processes that are active on 3 or more CPUs, with the threshold being gradually increased the more the PCID space gets exhausted.

Combined with the removal of unnecessary lru_add_drain calls (see https://lkml.org/lkml/2024/12/19/1388) this results in a nice performance boost for the will-it-scale tlb_flush2_threads test on an AMD Milan system with 36 cores:

- vanilla kernel:           527k loops/second
- lru_add_drain removal:    731k loops/second
- only INVLPGB:             527k loops/second

- lru_add_drain + INVLPGB: 1157k loops/second

Profiling with only the INVLPGB changes showed while TLB invalidation went down from 40% of the total CPU time to only around 4% of CPU time, the contention simply moved to the LRU lock.

Fixing both at the same time about doubles the
number of iterations per second from this case.

Some numbers closer to real world performance
can be found at Phoronix, thanks to Michael:

https://www.phoronix.com/news/AMD-INVLPGB-Linux-Benefits

My current plan is to implement support for Intel's RAR (Remote Action Request) TLB flushing in a follow-up series, after this thing has been merged into -tip. Making things any larger would just be unwieldy for reviewers.

v14:
- code & comment cleanups (Boris)
- drop "noinvlpgb" commandline option (Boris)
- fix !CONFIG_X86_BROADCAST_TLB_FLUSH compile anywhere in the series
v13:
- move invlpgb_count_max back to amd.c for resume (Boris, Oleksandr)
- fix Kconfig circular dependency (Tom, Boris)
- add performance numbers to the patch adding invlpgb for userspace (Ingo)
- drop page table RCU free patches (already in -tip)
v12
- make sure "nopcid" command line option turns off invlpgb (Brendan)
- add "noinvlpgb" kernel command line option
- split out kernel TLB flushing differently (Dave & Yosry)
- split up the patch that does invlpgb flushing for user processes (Dave)
- clean up get_flush_tlb_info (Boris)
- move invlpgb_count_max initialization to get_cpu_cap (Boris)
- bunch more comments as requested
v11:
- resolve conflict with CONFIG_PT_RECLAIM code
- a few more cleanups (Peter, Brendan, Nadav)
v10:
- simplify partial pages with min(nr, 1) in the invlpgb loop (Peter)
- document x86 paravirt, AMD invlpgb, and ARM64 flush without IPI (Brendan)
- remove IS_ENABLED(CONFIG_X86_BROADCAST_TLB_FLUSH) (Brendan)
- various cleanups (Brendan)
v9:
- print warning when start or end address was rounded (Peter)
- in the reclaim code, tlbsync at context switch time (Peter)
- fix !CONFIG_CPU_SUP_AMD compile error in arch_tlbbatch_add_pending (Jan)
v8:
- round start & end to handle non-page-aligned callers (Steven & Jan)
- fix up changelog & add tested-by tags (Manali)
v7:
- a few small code cleanups (Nadav)
- fix spurious VM_WARN_ON_ONCE in mm_global_asid
- code simplifications & better barriers (Peter & Dave)
v6:
- fix info->end check in flush_tlb_kernel_range (Michael)
- disable broadcast TLB flushing on 32 bit x86
v5:
- use byte assembly for compatibility with older toolchains (Borislav, Michael)
- ensure a panic on an invalid number of extra pages (Dave, Tom)
- add cant_migrate() assertion to tlbsync (Jann)
- a bunch more cleanups (Nadav)
- key TCE enabling off X86_FEATURE_TCE (Andrew)
- fix a race between reclaim and ASID transition (Jann)
v4:
- Use only bitmaps to track free global ASIDs (Nadav)
- Improved AMD initialization (Borislav & Tom)
- Various naming and documentation improvements (Peter, Nadav, Tom, Dave)
- Fixes for subtle race conditions (Jann)
v3:
- Remove paravirt tlb_remove_table call (thank you Qi Zheng)
- More suggested cleanups and changelog fixes by Peter and Nadav
v2:
- Apply suggestions by Peter and Borislav (thank you!)
- Fix bug in arch_tlbbatch_flush, where we need to do both
the TLBSYNC, and flush the CPUs that are in the cpumask.
- Some updates to comments and changelogs based on questions.