Skip to content

x86: Concurrent TLB flushes and other improvements

Nadav Amit kvm linux-hyperv lkml virtualization xen-devel 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest9 patches 35 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/9smp: Remove smp_call_function() and on_each_cpu() return valuesarch/alpha/kernel/smp.c arch/alpha/oprofile/common.c arch/ia64/kernel/perfmon.c +6 more+24 -470 replies
2/9smp: Run functions concurrently in smp_call_function_many()include/linux/smp.h kernel/smp.c+83 -770 replies
3/9x86/mm/tlb: Refactor common code into flush_tlb_on_cpus()arch/x86/mm/tlb.c+34 -282 replies
4/9x86/mm/tlb: Flush remote and local TLBs concurrentlyarch/x86/hyperv/mmu.c arch/x86/include/asm/paravirt.h arch/x86/include/asm/paravirt_types.h +5 more+87 -1210 replies
5/9x86/mm/tlb: Optimize local TLB flushesarch/x86/mm/tlb.c+6 -24 replies
6/9KVM: x86: Provide paravirtualized flush_tlb_multi()arch/x86/kernel/kvm.c+8 -49 replies
7/9smp: Do not mark call_function_data as sharedkernel/smp.c+1 -10 replies
8/9x86/tlb: Privatize cpu_tlbstatearch/x86/include/asm/tlbflush.h arch/x86/mm/init.c arch/x86/mm/tlb.c+32 -255 replies
9/9x86/apic: Use non-atomic operations when possiblearch/x86/kernel/apic/apic_flat_64.c arch/x86/kernel/apic/x2apic_cluster.c arch/x86/kernel/smp.c+4 -42 replies

Reviews trailers per patch

1 of 9 patches has a Reviewed-by or Acked-by.

PatchDave Hansen
1/9
2/9
3/9R-b
4/9
5/9
6/9
7/9
8/9
9/9

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [PATCH 0/9] x86: Concurrent TLB flushes and other improvements

Currently, local and remote TLB flushes are not performed concurrently, which introduces unnecessary overhead - each INVLPG can take 100s of cycles. This patch-set allows TLB flushes to be run concurrently: first request the remote CPUs to initiate the flush, then run it locally, and finally wait for the remote CPUs to finish their work.

In addition, there are various small optimizations to avoid unwarranted false-sharing and atomic operations.

The proposed changes should also improve the performance of other invocations of on_each_cpu(). Hopefully, no one has relied on this behavior of on_each_cpu() that invoked functions first remotely and only then locally [Peter says he remembers someone might do so, but without further information it is hard to know how to address it].

Running sysbench on dax w/emulated-pmem, write-cache disabled, and various mitigations (PTI, Spectre, MDS) disabled on Haswell:

sysbench fileio --file-total-size=3G --file-test-mode=rndwr \
--file-io-mode=mmap --threads=4 --file-fsync-mode=fdatasync run

			events (avg/stddev)
			-------------------

5.2-rc3: 1247669.0000/16075.39
+patchset: 1290607.0000/13617.56 (+3.4%)

Patch 1 does small cleanup. Patches 2-5 implement the concurrent execution of TLB flushes. Patches 6-9 deal with false-sharing and unnecessary atomic operations. There is still no implementation that uses the concurrent TLB flushes by Xen and Hyper-V.

There are various additional possible optimizations that were sent or are in development (async flushes, x2apic shorthands, fewer mm_tlb_gen accesses, etc.), but based on Andy's feedback, they will be sent later.

RFCv2 -> v1:
* Fix comment on flush_tlb_multi [Juergen]
* Removing async invalidation optimizations [Andy]
* Adding KVM support [Paolo]

Cc Richard Henderson <[email protected]>
Cc Ivan Kokshaysky <[email protected]>
Cc Matt Turner <[email protected]>
Cc Tony Luck <[email protected]>
Cc Fenghua Yu <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Josh Poimboeuf <[email protected]>
Cc Paolo Bonzini <[email protected]>

Nadav Amit (9):
smp: Remove smp_call_function() and on_each_cpu() return values
smp: Run functions concurrently in smp_call_function_many() x86/mm/tlb: Refactor common code into flush_tlb_on_cpus() x86/mm/tlb: Flush remote and local TLBs concurrently x86/mm/tlb: Optimize local TLB flushes
KVM: x86: Provide paravirtualized flush_tlb_multi()
smp: Do not mark call_function_data as shared x86/tlb: Privatize cpu_tlbstate
x86/apic: Use non-atomic operations when possible

 arch/alpha/kernel/smp.c               |  19 +---
 arch/alpha/oprofile/common.c          |   6 +-
 arch/ia64/kernel/perfmon.c            |  12 +--
 arch/ia64/kernel/uncached.c           |   8 +-
 arch/x86/hyperv/mmu.c                 |   2 +
 arch/x86/include/asm/paravirt.h       |   8 ++
 arch/x86/include/asm/paravirt_types.h |   6 ++
 arch/x86/include/asm/tlbflush.h       |  46 ++++----
 arch/x86/kernel/apic/apic_flat_64.c   |   4 +-
 arch/x86/kernel/apic/x2apic_cluster.c |   2 +-
 arch/x86/kernel/kvm.c                 |  11 +-
 arch/x86/kernel/paravirt.c            |   3 +
 arch/x86/kernel/smp.c                 |   2 +-
 arch/x86/lib/cache-smp.c              |   3 +-
 arch/x86/mm/init.c                    |   2 +-
 arch/x86/mm/tlb.c                     | 150 ++++++++++++++++++--------
 arch/x86/xen/mmu_pv.c                 |   2 +
 drivers/char/agp/generic.c            |   3 +-
 include/linux/smp.h                   |  32 ++++--
 kernel/smp.c                          | 141 +++++++++++-------------
 kernel/up.c                           |   3 +-
 21 files changed, 272 insertions(+), 193 deletions(-)
Signature · 1 line
2.20.1