Skip to content

KVM: x86: TLB flushing fixes and enhancements

Sean Christopherson kvm lkml 2 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v3 Latest37 patches 45 repliesIn review
  2. v232 patches 12 repliesSuperseded

Changes in v3 from the cover letter

- Fix freeing of roots during INVVPID, I botched things when tweaking Junaid's original patch.
- Move "last vpid02" logic into nested VMX flushing helper. [Paolo]
- Split "skip tlb flush" and "skip mmu sync" logic during fast roots switch. [Paolo]
- Unconditionally skip tlb flush during fast roots switch on nested VMX transitions, i.e. let nested_vmx_transition_tlb_flush() do the work. This avoids flushing when EPT=0 and VPID=1. [Paolo]
- Do more cr3->pgd conversions in related code that drove me bonkers when trying to figure out what was broken with VPID. I think this knocks off the last code that uses "cr3" for variables/functions that work with cr3 or eptp.

Patches in v3 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/37KVM: VMX: Flush all EPTP/VPID contexts on remote TLB flusharch/x86/kvm/vmx/vmx.h+27 -14 replies
2/37KVM: nVMX: Validate the EPTP when emulating INVEPT(EXTENT_CONTEXT)arch/x86/kvm/vmx/nested.c+5 -13 replies
3/37KVM: nVMX: Invalidate all EPTP contexts when emulating INVEPT for L1arch/x86/kvm/vmx/nested.c+4 -49 replies
4/37KVM: nVMX: Invalidate all roots when emulating INVVPID without EPTarch/x86/kvm/vmx/nested.c+14 -05 replies
5/37KVM: x86: Export kvm_propagate_fault() (as kvm_inject_emulated_page_fault)arch/x86/include/asm/kvm_host.h arch/x86/kvm/x86.c+8 -23 replies
6/37KVM: x86: Consolidate logic for injecting page faults to L1arch/x86/include/asm/kvm_host.h arch/x86/kvm/mmu/mmu.c arch/x86/kvm/mmu/paging_tmpl.h +1 more+10 -81 reply
7/37KVM: x86: Sync SPTEs when injecting page/EPT fault into L1arch/x86/kvm/vmx/nested.c arch/x86/kvm/vmx/vmx.c arch/x86/kvm/x86.c+51 -120 replies
8/37KVM: VMX: Skip global INVVPID fallback if vpid==0 in vpid_sync_context()arch/x86/kvm/vmx/ops.h+1 -11 reply
9/37KVM: VMX: Use vpid_sync_context() directly when possiblearch/x86/kvm/vmx/nested.c+6 -60 replies
10/37KVM: VMX: Move vpid_sync_vcpu_addr() down a few linesarch/x86/kvm/vmx/ops.h+13 -130 replies
11/37KVM: VMX: Handle INVVPID fallback logic in vpid_sync_vcpu_addr()arch/x86/kvm/vmx/ops.h arch/x86/kvm/vmx/vmx.c+6 -90 replies
12/37KVM: VMX: Drop redundant capability checks in low level INVVPID helpersarch/x86/kvm/vmx/ops.h arch/x86/kvm/vmx/vmx.h+2 -50 replies
13/37KVM: nVMX: Use vpid_sync_vcpu_addr() to emulate INVVPID with addressarch/x86/kvm/vmx/nested.c+1 -50 replies
14/37KVM: x86: Move "flush guest's TLB" logic to separate kvm_x86_ops hookarch/x86/include/asm/kvm_host.h arch/x86/kvm/svm.c arch/x86/kvm/vmx/vmx.c +1 more+26 -15 replies
15/37KVM: VMX: Clean up vmx_flush_tlb_gva()arch/x86/kvm/vmx/vmx.c+3 -70 replies
16/37KVM: x86: Drop @invalidate_gpa param from kvm_x86_ops' tlb_flush()arch/x86/include/asm/kvm_host.h arch/x86/kvm/mmu/mmu.c arch/x86/kvm/svm.c +3 more+24 -421 reply
17/37KVM: SVM: Wire up ->tlb_flush_guest() directly to svm_flush_tlb()arch/x86/kvm/svm.c+1 -61 reply
18/37KVM: VMX: Move vmx_flush_tlb() to vmx.carch/x86/kvm/vmx/vmx.c arch/x86/kvm/vmx/vmx.h+25 -251 reply
19/37KVM: nVMX: Move nested_get_vpid02() to vmx/nested.harch/x86/kvm/vmx/nested.c arch/x86/kvm/vmx/nested.h+7 -71 reply
20/37KVM: VMX: Introduce vmx_flush_tlb_current()arch/x86/kvm/vmx/vmx.c+18 -20 replies
21/37KVM: SVM: Document the ASID logic in svm_flush_tlb()arch/x86/kvm/svm.c+7 -00 replies
22/37KVM: x86: Rename ->tlb_flush() to ->tlb_flush_all()arch/x86/include/asm/kvm_host.h arch/x86/kvm/mmu/mmu.c arch/x86/kvm/svm.c +2 more+13 -150 replies
23/37KVM: nVMX: Add helper to handle TLB flushes on nested VM-Enter/VM-Exitarch/x86/kvm/vmx/nested.c+44 -446 replies
24/37KVM: x86: Introduce KVM_REQ_TLB_FLUSH_CURRENT to flush current ASIDarch/x86/include/asm/kvm_host.h arch/x86/kvm/svm.c arch/x86/kvm/vmx/nested.c +3 more+31 -30 replies
25/37KVM: x86/mmu: Use KVM_REQ_TLB_FLUSH_CURRENT for MMU specific flushesarch/x86/kvm/mmu/mmu.c+6 -60 replies
26/37KVM: nVMX: Selectively use TLB_FLUSH_CURRENT for nested VM-Enter/VM-Exitarch/x86/kvm/vmx/nested.c+5 -30 replies
27/37KVM: nVMX: Reload APIC access page on nested VM-Exit only if necessaryarch/x86/kvm/vmx/nested.c arch/x86/kvm/vmx/vmx.c arch/x86/kvm/vmx/vmx.h+12 -80 replies
28/37KVM: VMX: Retrieve APIC access page HPA only when necessaryarch/x86/include/asm/kvm_host.h arch/x86/kvm/vmx/vmx.c arch/x86/kvm/x86.c+16 -150 replies
29/37KVM: VMX: Don't reload APIC access page if its control is disabledarch/x86/kvm/vmx/vmx.c+13 -10 replies
30/37KVM: x86/mmu: Move fast_cr3_switch() side effects to __kvm_mmu_new_cr3()arch/x86/kvm/mmu/mmu.c+31 -380 replies
31/37KVM: x86/mmu: Add separate override for MMU sync during fast CR3 switcharch/x86/include/asm/kvm_host.h arch/x86/kvm/mmu/mmu.c arch/x86/kvm/vmx/nested.c +1 more+11 -91 reply
32/37KVM: x86/mmu: Add module param to force TLB flush on root reusearch/x86/kvm/mmu/mmu.c+5 -20 replies
33/37KVM: nVMX: Skip MMU sync on nested VMX transition when possiblearch/x86/kvm/mmu/mmu.c arch/x86/kvm/vmx/nested.c+44 -21 reply
34/37KVM: nVMX: Don't flush TLB on nested VMX transitionarch/x86/kvm/mmu/mmu.c arch/x86/kvm/vmx/nested.c+5 -32 replies
35/37KVM: nVMX: Free only the affected contexts when emulating INVEPTarch/x86/kvm/vmx/nested.c+21 -50 replies
36/37KVM: x86: Replace "cr3" with "pgd" in "new cr3/pgd" related codearch/x86/include/asm/kvm_host.h arch/x86/kvm/mmu/mmu.c arch/x86/kvm/vmx/nested.c +2 more+39 -390 replies
37/37KVM: VMX: Clean cr3/pgd handling in vmx_load_mmu_pgd()arch/x86/kvm/vmx/vmx.c+4 -30 replies

Reviews trailers per patch

16 of 37 patches have a Reviewed-by or Acked-by.

PatchLiran AlonVitaly KuznetsovPaolo BonziniMiaohe Lin
1/37R-b
2/37R-b
3/37R-b
4/37R-b
5/37R-b
6/37
7/37
8/37R-b
9/37R-b R-b R-b
10/37R-b R-b R-b
11/37R-b R-b R-b
12/37
13/37R-b R-b
14/37R-b
15/37R-b R-b
16/37R-b
17/37R-b
18/37R-b
19/37R-b
20/37
21/37
22/37
23/37
24/37
25/37
26/37
27/37
28/37
29/37
30/37
31/37
32/37
33/37
34/37
35/37
36/37
37/37

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 4 reviewers.

Read the v3 thread

Cover letter [PATCH v3 00/37] KVM: x86: TLB flushing fixes and enhancements

VMX TLB flushing cleanup series to fix a variety of bugs, and to avoid unnecessary TLB flushes on nested VMX transitions.

1) Nested VMX doesn't properly flush all ASIDs/contexts on system events, e.g. on mmu_notifier invalidate all contexts for L1 *and* L2 need to be invalidated, but KVM generally only flushes L1 or L2 (or just L1).

2) #1 is largely benign because nested VMX always flushes the new context on nested VM-Entry/VM-Exit.

High level overview:

a) Fix the main TLB flushing bug with a big hammer.

b) Fix a few other flushing related bugs.

c) Clean up vmx_tlb_flush(), i.e. what was v1 of this series.

d) Reintroduce current-ASID/context flushing to regain some of the precision that got blasted away by the big hammer in #1.

e) Fix random code paths that unnecessarily trigger TLB flushes on nested VMX transitions.

f) Stop flushing on every nested VMX transition.

g) Extra cleanup.

v3:
- Fix freeing of roots during INVVPID, I botched things when tweaking Junaid's original patch.
- Move "last vpid02" logic into nested VMX flushing helper. [Paolo]
- Split "skip tlb flush" and "skip mmu sync" logic during fast roots switch. [Paolo]
- Unconditionally skip tlb flush during fast roots switch on nested VMX transitions, i.e. let nested_vmx_transition_tlb_flush() do the work. This avoids flushing when EPT=0 and VPID=1. [Paolo]
- Do more cr3->pgd conversions in related code that drove me bonkers when trying to figure out what was broken with VPID. I think this knocks off the last code that uses "cr3" for variables/functions that work with cr3 or eptp.

v2:
- Basically a new series.

v1:
- https://patchwork.kernel.org/cover/11394987/

Junaid Shahid (2):
KVM: nVMX: Invalidate all roots when emulating INVVPID without EPT
KVM: x86: Sync SPTEs when injecting page/EPT fault into L1

Sean Christopherson (35):
KVM: VMX: Flush all EPTP/VPID contexts on remote TLB flush
KVM: nVMX: Validate the EPTP when emulating INVEPT(EXTENT_CONTEXT)
KVM: nVMX: Invalidate all EPTP contexts when emulating INVEPT for L1
KVM: x86: Export kvm_propagate_fault() (as
kvm_inject_emulated_page_fault)
KVM: x86: Consolidate logic for injecting page faults to L1
KVM: VMX: Skip global INVVPID fallback if vpid==0 in
vpid_sync_context()
KVM: VMX: Use vpid_sync_context() directly when possible
KVM: VMX: Move vpid_sync_vcpu_addr() down a few lines
KVM: VMX: Handle INVVPID fallback logic in vpid_sync_vcpu_addr()
KVM: VMX: Drop redundant capability checks in low level INVVPID
helpers
KVM: nVMX: Use vpid_sync_vcpu_addr() to emulate INVVPID with address
KVM: x86: Move "flush guest's TLB" logic to separate kvm_x86_ops hook
KVM: VMX: Clean up vmx_flush_tlb_gva()
KVM: x86: Drop @invalidate_gpa param from kvm_x86_ops' tlb_flush()
KVM: SVM: Wire up ->tlb_flush_guest() directly to svm_flush_tlb()
KVM: VMX: Move vmx_flush_tlb() to vmx.c
KVM: nVMX: Move nested_get_vpid02() to vmx/nested.h
KVM: VMX: Introduce vmx_flush_tlb_current()
KVM: SVM: Document the ASID logic in svm_flush_tlb()
KVM: x86: Rename ->tlb_flush() to ->tlb_flush_all()
KVM: nVMX: Add helper to handle TLB flushes on nested VM-Enter/VM-Exit
KVM: x86: Introduce KVM_REQ_TLB_FLUSH_CURRENT to flush current ASID
KVM: x86/mmu: Use KVM_REQ_TLB_FLUSH_CURRENT for MMU specific flushes
KVM: nVMX: Selectively use TLB_FLUSH_CURRENT for nested
VM-Enter/VM-Exit
KVM: nVMX: Reload APIC access page on nested VM-Exit only if necessary
KVM: VMX: Retrieve APIC access page HPA only when necessary
KVM: VMX: Don't reload APIC access page if its control is disabled
KVM: x86/mmu: Move fast_cr3_switch() side effects to

    __kvm_mmu_new_cr3()

KVM: x86/mmu: Add separate override for MMU sync during fast CR3
switch
KVM: x86/mmu: Add module param to force TLB flush on root reuse
KVM: nVMX: Skip MMU sync on nested VMX transition when possible
KVM: nVMX: Don't flush TLB on nested VMX transition
KVM: nVMX: Free only the affected contexts when emulating INVEPT
KVM: x86: Replace "cr3" with "pgd" in "new cr3/pgd" related code
KVM: VMX: Clean cr3/pgd handling in vmx_load_mmu_pgd()

 arch/x86/include/asm/kvm_host.h |  25 +++-
 arch/x86/kvm/mmu/mmu.c          | 145 +++++++++----------
 arch/x86/kvm/mmu/paging_tmpl.h  |   2 +-
 arch/x86/kvm/svm.c              |  19 ++-
 arch/x86/kvm/vmx/nested.c       | 249 ++++++++++++++++++++++----------
 arch/x86/kvm/vmx/nested.h       |   7 +
 arch/x86/kvm/vmx/ops.h          |  32 ++--
 arch/x86/kvm/vmx/vmx.c          | 119 ++++++++++++---
 arch/x86/kvm/vmx/vmx.h          |  19 +--
 arch/x86/kvm/x86.c              |  67 ++++++---
 arch/x86/kvm/x86.h              |   6 +
 11 files changed, 448 insertions(+), 242 deletions(-)
Signature · 1 line
2.24.1