Skip to contentSkip to conversation

IPI virtualization support for VM

PATCH Superseded Zeng Guang 1 participant 1 message Jul 16, 2021 KVM

v1 of 9 Superseded Latest is v9 →

lore
Share & tools
lore

Zeng Guang

author
#0 lore raw

cover IPI virtualization support for VM

Current IPI process in guest VM will virtualize the writing to interrupt command register(ICR) of the local APIC which will cause VM-exit anyway on source vCPU. Frequent VM-exit could induce much overhead accumulated if running IPI intensive task.

IPI virtualization as a new VT-x feature targets to eliminate VM-exits when issuing IPI on source vCPU. It introduces a new VM-execution control - "IPI virtualization"(bit4) in the tertiary processor-based VM-exection controls and a new data structure - "PID-pointer table address" and "Last PID-pointer index" referenced by the VMCS. When "IPI virtualization" is enabled, processor emulateds following kind of writes to APIC registers that would send IPIs, moreover without causing VM-exits.
- Memory-mapped ICR writes
- MSR-mapped ICR writes
- SENDUIPI execution

This patch series implement IPI virtualization support in KVM.

Patches 1-4 add tertiary processor-based VM-execution support framework.

Patch 5 implement interrupt dispatch support in x2APIC mode with APIC-write VM exit. In previous platform, no CPU would produce APIC-write VM exit with exit qulification 300H when the "virtual x2APIC mode" VM-execution control was 1.

Patch 6 implement IPI virtualization related function including feature enabling through tertiary processor-based VM-execution in various scenario of VMCS configuration, PID table setup in vCPU creation and vCPU block consideration.

Document for IPI virtualization is now available at the latest "Intel Architecture Instruction Set Extensions Programming Reference".

Document Link:
https://software.intel.com/content/www/us/en/develop/download/intel-architecture-instruction-set-extensions-programming-reference.html

We did experiment to measure average time sending IPI from source vCPU to the target vCPU completing the IPI handling by kvm unittest w/ and w/o IPI virtualization. When IPI virtualizatin enabled, it will reduce 22.21% and 15.98% cycles comsuming in xAPIC mode and x2APIC mode respectly.

KMV unittest:vmexit/ipi, 2 vCPU, AP runs without halt to ensure no VM exit impact on target vCPU.

		Cycles of IPI 				
		xAPIC mode		x2APIC mode	
	test	w/o IPIv  w/ IPIv	w/o IPIv  w/ IPIv
	1	6106	  4816		4265	  3768
	2	6244	  4656		4404	  3546
	3	6165	  4658		4233	  3474
	4	5992	  4710		4363	  3430
	5	6083	  4741		4215	  3551
	6	6238	  4904		4304	  3547
	7	6164	  4617		4263	  3709
	8	5984	  4763		4518	  3779
	9	5931	  4712		4645	  3667
	10	5955	  4530		4332	  3724
	11	5897	  4673		4283	  3569
	12	6140	  4794		4178	  3598
	13	6183	  4728		4363	  3628
	14	5991	  4994		4509	  3842
	15	5866	  4665		4520	  3739
	16	6032	  4654		4229	  3701
	17	6050	  4653		4185	  3726
	18	6004	  4792		4319	  3746
	19	5961	  4626		4196	  3392
	20	6194	  4576		4433	  3760

Average cycles 6059 4713.1 4337.85 3644.8
%Reduction -22.21% -15.98%

Gao Chao (1):
KVM: VMX: enable IPI virtualization

Robert Hoo (4):
x86/feat_ctl: Add new VMX feature, Tertiary VM-Execution control
KVM: VMX: Extend BUILD_CONTROLS_SHADOW macro to support 64-bit
variation
KVM: VMX: Detect Tertiary VM-Execution control when setup VMCS config
KVM: VMX: dump_vmcs() reports tertiary_exec_control field as well

Zeng Guang (1):
KVM: x86: Support interrupt dispatch in x2APIC mode with APIC-write VM
exit

 arch/x86/include/asm/msr-index.h   |   1 +
 arch/x86/include/asm/vmx.h         |  11 +++
 arch/x86/include/asm/vmxfeatures.h |   5 +-
 arch/x86/kernel/cpu/feat_ctl.c     |   9 +++
 arch/x86/kvm/lapic.c               |   9 ++-
 arch/x86/kvm/vmx/capabilities.h    |   8 ++
 arch/x86/kvm/vmx/evmcs.c           |   2 +
 arch/x86/kvm/vmx/evmcs.h           |   1 +
 arch/x86/kvm/vmx/posted_intr.c     |  22 ++++--
 arch/x86/kvm/vmx/vmcs.h            |   1 +
 arch/x86/kvm/vmx/vmx.c             | 123 +++++++++++++++++++++++++++--
 arch/x86/kvm/vmx/vmx.h             |  30 ++++---
 12 files changed, 198 insertions(+), 24 deletions(-)
Signature · 1 line
2.17.1