Skip to contentSkip to conversation

KVM monolithic v2

PATCH In review Andrea Arcangeli 3 participants 30 messages Sep 28, 2019 – Oct 16, 2019

Subsystems (5)
x86 KVM perf events KVM x86 +1 more subsystems
lore
Share & tools
lore
#0 lore raw

cover KVM monolithic v2

Hello,

as usual the last 4 patches could be splitted off but I did more measurements in that area and I altered the commit headers. They're fairly small commits compared to the previous part so I kept it considering they're needed to benchmark the previous part.

The KVM monolithic enhancement is easy to identify checking the word "monolithic" in the subject so there's no confusion about where that work stops.

This renames all functions in place mixed up in whatever location they existed in svm.c or vmx.c. If they require an inline call they're defined now as extern before the kvm_x86_ops structure.

Converting those small kvm_x86 functions to inlines requires more Makefile work and header restructuring so it's left for later.

The removal of kvm_x86_ops is also left for later because that requires lots of logic changes in the code scattered all over the place in KVM code. This patchset tries to do the conversion with as few logic changes as possible, so the code works the same and this only improves the implementation and the performance.

Doing the conversion plus the logic changes at the same time would pose the risk of not being able to identify through bisection if any regression is caused by a bug in the conversion or in one small commit to alter the logic and to remove the need of one more pointer to function.

It's best if each single removal of any pointer to functional change is done through a separate small commit. After all those small commits done incrementally with this patchset, the kvm_x86_ops structure can be deleted.

https://git.kernel.org/pub/scm/linux/kernel/git/andrea/aa.git/log/?h=kvm-mono2

Thanks,
Andrea

Andrea Arcangeli (14):
KVM: monolithic: x86: remove kvm.ko
KVM: monolithic: x86: disable linking vmx and svm at the same time
into the kernel
KVM: monolithic: x86: convert the kvm_x86_ops and kvm_pmu_ops methods
to external functions
KVM: monolithic: x86: handle the request_immediate_exit variation
KVM: monolithic: add more section prefixes in the KVM common code
KVM: monolithic: x86: remove __exit section prefix from
machine_unsetup
KVM: monolithic: x86: remove __init section prefix from

    kvm_x86_cpu_has_kvm_support

KVM: monolithic: x86: remove exports
KVM: monolithic: remove exports from KVM common code
KVM: monolithic: x86: drop the kvm_pmu_ops structure
KVM: x86: optimize more exit handlers in vmx.c
KVM: retpolines: x86: eliminate retpoline from vmx.c exit handlers
KVM: retpolines: x86: eliminate retpoline from svm.c exit handlers
x86: retpolines: eliminate retpoline from msr event handlers

 arch/x86/events/intel/core.c    |  11 +
 arch/x86/include/asm/kvm_host.h | 205 +++++++-
 arch/x86/kvm/Kconfig            |  24 +-
 arch/x86/kvm/Makefile           |   5 +-
 arch/x86/kvm/cpuid.c            |  27 +-
 arch/x86/kvm/hyperv.c           |   8 +-
 arch/x86/kvm/irq.c              |   4 -
 arch/x86/kvm/irq_comm.c         |   2 -
 arch/x86/kvm/kvm_cache_regs.h   |  10 +-
 arch/x86/kvm/lapic.c            |  46 +-
 arch/x86/kvm/mmu.c              |  50 +-
 arch/x86/kvm/mmu.h              |   4 +-
 arch/x86/kvm/mtrr.c             |   2 -
 arch/x86/kvm/pmu.c              |  27 +-
 arch/x86/kvm/pmu.h              |  37 +-
 arch/x86/kvm/pmu_amd.c          |  43 +-
 arch/x86/kvm/svm.c              | 682 ++++++++++++++++-----------
 arch/x86/kvm/trace.h            |   4 +-
 arch/x86/kvm/vmx/nested.c       |  84 ++--
 arch/x86/kvm/vmx/pmu_intel.c    |  46 +-
 arch/x86/kvm/vmx/vmx.c          | 795 ++++++++++++++++++--------------
 arch/x86/kvm/vmx/vmx.h          |  39 +-
 arch/x86/kvm/x86.c              | 418 +++++++----------
 arch/x86/kvm/x86.h              |   2 +-
 include/linux/kvm_host.h        |   4 +-
 virt/kvm/eventfd.c              |   1 -
 virt/kvm/kvm_main.c             |  71 +--
 27 files changed, 1413 insertions(+), 1238 deletions(-)
#1 lore raw

PATCH 1/14 KVM: monolithic: x86: remove kvm.ko

This is the first commit of a patch series that aims to replace the modular kvm.ko kernel module with a monolithic kvm-intel/kvm-amd model. This change has the only possible cons of wasting some disk space in /lib/modules/. The pros are that it saves CPUS and some minor RAM which are more scarse resources than disk space.

The pointer to function virtual template model cannot provide any runtime benefit because kvm-intel and kvm-amd can't be loaded at the same time.

This removes kvm.ko and it links and duplicates all kvm.ko objects to both kvm-amd and kvm-intel.

---
 arch/x86/kvm/Makefile | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
arch/x86/kvm/Makefile +2 -3
index 31ecf7a76d5a..68b81f381369 100644
@@ -12,9 +12,8 @@ kvm-y			+= x86.o mmu.o emulate.o i8259.o irq.o lapic.o \
 			   i8254.o ioapic.o irq_comm.o cpuid.o pmu.o mtrr.o \
 			   hyperv.o page_track.o debugfs.o
 
-kvm-intel-y		+= vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o vmx/evmcs.o vmx/nested.o
-kvm-amd-y		+= svm.o pmu_amd.o
+kvm-intel-y		+= vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o vmx/evmcs.o vmx/nested.o $(kvm-y)
+kvm-amd-y		+= svm.o pmu_amd.o $(kvm-y)
 
-obj-$(CONFIG_KVM)	+= kvm.o
 obj-$(CONFIG_KVM_INTEL)	+= kvm-intel.o
 obj-$(CONFIG_KVM_AMD)	+= kvm-amd.o
#2 lore raw

Re: [PATCH 01/14] KVM: monolithic: x86: remove kvm.ko

Quoted 6 lines · Andrea Arcangeli

This is the first commit of a patch series that aims to replace the modular kvm.ko kernel module with a monolithic kvm-intel/kvm-amd model. This change has the only possible cons of wasting some disk space in /lib/modules/. The pros are that it saves CPUS and some minor RAM which are more scarse resources than disk space.

Quoted 6 lines

The pointer to function virtual template model cannot provide any runtime benefit because kvm-intel and kvm-amd can't be loaded at the same time.

This removes kvm.ko and it links and duplicates all kvm.ko objects to both kvm-amd and kvm-intel.

The KVM config option should be changed to a bool and its help text updated. Maybe something similar to the help for VIRTUALIZATION to make it clear that enabling KVM on its own does nothing.

Quoted 22 lines
---
 arch/x86/kvm/Makefile | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
arch/x86/kvm/Makefile +2 -3
index 31ecf7a76d5a..68b81f381369 100644
@@ -12,9 +12,8 @@ kvm-y			+= x86.o mmu.o emulate.o i8259.o irq.o lapic.o \
 			   i8254.o ioapic.o irq_comm.o cpuid.o pmu.o mtrr.o \
 			   hyperv.o page_track.o debugfs.o
 
-kvm-intel-y		+= vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o vmx/evmcs.o vmx/nested.o
-kvm-amd-y		+= svm.o pmu_amd.o
+kvm-intel-y		+= vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o vmx/evmcs.o vmx/nested.o $(kvm-y)
+kvm-amd-y		+= svm.o pmu_amd.o $(kvm-y)
 
-obj-$(CONFIG_KVM)	+= kvm.o
 obj-$(CONFIG_KVM_INTEL)	+= kvm-intel.o
 obj-$(CONFIG_KVM_AMD)	+= kvm-amd.o
#3 lore raw

Re: [PATCH 01/14] KVM: monolithic: x86: remove kvm.ko

Quoted 11 lines · Sean Christopherson
Quoted 10 lines · Andrea Arcangeli

This is the first commit of a patch series that aims to replace the modular kvm.ko kernel module with a monolithic kvm-intel/kvm-amd model. This change has the only possible cons of wasting some disk space in /lib/modules/. The pros are that it saves CPUS and some minor RAM which are more scarse resources than disk space.

The pointer to function virtual template model cannot provide any runtime benefit because kvm-intel and kvm-amd can't be loaded at the same time.

Quoted 6 lines
Quoted 2 lines

This removes kvm.ko and it links and duplicates all kvm.ko objects to both kvm-amd and kvm-intel.

The KVM config option should be changed to a bool and its help text updated. Maybe something similar to the help for VIRTUALIZATION to make it clear that enabling KVM on its own does nothing.

Making KVM a bool doesn't work well, keeping it a tristate and keying off KVM=y to force Intel or AMD (as done in the next patch) looks like the cleanest implementation.

The help text should still be updated though.

Quoted 22 lines
Quoted 22 lines
---
 arch/x86/kvm/Makefile | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
arch/x86/kvm/Makefile +2 -3
index 31ecf7a76d5a..68b81f381369 100644
@@ -12,9 +12,8 @@ kvm-y			+= x86.o mmu.o emulate.o i8259.o irq.o lapic.o \
 			   i8254.o ioapic.o irq_comm.o cpuid.o pmu.o mtrr.o \
 			   hyperv.o page_track.o debugfs.o
 
-kvm-intel-y		+= vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o vmx/evmcs.o vmx/nested.o
-kvm-amd-y		+= svm.o pmu_amd.o
+kvm-intel-y		+= vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o vmx/evmcs.o vmx/nested.o $(kvm-y)
+kvm-amd-y		+= svm.o pmu_amd.o $(kvm-y)
 
-obj-$(CONFIG_KVM)	+= kvm.o
 obj-$(CONFIG_KVM_INTEL)	+= kvm-intel.o
 obj-$(CONFIG_KVM_AMD)	+= kvm-amd.o
maintainer
#4 lore raw

Re: [PATCH 01/14] KVM: monolithic: x86: remove kvm.ko

Quoted 6 lines · Sean Christopherson
Quoted 3 lines

The KVM config option should be changed to a bool and its help text updated. Maybe something similar to the help for VIRTUALIZATION to make it clear that enabling KVM on its own does nothing.

Making KVM a bool doesn't work well, keeping it a tristate and keying off KVM=y to force Intel or AMD (as done in the next patch) looks like the cleanest implementation.

Indeed, keeping the KVM option as tristate helps showing the right suboptions, similar to what Andrea did in patch 2. However, this patch already breaks the CONFIG_KVM_INTEL=y && CONFIG_KVM_AMD=y case I think, so it should be squashed with "KVM: monolithic: x86: disable linking vmx and svm at the same time into the kernel".

Quoted 1 line

The help text should still be updated though.

The patch doesn't change the fact that enabling KVM on its own does nothing, so the help text can be updated independently (patch welcome :)).

Thanks,

Paolo

#5 lore raw

PATCH 2/14 KVM: monolithic: x86: disable linking vmx and svm at the same time into the kernel

In reply to Andrea Arcangeli

Linking both vmx and svm into the kernel at the same time isn't possible anymore or the kvm_x86/kvm_x86_pmu external function names would collide.

---
 arch/x86/kvm/Kconfig | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
arch/x86/kvm/Kconfig +22 -2
index 840e12583b85..e1601c54355e 100644
@@ -59,9 +59,29 @@ config KVM
 
 	  If unsure, say N.
 
+if KVM=y
+
+choice
+	prompt "To link KVM statically into the kernel you need to choose"
+	help
+	  In order to build a kernel with support for both AMD and Intel
+	  CPUs, you need to set CONFIG_KVM=m.
+
+config KVM_AMD_STATIC
+	select KVM_AMD
+	bool "Link KVM AMD statically into the kernel"
+
+config KVM_INTEL_STATIC
+	select KVM_INTEL
+	bool "Link KVM Intel statically into the kernel"
+
+endchoice
+
+endif
+
 config KVM_INTEL
 	tristate "KVM for Intel processors support"
-	depends on KVM
+	depends on (KVM && !KVM_AMD_STATIC) || KVM_INTEL_STATIC
 	# for perf_guest_get_msrs():
 	depends on CPU_SUP_INTEL
 	---help---
@@ -73,7 +93,7 @@ config KVM_INTEL
 
 config KVM_AMD
 	tristate "KVM for AMD processors support"
-	depends on KVM
+	depends on (KVM && !KVM_INTEL_STATIC) || KVM_AMD_STATIC
 	---help---
 	  Provides support for KVM on AMD processors equipped with the AMD-V
 	  (SVM) extensions.
#6 lore raw

Re: [PATCH 02/14] KVM: monolithic: x86: disable linking vmx and svm at the same time into the kernel

Quoted 15 lines · Andrea Arcangeli

Linking both vmx and svm into the kernel at the same time isn't possible anymore or the kvm_x86/kvm_x86_pmu external function names would collide.

---
 arch/x86/kvm/Kconfig | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
arch/x86/kvm/Kconfig +0 -0
index 840e12583b85..e1601c54355e 100644

arch/x86/kvm/Kconfig config KVM · line 62

Quoted 4 lines
arch/x86/kvm/Kconfig +1 -0
@@ -59,9 +59,29 @@ config KVM
 
 	  If unsure, say N.
 
+if KVM=y

Hmm, I see why the previous patch left KVM as a tristate. I tried a variety of hacks to let KVM be a bool but nothing worked.

Quoted 4 lines
arch/x86/kvm/Kconfig +4 -0
+
+choice
+	prompt "To link KVM statically into the kernel you need to choose"
+	help

arch/x86/kvm/Kconfig config KVM · line 76

Quoted 10 lines
arch/x86/kvm/Kconfig +10 -0
+	  In order to build a kernel with support for both AMD and Intel
+	  CPUs, you need to set CONFIG_KVM=m.
+
+config KVM_AMD_STATIC
+	select KVM_AMD
+	bool "Link KVM AMD statically into the kernel"
+
+config KVM_INTEL_STATIC
+	select KVM_INTEL
+	bool "Link KVM Intel statically into the kernel"

The prompt and choice text is way too long, e.g. in my usual window it cuts off at:

To link KVM statically into the kernel you need to choose (Link KVM Intel statically into

Without the full text (the -> at the end), it's not obvious it's an option menu (AMD was selected by default for me and it took me a second to figure out what to hit enter on).

I think short and sweet is enough for the prompt, with the details of how build both buried in the help text.

choice
prompt "KVM built-in support"
help
Here be a long and detailed help text.

config KVM_AMD_STATIC
select KVM_AMD
bool "KVM AMD"

config KVM_INTEL_STATIC
select KVM_INTEL
bool "KVM Intel"

endchoice

The ends up looking like:

<*> Kernel-based Virtual Machine (KVM) support

           KVM built-in support (KVM Intel)  --->

-*- KVM for Intel processors support

Quoted 21 lines
arch/x86/kvm/Kconfig +7 -2
+
+endchoice
+
+endif
+
 config KVM_INTEL
 	tristate "KVM for Intel processors support"
-	depends on KVM
+	depends on (KVM && !KVM_AMD_STATIC) || KVM_INTEL_STATIC
 	# for perf_guest_get_msrs():
 	depends on CPU_SUP_INTEL
 	---help---
@@ -73,7 +93,7 @@ config KVM_INTEL
 
 config KVM_AMD
 	tristate "KVM for AMD processors support"
-	depends on KVM
+	depends on (KVM && !KVM_INTEL_STATIC) || KVM_AMD_STATIC
 	---help---
 	  Provides support for KVM on AMD processors equipped with the AMD-V
 	  (SVM) extensions.
maintainer
#7 lore raw

Re: [PATCH 02/14] KVM: monolithic: x86: disable linking vmx and svm at the same time into the kernel

Quoted 8 lines · Sean Christopherson

I think short and sweet is enough for the prompt, with the details of how build both buried in the help text.

choice
prompt "KVM built-in support"
help
Here be a long and detailed help text.

Quoted 7 lines

config KVM_AMD_STATIC
select KVM_AMD
bool "KVM AMD"

config KVM_INTEL_STATIC
select KVM_INTEL
bool "KVM Intel"

Or even just

	bool "AMD"
	...
	bool "Intel"
Quoted 7 lines

endchoice

The ends up looking like:

<*> Kernel-based Virtual Machine (KVM) support

           KVM built-in support (KVM Intel)  --->

-*- KVM for Intel processors support

On top of this, it's also nice to hide the KVM_INTEL/KVM_AMD prompts if linking statically. You can achieve that with

config KVM_INTEL
tristate
prompt "KVM for Intel processors support" if KVM=m depends on (KVM=m && m) || KVM_INTEL_STATIC

config KVM_AMD
tristate
prompt "KVM for AMD processors support" if KVM=m depends on (KVM=m && m) || KVM_AMD_STATIC

The left side of the "||" ensures that, if KVM=m, you can only choose module build for both KVM_INTEL and KVM_AMD. Having just "depends on KVM" would allow a pre-existing .config to choose the now-invalid combination

CONFIG_KVM=y
CONFIG_KVM_INTEL=y
CONFIG_KVM_AMD=y

The right side of the "||" part is just for documentation, to avoid that a selected symbol does not satisfy its dependencies.

Thanks,

Paolo

#8 lore raw

Re: [PATCH 02/14] KVM: monolithic: x86: disable linking vmx and svm at the same time into the kernel

Quoted 17 lines · Paolo Bonzini
Quoted 8 lines · Sean Christopherson

I think short and sweet is enough for the prompt, with the details of how build both buried in the help text.

choice
prompt "KVM built-in support"
help
Here be a long and detailed help text.

Quoted 7 lines

config KVM_AMD_STATIC
select KVM_AMD
bool "KVM AMD"

config KVM_INTEL_STATIC
select KVM_INTEL
bool "KVM Intel"

Quoted 5 lines

Or even just

	bool "AMD"
	...
	bool "Intel"

Ya.

Quoted 8 lines
Quoted 7 lines

endchoice

The ends up looking like:

<*> Kernel-based Virtual Machine (KVM) support

           KVM built-in support (KVM Intel)  --->

-*- KVM for Intel processors support

Quoted 6 lines

On top of this, it's also nice to hide the KVM_INTEL/KVM_AMD prompts if linking statically. You can achieve that with

config KVM_INTEL
tristate
prompt "KVM for Intel processors support" if KVM=m

That's painfully obvious now that I see it. I always forget about putting conditionals at the end...

Quoted 22 lines
    depends on (KVM=m && m) || KVM_INTEL_STATIC

config KVM_AMD
tristate
prompt "KVM for AMD processors support" if KVM=m depends on (KVM=m && m) || KVM_AMD_STATIC

The left side of the "||" ensures that, if KVM=m, you can only choose module build for both KVM_INTEL and KVM_AMD. Having just "depends on KVM" would allow a pre-existing .config to choose the now-invalid combination

CONFIG_KVM=y
CONFIG_KVM_INTEL=y
CONFIG_KVM_AMD=y

The right side of the "||" part is just for documentation, to avoid that a selected symbol does not satisfy its dependencies.

Thanks,

Paolo

#9 lore raw

PATCH 3/14 KVM: monolithic: x86: convert the kvm_x86_ops and kvm_pmu_ops methods to external functions

In reply to Andrea Arcangeli

This replaces all kvm_x86_ops and kvm_pmu_ops pointer to functions with regular external functions that don't require indirect calls.

In practice this optimization results in a double digit percent reduction in the vmexit latency with the default retpoline spectre v2 mitigation enabled in the host.

When the host is booted with spectre_v2=off this still results in a best case measurable improvement up to 2% that varies wildly depending on the CPU vendor implementation under high frequency timer guest workloads. Supposedly guest userland workloads making lots of use of the BTB may benefit too.

To reduce the rejecting parts while tracking upstream, this doesn't attempt to entirely remove the kvm_x86_ops structure yet, that is meant for a later cleanup. The pmu ops have been already cleaned up in this patchset because it was left completely unused right after the conversion from pointer to functions to external functions.

Further incremental minor optimizations that weren't possible before are now enabled by the monolithic model. For example it is now possible to convert some of the small kvm_x86_* external methods to inline functions. However that will require more Makefile tweaks and so it is left for later.

This is a list of the most common retpolines executed in KVM on VMX under a guest workload triggering a high resolution timer SIGALRM flood before the monolithic KVM patchset is applied.

[..]
@[

    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    cancel_hv_timer.isra.46+44
    restart_apic_timer+295
    kvm_set_msr_common+1435
    vmx_set_msr+478
    handle_wrmsr+85
    vcpu_enter_guest+772
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 65382
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    vcpu_enter_guest+1646
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 66164
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    kvm_read_l1_tsc+41
    __kvm_wait_lapic_expire+60
    vmx_vcpu_run.part.88+1091
    vcpu_enter_guest+423
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 66199
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    vcpu_enter_guest+4958
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 66227
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    restart_apic_timer+99
    kvm_set_msr_common+1435
    vmx_set_msr+478
    handle_wrmsr+85
    vcpu_enter_guest+772
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 130619
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    kvm_read_l1_tsc+41
    vmx_set_hv_timer+81
    restart_apic_timer+99
    kvm_set_msr_common+1435
    vmx_set_msr+478
    handle_wrmsr+85
    vcpu_enter_guest+772
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 130665
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    kvm_skip_emulated_instruction+49
    handle_wrmsr+102
    vcpu_enter_guest+772
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 131020
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    kvm_skip_emulated_instruction+82
    handle_wrmsr+102
    vcpu_enter_guest+772
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 131025
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    handle_wrmsr+85
    vcpu_enter_guest+772
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 131043
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    skip_emulated_instruction+48
    kvm_skip_emulated_instruction+82
    handle_wrmsr+102
    vcpu_enter_guest+772
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 131046
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    vcpu_enter_guest+4009
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 132405
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rcx+33
    vcpu_enter_guest+1689
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 197697
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    vmx_vcpu_run.part.88+358
    vcpu_enter_guest+423
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 198736
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    vcpu_enter_guest+575
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 198771
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    vcpu_enter_guest+423
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 198793
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    vcpu_enter_guest+486
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 198801
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    vcpu_enter_guest+168
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 198848
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    vcpu_enter_guest+772
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68

]: 397680

@total: 3816655

Here the same but on SVM:

[..]
@[

    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    clockevents_program_event+148
    hrtimer_start_range_ns+528
    start_sw_timer+356
    restart_apic_timer+111
    kvm_set_msr_common+1435
    msr_interception+138
    vcpu_enter_guest+772
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 36031
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    lapic_next_event+28
    clockevents_program_event+148
    hrtimer_start_range_ns+528
    start_sw_timer+356
    restart_apic_timer+111
    kvm_set_msr_common+1435
    msr_interception+138
    vcpu_enter_guest+772
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 36063
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    ktime_get+58
    clockevents_program_event+84
    hrtimer_try_to_cancel+168
    hrtimer_cancel+21
    kvm_set_lapic_tscdeadline_msr+43
    kvm_set_msr_common+1435
    msr_interception+138
    vcpu_enter_guest+772
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 36134
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    lapic_next_event+28
    clockevents_program_event+148
    hrtimer_try_to_cancel+168
    hrtimer_cancel+21
    kvm_set_lapic_tscdeadline_msr+43
    kvm_set_msr_common+1435
    msr_interception+138
    vcpu_enter_guest+772
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 36146
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    clockevents_program_event+148
    hrtimer_try_to_cancel+168
    hrtimer_cancel+21
    kvm_set_lapic_tscdeadline_msr+43
    kvm_set_msr_common+1435
    msr_interception+138
    vcpu_enter_guest+772
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 36190
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    ktime_get+58
    clockevents_program_event+84
    hrtimer_start_range_ns+528
    start_sw_timer+356
    restart_apic_timer+111
    kvm_set_msr_common+1435
    msr_interception+138
    vcpu_enter_guest+772
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 36281
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    vcpu_enter_guest+1646
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 37752
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    kvm_read_l1_tsc+41
    __kvm_wait_lapic_expire+60
    svm_vcpu_run+1276
]: 37886
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    vcpu_enter_guest+4958
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 37957
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    kvm_read_l1_tsc+41
    start_sw_timer+302
    restart_apic_timer+111
    kvm_set_msr_common+1435
    msr_interception+138
    vcpu_enter_guest+772
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 74358
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    ktime_get+58
    start_sw_timer+279
    restart_apic_timer+111
    kvm_set_msr_common+1435
    msr_interception+138
    vcpu_enter_guest+772
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 74558
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    kvm_skip_emulated_instruction+82
    msr_interception+356
    vcpu_enter_guest+772
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 74713
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    kvm_skip_emulated_instruction+49
    msr_interception+356
    vcpu_enter_guest+772
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 74757
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    msr_interception+138
    vcpu_enter_guest+772
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 74795
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    kvm_get_rflags+28
    svm_interrupt_allowed+50
    vcpu_enter_guest+4009
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 75647
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    vcpu_enter_guest+4009
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 75812
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rcx+33
    vcpu_enter_guest+1689
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 112579
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    vcpu_enter_guest+575
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 113371
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    vcpu_enter_guest+423
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 113386
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    vcpu_enter_guest+486
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 113414
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    vcpu_enter_guest+168
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 113601
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    vcpu_enter_guest+772
    kvm_arch_vcpu_ioctl_run+263
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68

]: 227076

@total: 3829460

---
 arch/x86/include/asm/kvm_host.h | 200 ++++++++-
 arch/x86/kvm/cpuid.c            |  22 +-
 arch/x86/kvm/hyperv.c           |   6 +-
 arch/x86/kvm/kvm_cache_regs.h   |  10 +-
 arch/x86/kvm/lapic.c            |  30 +-
 arch/x86/kvm/mmu.c              |  26 +-
 arch/x86/kvm/mmu.h              |   4 +-
 arch/x86/kvm/pmu.c              |  24 +-
 arch/x86/kvm/pmu.h              |  19 +-
 arch/x86/kvm/pmu_amd.c          |  52 +--
 arch/x86/kvm/svm.c              | 659 ++++++++++++++++------------
 arch/x86/kvm/trace.h            |   4 +-
 arch/x86/kvm/vmx/nested.c       |  84 ++--
 arch/x86/kvm/vmx/pmu_intel.c    |  55 ++-
 arch/x86/kvm/vmx/vmx.c          | 732 ++++++++++++++++++--------------
 arch/x86/kvm/vmx/vmx.h          |  39 +-
 arch/x86/kvm/x86.c              | 308 +++++++-------
 arch/x86/kvm/x86.h              |   2 +-
 18 files changed, 1345 insertions(+), 931 deletions(-)
arch/x86/include/asm/kvm_host.h +195 -5
index a3a3ec73fa2f..47eeb92d4b4a 100644
@@ -1002,6 +1002,196 @@ struct kvm_lapic_irq {
 	bool msi_redir_hint;
 };
 
+extern __init int kvm_x86_cpu_has_kvm_support(void);
+extern __init int kvm_x86_disabled_by_bios(void);
+extern int kvm_x86_hardware_enable(void);
+extern void kvm_x86_hardware_disable(void);
+extern __init int kvm_x86_check_processor_compatibility(void);
+extern __init int kvm_x86_hardware_setup(void);
+extern __exit void kvm_x86_hardware_unsetup(void);
+extern bool kvm_x86_cpu_has_accelerated_tpr(void);
+extern bool kvm_x86_has_emulated_msr(int index);
+extern void kvm_x86_cpuid_update(struct kvm_vcpu *vcpu);
+extern struct kvm *kvm_x86_vm_alloc(void);
+extern void kvm_x86_vm_free(struct kvm *kvm);
+extern int kvm_x86_vm_init(struct kvm *kvm);
+extern void kvm_x86_vm_destroy(struct kvm *kvm);
+/* Create, but do not attach this VCPU */
+extern struct kvm_vcpu *kvm_x86_vcpu_create(struct kvm *kvm, unsigned id);
+extern void kvm_x86_vcpu_free(struct kvm_vcpu *vcpu);
+extern void kvm_x86_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event);
+extern void kvm_x86_prepare_guest_switch(struct kvm_vcpu *vcpu);
+extern void kvm_x86_vcpu_load(struct kvm_vcpu *vcpu, int cpu);
+extern void kvm_x86_vcpu_put(struct kvm_vcpu *vcpu);
+extern void kvm_x86_update_bp_intercept(struct kvm_vcpu *vcpu);
+extern int kvm_x86_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr);
+extern int kvm_x86_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr);
+extern u64 kvm_x86_get_segment_base(struct kvm_vcpu *vcpu, int seg);
+extern void kvm_x86_get_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var,
+				int seg);
+extern int kvm_x86_get_cpl(struct kvm_vcpu *vcpu);
+extern void kvm_x86_set_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var,
+				int seg);
+extern void kvm_x86_get_cs_db_l_bits(struct kvm_vcpu *vcpu, int *db, int *l);
+extern void kvm_x86_decache_cr0_guest_bits(struct kvm_vcpu *vcpu);
+extern void kvm_x86_decache_cr3(struct kvm_vcpu *vcpu);
+extern void kvm_x86_decache_cr4_guest_bits(struct kvm_vcpu *vcpu);
+extern void kvm_x86_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0);
+extern void kvm_x86_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3);
+extern int kvm_x86_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4);
+extern void kvm_x86_set_efer(struct kvm_vcpu *vcpu, u64 efer);
+extern void kvm_x86_get_idt(struct kvm_vcpu *vcpu, struct desc_ptr *dt);
+extern void kvm_x86_set_idt(struct kvm_vcpu *vcpu, struct desc_ptr *dt);
+extern void kvm_x86_get_gdt(struct kvm_vcpu *vcpu, struct desc_ptr *dt);
+extern void kvm_x86_set_gdt(struct kvm_vcpu *vcpu, struct desc_ptr *dt);
+extern u64 kvm_x86_get_dr6(struct kvm_vcpu *vcpu);
+extern void kvm_x86_set_dr6(struct kvm_vcpu *vcpu, unsigned long value);
+extern void kvm_x86_sync_dirty_debug_regs(struct kvm_vcpu *vcpu);
+extern void kvm_x86_set_dr7(struct kvm_vcpu *vcpu, unsigned long value);
+extern void kvm_x86_cache_reg(struct kvm_vcpu *vcpu, enum kvm_reg reg);
+extern unsigned long kvm_x86_get_rflags(struct kvm_vcpu *vcpu);
+extern void kvm_x86_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags);
+extern void kvm_x86_tlb_flush(struct kvm_vcpu *vcpu, bool invalidate_gpa);
+extern int kvm_x86_tlb_remote_flush(struct kvm *kvm);
+extern int kvm_x86_tlb_remote_flush_with_range(struct kvm *kvm,
+					       struct kvm_tlb_range *range);
+/*
+ * Flush any TLB entries associated with the given GVA.
+ * Does not need to flush GPA->HPA mappings.
+ * Can potentially get non-canonical addresses through INVLPGs, which
+ * the implementation may choose to ignore if appropriate.
+ */
+extern void kvm_x86_tlb_flush_gva(struct kvm_vcpu *vcpu, gva_t addr);
+extern void kvm_x86_run(struct kvm_vcpu *vcpu);
+extern int kvm_x86_handle_exit(struct kvm_vcpu *vcpu);
+extern int kvm_x86_skip_emulated_instruction(struct kvm_vcpu *vcpu);
+extern void kvm_x86_set_interrupt_shadow(struct kvm_vcpu *vcpu, int mask);
+extern u32 kvm_x86_get_interrupt_shadow(struct kvm_vcpu *vcpu);
+extern void kvm_x86_patch_hypercall(struct kvm_vcpu *vcpu,
+				    unsigned char *hypercall_addr);
+extern void kvm_x86_set_irq(struct kvm_vcpu *vcpu);
+extern void kvm_x86_set_nmi(struct kvm_vcpu *vcpu);
+extern void kvm_x86_queue_exception(struct kvm_vcpu *vcpu);
+extern void kvm_x86_cancel_injection(struct kvm_vcpu *vcpu);
+extern int kvm_x86_interrupt_allowed(struct kvm_vcpu *vcpu);
+extern int kvm_x86_nmi_allowed(struct kvm_vcpu *vcpu);
+extern bool kvm_x86_get_nmi_mask(struct kvm_vcpu *vcpu);
+extern void kvm_x86_set_nmi_mask(struct kvm_vcpu *vcpu, bool masked);
+extern void kvm_x86_enable_nmi_window(struct kvm_vcpu *vcpu);
+extern void kvm_x86_enable_irq_window(struct kvm_vcpu *vcpu);
+extern void kvm_x86_update_cr8_intercept(struct kvm_vcpu *vcpu, int tpr,
+					 int irr);
+extern bool kvm_x86_get_enable_apicv(struct kvm_vcpu *vcpu);
+extern void kvm_x86_refresh_apicv_exec_ctrl(struct kvm_vcpu *vcpu);
+extern void kvm_x86_hwapic_irr_update(struct kvm_vcpu *vcpu, int max_irr);
+extern void kvm_x86_hwapic_isr_update(struct kvm_vcpu *vcpu, int isr);
+extern bool kvm_x86_guest_apic_has_interrupt(struct kvm_vcpu *vcpu);
+extern void kvm_x86_load_eoi_exitmap(struct kvm_vcpu *vcpu,
+				     u64 *eoi_exit_bitmap);
+extern void kvm_x86_set_virtual_apic_mode(struct kvm_vcpu *vcpu);
+extern void kvm_x86_set_apic_access_page_addr(struct kvm_vcpu *vcpu,
+					      hpa_t hpa);
+extern void kvm_x86_deliver_posted_interrupt(struct kvm_vcpu *vcpu,
+					     int vector);
+extern int kvm_x86_sync_pir_to_irr(struct kvm_vcpu *vcpu);
+extern int kvm_x86_set_tss_addr(struct kvm *kvm, unsigned int addr);
+extern int kvm_x86_set_identity_map_addr(struct kvm *kvm, u64 ident_addr);
+extern int kvm_x86_get_tdp_level(struct kvm_vcpu *vcpu);
+extern u64 kvm_x86_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio);
+extern int kvm_x86_get_lpage_level(void);
+extern bool kvm_x86_rdtscp_supported(void);
+extern bool kvm_x86_invpcid_supported(void);
+extern void kvm_x86_set_tdp_cr3(struct kvm_vcpu *vcpu, unsigned long cr3);
+extern void kvm_x86_set_supported_cpuid(u32 func,
+					struct kvm_cpuid_entry2 *entry);
+extern bool kvm_x86_has_wbinvd_exit(void);
+extern u64 kvm_x86_read_l1_tsc_offset(struct kvm_vcpu *vcpu);
+/* Returns actual tsc_offset set in active VMCS */
+extern u64 kvm_x86_write_l1_tsc_offset(struct kvm_vcpu *vcpu, u64 offset);
+extern void kvm_x86_get_exit_info(struct kvm_vcpu *vcpu, u64 *info1,
+				  u64 *info2);
+extern int kvm_x86_check_intercept(struct kvm_vcpu *vcpu,
+				   struct x86_instruction_info *info,
+				   enum x86_intercept_stage stage);
+extern void kvm_x86_handle_exit_irqoff(struct kvm_vcpu *vcpu);
+extern bool kvm_x86_mpx_supported(void);
+extern bool kvm_x86_xsaves_supported(void);
+extern bool kvm_x86_umip_emulated(void);
+extern bool kvm_x86_pt_supported(void);
+extern int kvm_x86_check_nested_events(struct kvm_vcpu *vcpu,
+				       bool external_intr);
+extern void kvm_x86_request_immediate_exit(struct kvm_vcpu *vcpu);
+extern void kvm_x86_sched_in(struct kvm_vcpu *kvm, int cpu);
+/*
+ * Arch-specific dirty logging hooks. These hooks are only supposed to
+ * be valid if the specific arch has hardware-accelerated dirty logging
+ * mechanism. Currently only for PML on VMX.
+ *
+ *  - slot_enable_log_dirty:
+ *	called when enabling log dirty mode for the slot.
+ *  - slot_disable_log_dirty:
+ *	called when disabling log dirty mode for the slot.
+ *	also called when slot is created with log dirty disabled.
+ *  - flush_log_dirty:
+ *	called before reporting dirty_bitmap to userspace.
+ *  - enable_log_dirty_pt_masked:
+ *	called when reenabling log dirty for the GFNs in the mask after
+ *	corresponding bits are cleared in slot->dirty_bitmap.
+ */
+extern void kvm_x86_slot_enable_log_dirty(struct kvm *kvm,
+					  struct kvm_memory_slot *slot);
+extern void kvm_x86_slot_disable_log_dirty(struct kvm *kvm,
+					   struct kvm_memory_slot *slot);
+extern void kvm_x86_flush_log_dirty(struct kvm *kvm);
+extern void kvm_x86_enable_log_dirty_pt_masked(struct kvm *kvm,
+					       struct kvm_memory_slot *slot,
+					       gfn_t offset,
+					       unsigned long mask);
+extern int kvm_x86_write_log_dirty(struct kvm_vcpu *vcpu);
+/*
+ * Architecture specific hooks for vCPU blocking due to
+ * HLT instruction.
+ * Returns for .pre_block():
+ *    - 0 means continue to block the vCPU.
+ *    - 1 means we cannot block the vCPU since some event
+ *        happens during this period, such as, 'ON' bit in
+ *        posted-interrupts descriptor is set.
+ */
+extern int kvm_x86_pre_block(struct kvm_vcpu *vcpu);
+extern void kvm_x86_post_block(struct kvm_vcpu *vcpu);
+extern void kvm_x86_vcpu_blocking(struct kvm_vcpu *vcpu);
+extern void kvm_x86_vcpu_unblocking(struct kvm_vcpu *vcpu);
+extern int kvm_x86_update_pi_irte(struct kvm *kvm, unsigned int host_irq,
+				  uint32_t guest_irq, bool set);
+extern void kvm_x86_apicv_post_state_restore(struct kvm_vcpu *vcpu);
+extern bool kvm_x86_dy_apicv_has_pending_interrupt(struct kvm_vcpu *vcpu);
+extern int kvm_x86_set_hv_timer(struct kvm_vcpu *vcpu, u64 guest_deadline_tsc,
+				bool *expired);
+extern void kvm_x86_cancel_hv_timer(struct kvm_vcpu *vcpu);
+extern void kvm_x86_setup_mce(struct kvm_vcpu *vcpu);
+extern int kvm_x86_get_nested_state(struct kvm_vcpu *vcpu,
+				    struct kvm_nested_state __user *user_kvm_nested_state,
+				    unsigned user_data_size);
+extern int kvm_x86_set_nested_state(struct kvm_vcpu *vcpu,
+				    struct kvm_nested_state __user *user_kvm_nested_state,
+				    struct kvm_nested_state *kvm_state);
+extern void kvm_x86_get_vmcs12_pages(struct kvm_vcpu *vcpu);
+extern int kvm_x86_smi_allowed(struct kvm_vcpu *vcpu);
+extern int kvm_x86_pre_enter_smm(struct kvm_vcpu *vcpu, char *smstate);
+extern int kvm_x86_pre_leave_smm(struct kvm_vcpu *vcpu, const char *smstate);
+extern int kvm_x86_enable_smi_window(struct kvm_vcpu *vcpu);
+extern int kvm_x86_mem_enc_op(struct kvm *kvm, void __user *argp);
+extern int kvm_x86_mem_enc_reg_region(struct kvm *kvm,
+				      struct kvm_enc_region *argp);
+extern int kvm_x86_mem_enc_unreg_region(struct kvm *kvm,
+					struct kvm_enc_region *argp);
+extern int kvm_x86_get_msr_feature(struct kvm_msr_entry *entry);
+extern int kvm_x86_nested_enable_evmcs(struct kvm_vcpu *vcpu,
+				       uint16_t *vmcs_version);
+extern uint16_t kvm_x86_nested_get_evmcs_version(struct kvm_vcpu *vcpu);
+extern bool kvm_x86_need_emulation_on_page_fault(struct kvm_vcpu *vcpu);
+extern bool kvm_x86_apic_init_signal_blocked(struct kvm_vcpu *vcpu);
+
 struct kvm_x86_ops {
 	int (*cpu_has_kvm_support)(void);          /* __init */
 	int (*disabled_by_bios)(void);             /* __init */
@@ -1228,19 +1418,19 @@ extern struct kmem_cache *x86_fpu_cache;
 #define __KVM_HAVE_ARCH_VM_ALLOC
 static inline struct kvm *kvm_arch_alloc_vm(void)
 {
-	return kvm_x86_ops->vm_alloc();
+	return kvm_x86_vm_alloc();
 }
 
 static inline void kvm_arch_free_vm(struct kvm *kvm)
 {
-	return kvm_x86_ops->vm_free(kvm);
+	return kvm_x86_vm_free(kvm);
 }
 
 #define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLB
 static inline int kvm_arch_flush_remote_tlb(struct kvm *kvm)
 {
 	if (kvm_x86_ops->tlb_remote_flush &&
-	    !kvm_x86_ops->tlb_remote_flush(kvm))
+	    !kvm_x86_tlb_remote_flush(kvm))
 		return 0;
 	else
 		return -ENOTSUPP;
@@ -1597,13 +1787,13 @@ static inline bool kvm_irq_is_postable(struct kvm_lapic_irq *irq)
 static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu)
 {
 	if (kvm_x86_ops->vcpu_blocking)
-		kvm_x86_ops->vcpu_blocking(vcpu);
+		kvm_x86_vcpu_blocking(vcpu);
 }
 
 static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu)
 {
 	if (kvm_x86_ops->vcpu_unblocking)
-		kvm_x86_ops->vcpu_unblocking(vcpu);
+		kvm_x86_vcpu_unblocking(vcpu);
 }
 
 static inline void kvm_arch_vcpu_block_finish(struct kvm_vcpu *vcpu) {}
arch/x86/kvm/cpuid.c +11 -11
index dd5985eb61b4..2eacf9cea254 100644
@@ -48,7 +48,7 @@ static u32 xstate_required_size(u64 xstate_bv, bool compacted)
 bool kvm_mpx_supported(void)
 {
 	return ((host_xcr0 & (XFEATURE_MASK_BNDREGS | XFEATURE_MASK_BNDCSR))
-		 && kvm_x86_ops->mpx_supported());
+		 && kvm_x86_mpx_supported());
 }
 EXPORT_SYMBOL_GPL(kvm_mpx_supported);
 
@@ -232,7 +232,7 @@ int kvm_vcpu_ioctl_set_cpuid(struct kvm_vcpu *vcpu,
 	vcpu->arch.cpuid_nent = cpuid->nent;
 	cpuid_fix_nx_cap(vcpu);
 	kvm_apic_set_version(vcpu);
-	kvm_x86_ops->cpuid_update(vcpu);
+	kvm_x86_cpuid_update(vcpu);
 	r = kvm_update_cpuid(vcpu);
 
 out:
@@ -255,7 +255,7 @@ int kvm_vcpu_ioctl_set_cpuid2(struct kvm_vcpu *vcpu,
 		goto out;
 	vcpu->arch.cpuid_nent = cpuid->nent;
 	kvm_apic_set_version(vcpu);
-	kvm_x86_ops->cpuid_update(vcpu);
+	kvm_x86_cpuid_update(vcpu);
 	r = kvm_update_cpuid(vcpu);
 out:
 	return r;
@@ -341,10 +341,10 @@ static int __do_cpuid_func_emulated(struct kvm_cpuid_entry2 *entry,
 
 static inline void do_cpuid_7_mask(struct kvm_cpuid_entry2 *entry, int index)
 {
-	unsigned f_invpcid = kvm_x86_ops->invpcid_supported() ? F(INVPCID) : 0;
+	unsigned f_invpcid = kvm_x86_invpcid_supported() ? F(INVPCID) : 0;
 	unsigned f_mpx = kvm_mpx_supported() ? F(MPX) : 0;
-	unsigned f_umip = kvm_x86_ops->umip_emulated() ? F(UMIP) : 0;
-	unsigned f_intel_pt = kvm_x86_ops->pt_supported() ? F(INTEL_PT) : 0;
+	unsigned f_umip = kvm_x86_umip_emulated() ? F(UMIP) : 0;
+	unsigned f_intel_pt = kvm_x86_pt_supported() ? F(INTEL_PT) : 0;
 	unsigned f_la57;
 
 	/* cpuid 7.0.ebx */
@@ -426,16 +426,16 @@ static inline int __do_cpuid_func(struct kvm_cpuid_entry2 *entry, u32 function,
 	int r;
 	unsigned f_nx = is_efer_nx() ? F(NX) : 0;
 #ifdef CONFIG_X86_64
-	unsigned f_gbpages = (kvm_x86_ops->get_lpage_level() == PT_PDPE_LEVEL)
+	unsigned f_gbpages = (kvm_x86_get_lpage_level() == PT_PDPE_LEVEL)
 				? F(GBPAGES) : 0;
 	unsigned f_lm = F(LM);
 #else
 	unsigned f_gbpages = 0;
 	unsigned f_lm = 0;
 #endif
-	unsigned f_rdtscp = kvm_x86_ops->rdtscp_supported() ? F(RDTSCP) : 0;
-	unsigned f_xsaves = kvm_x86_ops->xsaves_supported() ? F(XSAVES) : 0;
-	unsigned f_intel_pt = kvm_x86_ops->pt_supported() ? F(INTEL_PT) : 0;
+	unsigned f_rdtscp = kvm_x86_rdtscp_supported() ? F(RDTSCP) : 0;
+	unsigned f_xsaves = kvm_x86_xsaves_supported() ? F(XSAVES) : 0;
+	unsigned f_intel_pt = kvm_x86_pt_supported() ? F(INTEL_PT) : 0;
 
 	/* cpuid 1.edx */
 	const u32 kvm_cpuid_1_edx_x86_features =
@@ -786,7 +786,7 @@ static inline int __do_cpuid_func(struct kvm_cpuid_entry2 *entry, u32 function,
 		break;
 	}
 
-	kvm_x86_ops->set_supported_cpuid(function, entry);
+	kvm_x86_set_supported_cpuid(function, entry);
 
 	r = 0;
 
arch/x86/kvm/hyperv.c +3 -3
index fff790a3f4ee..1621bd0ce00c 100644
@@ -1015,7 +1015,7 @@ static int kvm_hv_set_msr_pw(struct kvm_vcpu *vcpu, u32 msr, u64 data,
 		addr = gfn_to_hva(kvm, gfn);
 		if (kvm_is_error_hva(addr))
 			return 1;
-		kvm_x86_ops->patch_hypercall(vcpu, instructions);
+		kvm_x86_patch_hypercall(vcpu, instructions);
 		((unsigned char *)instructions)[3] = 0xc3; /* ret */
 		if (__copy_to_user((void __user *)addr, instructions, 4))
 			return 1;
@@ -1600,7 +1600,7 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
 	 * hypercall generates UD from non zero cpl and real mode
 	 * per HYPER-V spec
 	 */
-	if (kvm_x86_ops->get_cpl(vcpu) != 0 || !is_protmode(vcpu)) {
+	if (kvm_x86_get_cpl(vcpu) != 0 || !is_protmode(vcpu)) {
 		kvm_queue_exception(vcpu, UD_VECTOR);
 		return 1;
 	}
@@ -1794,7 +1794,7 @@ int kvm_vcpu_ioctl_get_hv_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid2 *cpuid,
 	int i, nent = ARRAY_SIZE(cpuid_entries);
 
 	if (kvm_x86_ops->nested_get_evmcs_version)
-		evmcs_ver = kvm_x86_ops->nested_get_evmcs_version(vcpu);
+		evmcs_ver = kvm_x86_nested_get_evmcs_version(vcpu);
 
 	/* Skip NESTED_FEATURES if eVMCS is not supported */
 	if (!evmcs_ver)
arch/x86/kvm/kvm_cache_regs.h +5 -5
index 1cc6c47dc77e..5904f4fd1e15 100644
@@ -41,7 +41,7 @@ static inline unsigned long kvm_register_read(struct kvm_vcpu *vcpu,
 					      enum kvm_reg reg)
 {
 	if (!test_bit(reg, (unsigned long *)&vcpu->arch.regs_avail))
-		kvm_x86_ops->cache_reg(vcpu, reg);
+		kvm_x86_cache_reg(vcpu, reg);
 
 	return vcpu->arch.regs[reg];
 }
@@ -81,7 +81,7 @@ static inline u64 kvm_pdptr_read(struct kvm_vcpu *vcpu, int index)
 
 	if (!test_bit(VCPU_EXREG_PDPTR,
 		      (unsigned long *)&vcpu->arch.regs_avail))
-		kvm_x86_ops->cache_reg(vcpu, (enum kvm_reg)VCPU_EXREG_PDPTR);
+		kvm_x86_cache_reg(vcpu, (enum kvm_reg)VCPU_EXREG_PDPTR);
 
 	return vcpu->arch.walk_mmu->pdptrs[index];
 }
@@ -90,7 +90,7 @@ static inline ulong kvm_read_cr0_bits(struct kvm_vcpu *vcpu, ulong mask)
 {
 	ulong tmask = mask & KVM_POSSIBLE_CR0_GUEST_BITS;
 	if (tmask & vcpu->arch.cr0_guest_owned_bits)
-		kvm_x86_ops->decache_cr0_guest_bits(vcpu);
+		kvm_x86_decache_cr0_guest_bits(vcpu);
 	return vcpu->arch.cr0 & mask;
 }
 
@@ -103,14 +103,14 @@ static inline ulong kvm_read_cr4_bits(struct kvm_vcpu *vcpu, ulong mask)
 {
 	ulong tmask = mask & KVM_POSSIBLE_CR4_GUEST_BITS;
 	if (tmask & vcpu->arch.cr4_guest_owned_bits)
-		kvm_x86_ops->decache_cr4_guest_bits(vcpu);
+		kvm_x86_decache_cr4_guest_bits(vcpu);
 	return vcpu->arch.cr4 & mask;
 }
 
 static inline ulong kvm_read_cr3(struct kvm_vcpu *vcpu)
 {
 	if (!test_bit(VCPU_EXREG_CR3, (ulong *)&vcpu->arch.regs_avail))
-		kvm_x86_ops->decache_cr3(vcpu);
+		kvm_x86_decache_cr3(vcpu);
 	return vcpu->arch.cr3;
 }
 
arch/x86/kvm/lapic.c +15 -15
index 8675458c2205..a588907f07c6 100644
@@ -453,7 +453,7 @@ static inline void apic_clear_irr(int vec, struct kvm_lapic *apic)
 	if (unlikely(vcpu->arch.apicv_active)) {
 		/* need to update RVI */
 		kvm_lapic_clear_vector(vec, apic->regs + APIC_IRR);
-		kvm_x86_ops->hwapic_irr_update(vcpu,
+		kvm_x86_hwapic_irr_update(vcpu,
 				apic_find_highest_irr(apic));
 	} else {
 		apic->irr_pending = false;
@@ -478,7 +478,7 @@ static inline void apic_set_isr(int vec, struct kvm_lapic *apic)
 	 * just set SVI.
 	 */
 	if (unlikely(vcpu->arch.apicv_active))
-		kvm_x86_ops->hwapic_isr_update(vcpu, vec);
+		kvm_x86_hwapic_isr_update(vcpu, vec);
 	else {
 		++apic->isr_count;
 		BUG_ON(apic->isr_count > MAX_APIC_VECTOR);
@@ -526,7 +526,7 @@ static inline void apic_clear_isr(int vec, struct kvm_lapic *apic)
 	 * and must be left alone.
 	 */
 	if (unlikely(vcpu->arch.apicv_active))
-		kvm_x86_ops->hwapic_isr_update(vcpu,
+		kvm_x86_hwapic_isr_update(vcpu,
 					       apic_find_highest_isr(apic));
 	else {
 		--apic->isr_count;
@@ -669,7 +669,7 @@ static int apic_has_interrupt_for_ppr(struct kvm_lapic *apic, u32 ppr)
 {
 	int highest_irr;
 	if (apic->vcpu->arch.apicv_active)
-		highest_irr = kvm_x86_ops->sync_pir_to_irr(apic->vcpu);
+		highest_irr = kvm_x86_sync_pir_to_irr(apic->vcpu);
 	else
 		highest_irr = apic_find_highest_irr(apic);
 	if (highest_irr == -1 || (highest_irr & 0xF0) <= ppr)
@@ -1059,7 +1059,7 @@ static int __apic_accept_irq(struct kvm_lapic *apic, int delivery_mode,
 		}
 
 		if (vcpu->arch.apicv_active)
-			kvm_x86_ops->deliver_posted_interrupt(vcpu, vector);
+			kvm_x86_deliver_posted_interrupt(vcpu, vector);
 		else {
 			kvm_lapic_set_irr(vector, apic);
 
@@ -1704,7 +1704,7 @@ static void cancel_hv_timer(struct kvm_lapic *apic)
 {
 	WARN_ON(preemptible());
 	WARN_ON(!apic->lapic_timer.hv_timer_in_use);
-	kvm_x86_ops->cancel_hv_timer(apic->vcpu);
+	kvm_x86_cancel_hv_timer(apic->vcpu);
 	apic->lapic_timer.hv_timer_in_use = false;
 }
 
@@ -1721,7 +1721,7 @@ static bool start_hv_timer(struct kvm_lapic *apic)
 	if (!ktimer->tscdeadline)
 		return false;
 
-	if (kvm_x86_ops->set_hv_timer(vcpu, ktimer->tscdeadline, &expired))
+	if (kvm_x86_set_hv_timer(vcpu, ktimer->tscdeadline, &expired))
 		return false;
 
 	ktimer->hv_timer_in_use = true;
@@ -2141,7 +2141,7 @@ void kvm_lapic_set_base(struct kvm_vcpu *vcpu, u64 value)
 		kvm_apic_set_x2apic_id(apic, vcpu->vcpu_id);
 
 	if ((old_value ^ value) & (MSR_IA32_APICBASE_ENABLE | X2APIC_ENABLE))
-		kvm_x86_ops->set_virtual_apic_mode(vcpu);
+		kvm_x86_set_virtual_apic_mode(vcpu);
 
 	apic->base_address = apic->vcpu->arch.apic_base &
 			     MSR_IA32_APICBASE_BASE;
@@ -2204,9 +2204,9 @@ void kvm_lapic_reset(struct kvm_vcpu *vcpu, bool init_event)
 	vcpu->arch.pv_eoi.msr_val = 0;
 	apic_update_ppr(apic);
 	if (vcpu->arch.apicv_active) {
-		kvm_x86_ops->apicv_post_state_restore(vcpu);
-		kvm_x86_ops->hwapic_irr_update(vcpu, -1);
-		kvm_x86_ops->hwapic_isr_update(vcpu, -1);
+		kvm_x86_apicv_post_state_restore(vcpu);
+		kvm_x86_hwapic_irr_update(vcpu, -1);
+		kvm_x86_hwapic_isr_update(vcpu, -1);
 	}
 
 	vcpu->arch.apic_arb_prio = 0;
@@ -2458,10 +2458,10 @@ int kvm_apic_set_state(struct kvm_vcpu *vcpu, struct kvm_lapic_state *s)
 				1 : count_vectors(apic->regs + APIC_ISR);
 	apic->highest_isr_cache = -1;
 	if (vcpu->arch.apicv_active) {
-		kvm_x86_ops->apicv_post_state_restore(vcpu);
-		kvm_x86_ops->hwapic_irr_update(vcpu,
+		kvm_x86_apicv_post_state_restore(vcpu);
+		kvm_x86_hwapic_irr_update(vcpu,
 				apic_find_highest_irr(apic));
-		kvm_x86_ops->hwapic_isr_update(vcpu,
+		kvm_x86_hwapic_isr_update(vcpu,
 				apic_find_highest_isr(apic));
 	}
 	kvm_make_request(KVM_REQ_EVENT, vcpu);
@@ -2713,7 +2713,7 @@ void kvm_apic_accept_events(struct kvm_vcpu *vcpu)
 	 * KVM_MP_STATE_INIT_RECEIVED state), just eat SIPIs
 	 * and leave the INIT pending.
 	 */
-	if (is_smm(vcpu) || kvm_x86_ops->apic_init_signal_blocked(vcpu)) {
+	if (is_smm(vcpu) || kvm_x86_apic_init_signal_blocked(vcpu)) {
 		WARN_ON_ONCE(vcpu->arch.mp_state == KVM_MP_STATE_INIT_RECEIVED);
 		if (test_bit(KVM_APIC_SIPI, &apic->pending_events))
 			clear_bit(KVM_APIC_SIPI, &apic->pending_events);
arch/x86/kvm/mmu.c +13 -13
index a10af9c87f8a..80124d00c504 100644
@@ -283,7 +283,7 @@ static void kvm_flush_remote_tlbs_with_range(struct kvm *kvm,
 	int ret = -ENOTSUPP;
 
 	if (range && kvm_x86_ops->tlb_remote_flush_with_range)
-		ret = kvm_x86_ops->tlb_remote_flush_with_range(kvm, range);
+		ret = kvm_x86_tlb_remote_flush_with_range(kvm, range);
 
 	if (ret)
 		kvm_flush_remote_tlbs(kvm);
@@ -1265,7 +1265,7 @@ static int mapping_level(struct kvm_vcpu *vcpu, gfn_t large_gfn,
 	if (host_level == PT_PAGE_TABLE_LEVEL)
 		return host_level;
 
-	max_level = min(kvm_x86_ops->get_lpage_level(), host_level);
+	max_level = min(kvm_x86_get_lpage_level(), host_level);
 
 	for (level = PT_DIRECTORY_LEVEL; level <= max_level; ++level)
 		if (__mmu_gfn_lpage_is_disallowed(large_gfn, level, slot))
@@ -1719,7 +1719,7 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm,
 				gfn_t gfn_offset, unsigned long mask)
 {
 	if (kvm_x86_ops->enable_log_dirty_pt_masked)
-		kvm_x86_ops->enable_log_dirty_pt_masked(kvm, slot, gfn_offset,
+		kvm_x86_enable_log_dirty_pt_masked(kvm, slot, gfn_offset,
 				mask);
 	else
 		kvm_mmu_write_protect_pt_masked(kvm, slot, gfn_offset, mask);
@@ -1735,7 +1735,7 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm,
 int kvm_arch_write_log_dirty(struct kvm_vcpu *vcpu)
 {
 	if (kvm_x86_ops->write_log_dirty)
-		return kvm_x86_ops->write_log_dirty(vcpu);
+		return kvm_x86_write_log_dirty(vcpu);
 
 	return 0;
 }
@@ -2987,7 +2987,7 @@ static int set_spte(struct kvm_vcpu *vcpu, u64 *sptep,
 	if (level > PT_PAGE_TABLE_LEVEL)
 		spte |= PT_PAGE_SIZE_MASK;
 	if (tdp_enabled)
-		spte |= kvm_x86_ops->get_mt_mask(vcpu, gfn,
+		spte |= kvm_x86_get_mt_mask(vcpu, gfn,
 			kvm_is_mmio_pfn(pfn));
 
 	if (host_writable)
@@ -4258,7 +4258,7 @@ static bool fast_cr3_switch(struct kvm_vcpu *vcpu, gpa_t new_cr3,
 			kvm_make_request(KVM_REQ_LOAD_CR3, vcpu);
 			if (!skip_tlb_flush) {
 				kvm_make_request(KVM_REQ_MMU_SYNC, vcpu);
-				kvm_x86_ops->tlb_flush(vcpu, true);
+				kvm_x86_tlb_flush(vcpu, true);
 			}
 
 			/*
@@ -4853,7 +4853,7 @@ kvm_calc_tdp_mmu_root_page_role(struct kvm_vcpu *vcpu, bool base_only)
 	union kvm_mmu_role role = kvm_calc_mmu_role_common(vcpu, base_only);
 
 	role.base.ad_disabled = (shadow_accessed_mask == 0);
-	role.base.level = kvm_x86_ops->get_tdp_level(vcpu);
+	role.base.level = kvm_x86_get_tdp_level(vcpu);
 	role.base.direct = true;
 	role.base.gpte_is_8_bytes = true;
 
@@ -4875,7 +4875,7 @@ static void init_kvm_tdp_mmu(struct kvm_vcpu *vcpu)
 	context->sync_page = nonpaging_sync_page;
 	context->invlpg = nonpaging_invlpg;
 	context->update_pte = nonpaging_update_pte;
-	context->shadow_root_level = kvm_x86_ops->get_tdp_level(vcpu);
+	context->shadow_root_level = kvm_x86_get_tdp_level(vcpu);
 	context->direct_map = true;
 	context->set_cr3 = kvm_x86_ops->set_tdp_cr3;
 	context->get_cr3 = get_cr3;
@@ -5132,7 +5132,7 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu)
 	if (r)
 		goto out;
 	kvm_mmu_load_cr3(vcpu);
-	kvm_x86_ops->tlb_flush(vcpu, true);
+	kvm_x86_tlb_flush(vcpu, true);
 out:
 	return r;
 }
@@ -5446,7 +5446,7 @@ int kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gva_t cr2, u64 error_code,
 	 * guest, with the exception of AMD Erratum 1096 which is unrecoverable.
 	 */
 	if (unlikely(insn && !insn_len)) {
-		if (!kvm_x86_ops->need_emulation_on_page_fault(vcpu))
+		if (!kvm_x86_need_emulation_on_page_fault(vcpu))
 			return 1;
 	}
 
@@ -5492,7 +5492,7 @@ void kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva)
 		if (VALID_PAGE(mmu->prev_roots[i].hpa))
 			mmu->invlpg(vcpu, gva, mmu->prev_roots[i].hpa);
 
-	kvm_x86_ops->tlb_flush_gva(vcpu, gva);
+	kvm_x86_tlb_flush_gva(vcpu, gva);
 	++vcpu->stat.invlpg;
 }
 EXPORT_SYMBOL_GPL(kvm_mmu_invlpg);
@@ -5517,7 +5517,7 @@ void kvm_mmu_invpcid_gva(struct kvm_vcpu *vcpu, gva_t gva, unsigned long pcid)
 	}
 
 	if (tlb_flush)
-		kvm_x86_ops->tlb_flush_gva(vcpu, gva);
+		kvm_x86_tlb_flush_gva(vcpu, gva);
 
 	++vcpu->stat.invlpg;
 
@@ -5634,7 +5634,7 @@ static int alloc_mmu_pages(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu)
 	 * SVM's 32-bit NPT support, TDP paging doesn't use PAE paging and can
 	 * skip allocating the PDP table.
 	 */
-	if (tdp_enabled && kvm_x86_ops->get_tdp_level(vcpu) > PT32E_ROOT_LEVEL)
+	if (tdp_enabled && kvm_x86_get_tdp_level(vcpu) > PT32E_ROOT_LEVEL)
 		return 0;
 
 	page = alloc_page(GFP_KERNEL_ACCOUNT | __GFP_DMA32);
arch/x86/kvm/mmu.h +2 -2
index 11f8ec89433b..8ac288bc42eb 100644
@@ -157,8 +157,8 @@ static inline u8 permission_fault(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,
 				  unsigned pte_access, unsigned pte_pkey,
 				  unsigned pfec)
 {
-	int cpl = kvm_x86_ops->get_cpl(vcpu);
-	unsigned long rflags = kvm_x86_ops->get_rflags(vcpu);
+	int cpl = kvm_x86_get_cpl(vcpu);
+	unsigned long rflags = kvm_x86_get_rflags(vcpu);
 
 	/*
 	 * If CPL < 3, SMAP prevention are disabled if EFLAGS.AC = 1.
arch/x86/kvm/pmu.c +12 -12
index 46875bbd0419..144e5d0c25ff 100644
@@ -183,7 +183,7 @@ void reprogram_gp_counter(struct kvm_pmc *pmc, u64 eventsel)
 			  ARCH_PERFMON_EVENTSEL_CMASK |
 			  HSW_IN_TX |
 			  HSW_IN_TX_CHECKPOINTED))) {
-		config = kvm_x86_ops->pmu_ops->find_arch_event(pmc_to_pmu(pmc),
+		config = kvm_x86_pmu_find_arch_event(pmc_to_pmu(pmc),
 						      event_select,
 						      unit_mask);
 		if (config != PERF_COUNT_HW_MAX)
@@ -225,7 +225,7 @@ void reprogram_fixed_counter(struct kvm_pmc *pmc, u8 ctrl, int idx)
 	}
 
 	pmc_reprogram_counter(pmc, PERF_TYPE_HARDWARE,
-			      kvm_x86_ops->pmu_ops->find_fixed_event(idx),
+			      kvm_x86_pmu_find_fixed_event(idx),
 			      !(en_field & 0x2), /* exclude user */
 			      !(en_field & 0x1), /* exclude kernel */
 			      pmi, false, false);
@@ -234,7 +234,7 @@ EXPORT_SYMBOL_GPL(reprogram_fixed_counter);
 
 void reprogram_counter(struct kvm_pmu *pmu, int pmc_idx)
 {
-	struct kvm_pmc *pmc = kvm_x86_ops->pmu_ops->pmc_idx_to_pmc(pmu, pmc_idx);
+	struct kvm_pmc *pmc = kvm_x86_pmu_pmc_idx_to_pmc(pmu, pmc_idx);
 
 	if (!pmc)
 		return;
@@ -259,7 +259,7 @@ void kvm_pmu_handle_event(struct kvm_vcpu *vcpu)
 	bitmask = pmu->reprogram_pmi;
 
 	for_each_set_bit(bit, (unsigned long *)&bitmask, X86_PMC_IDX_MAX) {
-		struct kvm_pmc *pmc = kvm_x86_ops->pmu_ops->pmc_idx_to_pmc(pmu, bit);
+		struct kvm_pmc *pmc = kvm_x86_pmu_pmc_idx_to_pmc(pmu, bit);
 
 		if (unlikely(!pmc || !pmc->perf_event)) {
 			clear_bit(bit, (unsigned long *)&pmu->reprogram_pmi);
@@ -273,7 +273,7 @@ void kvm_pmu_handle_event(struct kvm_vcpu *vcpu)
 /* check if idx is a valid index to access PMU */
 int kvm_pmu_is_valid_msr_idx(struct kvm_vcpu *vcpu, unsigned idx)
 {
-	return kvm_x86_ops->pmu_ops->is_valid_msr_idx(vcpu, idx);
+	return kvm_x86_pmu_is_valid_msr_idx(vcpu, idx);
 }
 
 bool is_vmware_backdoor_pmc(u32 pmc_idx)
@@ -323,7 +323,7 @@ int kvm_pmu_rdpmc(struct kvm_vcpu *vcpu, unsigned idx, u64 *data)
 	if (is_vmware_backdoor_pmc(idx))
 		return kvm_pmu_rdpmc_vmware(vcpu, idx, data);
 
-	pmc = kvm_x86_ops->pmu_ops->msr_idx_to_pmc(vcpu, idx, &mask);
+	pmc = kvm_x86_pmu_msr_idx_to_pmc(vcpu, idx, &mask);
 	if (!pmc)
 		return 1;
 
@@ -339,17 +339,17 @@ void kvm_pmu_deliver_pmi(struct kvm_vcpu *vcpu)
 
 bool kvm_pmu_is_valid_msr(struct kvm_vcpu *vcpu, u32 msr)
 {
-	return kvm_x86_ops->pmu_ops->is_valid_msr(vcpu, msr);
+	return kvm_x86_pmu_is_valid_msr(vcpu, msr);
 }
 
 int kvm_pmu_get_msr(struct kvm_vcpu *vcpu, u32 msr, u64 *data)
 {
-	return kvm_x86_ops->pmu_ops->get_msr(vcpu, msr, data);
+	return kvm_x86_pmu_get_msr(vcpu, msr, data);
 }
 
 int kvm_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 {
-	return kvm_x86_ops->pmu_ops->set_msr(vcpu, msr_info);
+	return kvm_x86_pmu_set_msr(vcpu, msr_info);
 }
 
 /* refresh PMU settings. This function generally is called when underlying
@@ -358,7 +358,7 @@ int kvm_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
  */
 void kvm_pmu_refresh(struct kvm_vcpu *vcpu)
 {
-	kvm_x86_ops->pmu_ops->refresh(vcpu);
+	kvm_x86_pmu_refresh(vcpu);
 }
 
 void kvm_pmu_reset(struct kvm_vcpu *vcpu)
@@ -366,7 +366,7 @@ void kvm_pmu_reset(struct kvm_vcpu *vcpu)
 	struct kvm_pmu *pmu = vcpu_to_pmu(vcpu);
 
 	irq_work_sync(&pmu->irq_work);
-	kvm_x86_ops->pmu_ops->reset(vcpu);
+	kvm_x86_pmu_reset(vcpu);
 }
 
 void kvm_pmu_init(struct kvm_vcpu *vcpu)
@@ -374,7 +374,7 @@ void kvm_pmu_init(struct kvm_vcpu *vcpu)
 	struct kvm_pmu *pmu = vcpu_to_pmu(vcpu);
 
 	memset(pmu, 0, sizeof(*pmu));
-	kvm_x86_ops->pmu_ops->init(vcpu);
+	kvm_x86_pmu_init(vcpu);
 	init_irq_work(&pmu->irq_work, kvm_pmi_trigger_fn);
 	kvm_pmu_refresh(vcpu);
 }
arch/x86/kvm/pmu.h +18 -1
index 58265f761c3b..82f07e3492df 100644
@@ -19,6 +19,23 @@ struct kvm_event_hw_type_mapping {
 	unsigned event_type;
 };
 
+extern unsigned kvm_x86_pmu_find_arch_event(struct kvm_pmu *pmu,
+					    u8 event_select, u8 unit_mask);
+extern unsigned kvm_x86_pmu_find_fixed_event(int idx);
+extern bool kvm_x86_pmu_pmc_is_enabled(struct kvm_pmc *pmc);
+extern struct kvm_pmc *kvm_x86_pmu_pmc_idx_to_pmc(struct kvm_pmu *pmu,
+						  int pmc_idx);
+extern struct kvm_pmc *kvm_x86_pmu_msr_idx_to_pmc(struct kvm_vcpu *vcpu,
+						  unsigned idx, u64 *mask);
+extern int kvm_x86_pmu_is_valid_msr_idx(struct kvm_vcpu *vcpu, unsigned idx);
+extern bool kvm_x86_pmu_is_valid_msr(struct kvm_vcpu *vcpu, u32 msr);
+extern int kvm_x86_pmu_get_msr(struct kvm_vcpu *vcpu, u32 msr, u64 *data);
+extern int kvm_x86_pmu_set_msr(struct kvm_vcpu *vcpu,
+			       struct msr_data *msr_info);
+extern void kvm_x86_pmu_refresh(struct kvm_vcpu *vcpu);
+extern void kvm_x86_pmu_init(struct kvm_vcpu *vcpu);
+extern void kvm_x86_pmu_reset(struct kvm_vcpu *vcpu);
+
 struct kvm_pmu_ops {
 	unsigned (*find_arch_event)(struct kvm_pmu *pmu, u8 event_select,
 				    u8 unit_mask);
@@ -76,7 +93,7 @@ static inline bool pmc_is_fixed(struct kvm_pmc *pmc)
 
 static inline bool pmc_is_enabled(struct kvm_pmc *pmc)
 {
-	return kvm_x86_ops->pmu_ops->pmc_is_enabled(pmc);
+	return kvm_x86_pmu_pmc_is_enabled(pmc);
 }
 
 /* returns general purpose PMC with the specified MSR. Note that it can be
arch/x86/kvm/pmu_amd.c +26 -26

126 lines not on this page · raw message

arch/x86/kvm/svm.c +385 -274

1,392 lines not on this page · raw message

arch/x86/kvm/trace.h +2 -2
index 7c741a0c5f80..c52bfb96ce40 100644
@@ -240,7 +240,7 @@ TRACE_EVENT(kvm_exit,
 		__entry->guest_rip	= kvm_rip_read(vcpu);
 		__entry->isa            = isa;
 		__entry->vcpu_id        = vcpu->vcpu_id;
-		kvm_x86_ops->get_exit_info(vcpu, &__entry->info1,
+		kvm_x86_get_exit_info(vcpu, &__entry->info1,
 					   &__entry->info2);
 	),
 
@@ -744,7 +744,7 @@ TRACE_EVENT(kvm_emulate_insn,
 		),
 
 	TP_fast_assign(
-		__entry->csbase = kvm_x86_ops->get_segment_base(vcpu, VCPU_SREG_CS);
+		__entry->csbase = kvm_x86_get_segment_base(vcpu, VCPU_SREG_CS);
 		__entry->len = vcpu->arch.emulate_ctxt.fetch.ptr
 			       - vcpu->arch.emulate_ctxt.fetch.data;
 		__entry->rip = vcpu->arch.emulate_ctxt._eip - __entry->len;
arch/x86/kvm/vmx/nested.c +42 -42

277 lines not on this page · raw message

arch/x86/kvm/vmx/pmu_intel.c +27 -28

137 lines not on this page · raw message

arch/x86/kvm/vmx/vmx.c +417 -315

356 lines not on this page · raw message

arch/x86/kvm/vmx/vmx.h +17 -22
index 64d5a4890aa9..21d87cd18c1c 100644

Lines of this file are not shown. View the raw message.

arch/x86/kvm/x86.c +154 -154

308 lines not on this page · raw message

arch/x86/kvm/x86.h +1 -1
index b5274e2a53cf..a7cd551cd982 100644

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Part of the diff is not shown here. View the raw message.

#10 lore raw

PATCH 4/14 KVM: monolithic: x86: handle the request_immediate_exit variation

In reply to Andrea Arcangeli

request_immediate_exit is one of those few cases where the pointer to function of the method isn't fixed at build time and it requires special handling because hardware_setup() may override it at runtime.

---
 arch/x86/kvm/vmx/vmx.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/vmx.c +4 -1
index bb122ab4b96c..aad44e62b20a 100644
@@ -7022,7 +7022,10 @@ void kvm_x86_set_supported_cpuid(u32 func, struct kvm_cpuid_entry2 *entry)
 
 void kvm_x86_request_immediate_exit(struct kvm_vcpu *vcpu)
 {
-	to_vmx(vcpu)->req_immediate_exit = true;
+	if (likely(enable_preemption_timer))
+		to_vmx(vcpu)->req_immediate_exit = true;
+	else
+		__kvm_request_immediate_exit(vcpu);
 }
 
 int kvm_x86_check_intercept(struct kvm_vcpu *vcpu,
#11 lore raw

PATCH 5/14 KVM: monolithic: add more section prefixes in the KVM common code

In reply to Andrea Arcangeli

Add more section prefixes of some KVM common code function because with the monolithic KVM model the section checker can now do a more accurate static analysis at build time and this allows to build without CONFIG_SECTION_MISMATCH_WARN_ONLY=n.

---
 arch/x86/kvm/x86.c       | 4 ++--
 include/linux/kvm_host.h | 4 ++--
 virt/kvm/kvm_main.c      | 6 +++---
 3 files changed, 7 insertions(+), 7 deletions(-)
arch/x86/kvm/x86.c +2 -2
index 8e593d28ff95..601190de4f87 100644
@@ -9206,7 +9206,7 @@ void kvm_arch_hardware_disable(void)
 	drop_user_return_notifiers();
 }
 
-int kvm_arch_hardware_setup(void)
+__init int kvm_arch_hardware_setup(void)
 {
 	int r;
 
@@ -9237,7 +9237,7 @@ void kvm_arch_hardware_unsetup(void)
 	kvm_x86_hardware_unsetup();
 }
 
-int kvm_arch_check_processor_compat(void)
+__init int kvm_arch_check_processor_compat(void)
 {
 	return kvm_x86_check_processor_compatibility();
 }
include/linux/kvm_host.h +2 -2
index fcb46b3374c6..8621916998e1 100644
@@ -867,9 +867,9 @@ void kvm_arch_create_vcpu_debugfs(struct kvm_vcpu *vcpu);
 
 int kvm_arch_hardware_enable(void);
 void kvm_arch_hardware_disable(void);
-int kvm_arch_hardware_setup(void);
+__init int kvm_arch_hardware_setup(void);
 void kvm_arch_hardware_unsetup(void);
-int kvm_arch_check_processor_compat(void);
+__init int kvm_arch_check_processor_compat(void);
 int kvm_arch_vcpu_runnable(struct kvm_vcpu *vcpu);
 bool kvm_arch_vcpu_in_kernel(struct kvm_vcpu *vcpu);
 int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu);
virt/kvm/kvm_main.c +3 -3
index e6de3159e682..9aa448ea688f 100644
@@ -4235,13 +4235,13 @@ static void kvm_sched_out(struct preempt_notifier *pn,
 	kvm_arch_vcpu_put(vcpu);
 }
 
-static void check_processor_compat(void *rtn)
+static __init void check_processor_compat(void *rtn)
 {
 	*(int *)rtn = kvm_arch_check_processor_compat();
 }
 
-int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align,
-		  struct module *module)
+__init int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align,
+		    struct module *module)
 {
 	int r;
 	int cpu;
#12 lore raw

PATCH 6/14 KVM: monolithic: x86: remove __exit section prefix from machine_unsetup

In reply to Andrea Arcangeli

Adjusts the section prefixes of some KVM x86 code function because with the monolithic KVM model the section checker can now do a more accurate static analysis at build time and it found a potentially kernel crashing bug. This also allows to build without CONFIG_SECTION_MISMATCH_WARN_ONLY=n.

The __exit removed from machine_unsetup is because kvm_arch_hardware_unsetup() is called by kvm_init() which is in the __init section. It's not allowed to call a function located in the __exit section and dropped during the kernel link from the __init section or the kernel will crash if that call is made.

---
 arch/x86/include/asm/kvm_host.h | 4 ++--
 arch/x86/kvm/svm.c              | 2 +-
 arch/x86/kvm/vmx/vmx.c          | 2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -2
index 47eeb92d4b4a..0ae65148e5ed 100644
@@ -1008,7 +1008,7 @@ extern int kvm_x86_hardware_enable(void);
 extern void kvm_x86_hardware_disable(void);
 extern __init int kvm_x86_check_processor_compatibility(void);
 extern __init int kvm_x86_hardware_setup(void);
-extern __exit void kvm_x86_hardware_unsetup(void);
+extern void kvm_x86_hardware_unsetup(void);
 extern bool kvm_x86_cpu_has_accelerated_tpr(void);
 extern bool kvm_x86_has_emulated_msr(int index);
 extern void kvm_x86_cpuid_update(struct kvm_vcpu *vcpu);
@@ -1199,7 +1199,7 @@ struct kvm_x86_ops {
 	void (*hardware_disable)(void);
 	int (*check_processor_compatibility)(void);/* __init */
 	int (*hardware_setup)(void);               /* __init */
-	void (*hardware_unsetup)(void);            /* __exit */
+	void (*hardware_unsetup)(void);
 	bool (*cpu_has_accelerated_tpr)(void);
 	bool (*has_emulated_msr)(int index);
 	void (*cpuid_update)(struct kvm_vcpu *vcpu);
arch/x86/kvm/svm.c +1 -1
index aa8c0efdc441..057ba1f8d7b3 100644
@@ -1405,7 +1405,7 @@ __init int kvm_x86_hardware_setup(void)
 	return r;
 }
 
-__exit void kvm_x86_hardware_unsetup(void)
+void kvm_x86_hardware_unsetup(void)
 {
 	int cpu;
 
arch/x86/kvm/vmx/vmx.c +1 -1
index aad44e62b20a..2ae162eb082e 100644
@@ -7645,7 +7645,7 @@ __init int kvm_x86_hardware_setup(void)
 	return r;
 }
 
-__exit void kvm_x86_hardware_unsetup(void)
+void kvm_x86_hardware_unsetup(void)
 {
 	if (nested)
 		nested_vmx_hardware_unsetup();
#13 lore raw

PATCH 7/14 KVM: monolithic: x86: remove __init section prefix from kvm_x86_cpu_has_kvm_support

In reply to Andrea Arcangeli

Adjusts the section prefixes of some KVM x86 code function because with the monolithic KVM model the section checker can now do a more accurate static analysis at build time. This also allows to build without CONFIG_SECTION_MISMATCH_WARN_ONLY=n.

The __init needs to be removed on vmx despite it's only svm calling it from kvm_x86_hardware_enable which is eventually called by hardware_enable_nolock() or there's a (potentially false positive) warning (false positive because this function isn't called in the vmx case). If this isn't needed the right cleanup isn't to put it in the __init section, but to drop it. As long as it's defined in vmx as a kvm_x86 operation, it's expectable that might eventually be called at runtime while hot plugging new CPUs.

---
 arch/x86/include/asm/kvm_host.h | 2 +-
 arch/x86/kvm/vmx/vmx.c          | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -1
index 0ae65148e5ed..75affbf7861b 100644
@@ -1002,7 +1002,7 @@ struct kvm_lapic_irq {
 	bool msi_redir_hint;
 };
 
-extern __init int kvm_x86_cpu_has_kvm_support(void);
+extern int kvm_x86_cpu_has_kvm_support(void);
 extern __init int kvm_x86_disabled_by_bios(void);
 extern int kvm_x86_hardware_enable(void);
 extern void kvm_x86_hardware_disable(void);
arch/x86/kvm/vmx/vmx.c +1 -1
index 2ae162eb082e..faccffc4709e 100644
@@ -2094,7 +2094,7 @@ void kvm_x86_cache_reg(struct kvm_vcpu *vcpu, enum kvm_reg reg)
 	}
 }
 
-__init int kvm_x86_cpu_has_kvm_support(void)
+int kvm_x86_cpu_has_kvm_support(void)
 {
 	return cpu_has_vmx();
 }
#14 lore raw

PATCH 8/14 KVM: monolithic: x86: remove exports

In reply to Andrea Arcangeli

The exports would be duplicated across kvm-amd and kvm-intel if they're kept, this cleanup various harmless warnings about it.

The following warning remains for now to be able to load the kvmgt driver. These remaining warnings can be handled later.

WARNING: arch/x86/kvm/kvm-amd: 'kvm_slot_page_track_add_page' exported twice. Previous export was in arch/x86/kvm/kvm-intel.ko
WARNING: arch/x86/kvm/kvm-amd: 'kvm_slot_page_track_remove_page' exported twice. Previous export was in arch/x86/kvm/kvm-intel.ko
WARNING: arch/x86/kvm/kvm-amd: 'kvm_page_track_register_notifier' exported twice. Previous export was in arch/x86/kvm/kvm-intel.ko
WARNING: arch/x86/kvm/kvm-amd: 'kvm_page_track_unregister_notifier' exported twice. Previous export was in arch/x86/kvm/kvm-intel.ko

---
 arch/x86/kvm/cpuid.c    |   5 --
 arch/x86/kvm/hyperv.c   |   2 -
 arch/x86/kvm/irq.c      |   4 --
 arch/x86/kvm/irq_comm.c |   2 -
 arch/x86/kvm/lapic.c    |  16 ------
 arch/x86/kvm/mmu.c      |  24 ---------
 arch/x86/kvm/mtrr.c     |   2 -
 arch/x86/kvm/pmu.c      |   3 --
 arch/x86/kvm/x86.c      | 106 ----------------------------------------
 9 files changed, 164 deletions(-)
arch/x86/kvm/cpuid.c +0 -5
index 2eacf9cea254..3c83921eb8b6 100644
@@ -50,7 +50,6 @@ bool kvm_mpx_supported(void)
 	return ((host_xcr0 & (XFEATURE_MASK_BNDREGS | XFEATURE_MASK_BNDCSR))
 		 && kvm_x86_mpx_supported());
 }
-EXPORT_SYMBOL_GPL(kvm_mpx_supported);
 
 u64 kvm_supported_xcr0(void)
 {
@@ -192,7 +191,6 @@ int cpuid_query_maxphyaddr(struct kvm_vcpu *vcpu)
 not_found:
 	return 36;
 }
-EXPORT_SYMBOL_GPL(cpuid_query_maxphyaddr);
 
 /* when an old userspace process fills a new kernel module */
 int kvm_vcpu_ioctl_set_cpuid(struct kvm_vcpu *vcpu,
@@ -960,7 +958,6 @@ struct kvm_cpuid_entry2 *kvm_find_cpuid_entry(struct kvm_vcpu *vcpu,
 	}
 	return best;
 }
-EXPORT_SYMBOL_GPL(kvm_find_cpuid_entry);
 
 /*
  * If no match is found, check whether we exceed the vCPU's limit
@@ -1011,7 +1008,6 @@ bool kvm_cpuid(struct kvm_vcpu *vcpu, u32 *eax, u32 *ebx,
 	trace_kvm_cpuid(function, *eax, *ebx, *ecx, *edx, entry_found);
 	return entry_found;
 }
-EXPORT_SYMBOL_GPL(kvm_cpuid);
 
 int kvm_emulate_cpuid(struct kvm_vcpu *vcpu)
 {
@@ -1029,4 +1025,3 @@ int kvm_emulate_cpuid(struct kvm_vcpu *vcpu)
 	kvm_rdx_write(vcpu, edx);
 	return kvm_skip_emulated_instruction(vcpu);
 }
-EXPORT_SYMBOL_GPL(kvm_emulate_cpuid);
arch/x86/kvm/hyperv.c +0 -2
index 1621bd0ce00c..07b2404cd469 100644
@@ -711,7 +711,6 @@ bool kvm_hv_assist_page_enabled(struct kvm_vcpu *vcpu)
 		return false;
 	return vcpu->arch.pv_eoi.msr_val & KVM_MSR_ENABLED;
 }
-EXPORT_SYMBOL_GPL(kvm_hv_assist_page_enabled);
 
 bool kvm_hv_get_assist_page(struct kvm_vcpu *vcpu,
 			    struct hv_vp_assist_page *assist_page)
@@ -721,7 +720,6 @@ bool kvm_hv_get_assist_page(struct kvm_vcpu *vcpu,
 	return !kvm_read_guest_cached(vcpu->kvm, &vcpu->arch.pv_eoi.data,
 				      assist_page, sizeof(*assist_page));
 }
-EXPORT_SYMBOL_GPL(kvm_hv_get_assist_page);
 
 static void stimer_prepare_msg(struct kvm_vcpu_hv_stimer *stimer)
 {
arch/x86/kvm/irq.c +0 -4
index e330e7d125f7..ba4300f36a32 100644
@@ -26,7 +26,6 @@ int kvm_cpu_has_pending_timer(struct kvm_vcpu *vcpu)
 
 	return 0;
 }
-EXPORT_SYMBOL(kvm_cpu_has_pending_timer);
 
 /*
  * check if there is a pending userspace external interrupt
@@ -109,7 +108,6 @@ int kvm_cpu_has_interrupt(struct kvm_vcpu *v)
 
 	return kvm_apic_has_interrupt(v) != -1;	/* LAPIC */
 }
-EXPORT_SYMBOL_GPL(kvm_cpu_has_interrupt);
 
 /*
  * Read pending interrupt(from non-APIC source)
@@ -146,14 +144,12 @@ int kvm_cpu_get_interrupt(struct kvm_vcpu *v)
 
 	return kvm_get_apic_interrupt(v);	/* APIC */
 }
-EXPORT_SYMBOL_GPL(kvm_cpu_get_interrupt);
 
 void kvm_inject_pending_timer_irqs(struct kvm_vcpu *vcpu)
 {
 	if (lapic_in_kernel(vcpu))
 		kvm_inject_apic_timer_irqs(vcpu);
 }
-EXPORT_SYMBOL_GPL(kvm_inject_pending_timer_irqs);
 
 void __kvm_migrate_timers(struct kvm_vcpu *vcpu)
 {
arch/x86/kvm/irq_comm.c +0 -2
index 8ecd48d31800..64a13d5fcc9f 100644
@@ -122,7 +122,6 @@ void kvm_set_msi_irq(struct kvm *kvm, struct kvm_kernel_irq_routing_entry *e,
 	irq->level = 1;
 	irq->shorthand = 0;
 }
-EXPORT_SYMBOL_GPL(kvm_set_msi_irq);
 
 static inline bool kvm_msi_route_invalid(struct kvm *kvm,
 		struct kvm_kernel_irq_routing_entry *e)
@@ -346,7 +345,6 @@ bool kvm_intr_is_single_vcpu(struct kvm *kvm, struct kvm_lapic_irq *irq,
 
 	return r == 1;
 }
-EXPORT_SYMBOL_GPL(kvm_intr_is_single_vcpu);
 
 #define IOAPIC_ROUTING_ENTRY(irq) \
 	{ .gsi = irq, .type = KVM_IRQ_ROUTING_IRQCHIP,	\
arch/x86/kvm/lapic.c +0 -16
index a588907f07c6..7ddea52674a4 100644
@@ -122,7 +122,6 @@ bool kvm_can_post_timer_interrupt(struct kvm_vcpu *vcpu)
 {
 	return pi_inject_timer && kvm_vcpu_apicv_active(vcpu);
 }
-EXPORT_SYMBOL_GPL(kvm_can_post_timer_interrupt);
 
 static bool kvm_use_posted_timer_interrupt(struct kvm_vcpu *vcpu)
 {
@@ -412,7 +411,6 @@ bool __kvm_apic_update_irr(u32 *pir, void *regs, int *max_irr)
 	return ((max_updated_irr != -1) &&
 		(max_updated_irr == *max_irr));
 }
-EXPORT_SYMBOL_GPL(__kvm_apic_update_irr);
 
 bool kvm_apic_update_irr(struct kvm_vcpu *vcpu, u32 *pir, int *max_irr)
 {
@@ -420,7 +418,6 @@ bool kvm_apic_update_irr(struct kvm_vcpu *vcpu, u32 *pir, int *max_irr)
 
 	return __kvm_apic_update_irr(pir, apic->regs, max_irr);
 }
-EXPORT_SYMBOL_GPL(kvm_apic_update_irr);
 
 static inline int apic_search_irr(struct kvm_lapic *apic)
 {
@@ -544,7 +541,6 @@ int kvm_lapic_find_highest_irr(struct kvm_vcpu *vcpu)
 	 */
 	return apic_find_highest_irr(vcpu->arch.apic);
 }
-EXPORT_SYMBOL_GPL(kvm_lapic_find_highest_irr);
 
 static int __apic_accept_irq(struct kvm_lapic *apic, int delivery_mode,
 			     int vector, int level, int trig_mode,
@@ -712,7 +708,6 @@ void kvm_apic_update_ppr(struct kvm_vcpu *vcpu)
 {
 	apic_update_ppr(vcpu->arch.apic);
 }
-EXPORT_SYMBOL_GPL(kvm_apic_update_ppr);
 
 static void apic_set_tpr(struct kvm_lapic *apic, u32 tpr)
 {
@@ -823,7 +818,6 @@ bool kvm_apic_match_dest(struct kvm_vcpu *vcpu, struct kvm_lapic *source,
 		return false;
 	}
 }
-EXPORT_SYMBOL_GPL(kvm_apic_match_dest);
 
 int kvm_vector_to_index(u32 vector, u32 dest_vcpus,
 		       const unsigned long *bitmap, u32 bitmap_size)
@@ -1196,7 +1190,6 @@ void kvm_apic_set_eoi_accelerated(struct kvm_vcpu *vcpu, int vector)
 	kvm_ioapic_send_eoi(apic, vector);
 	kvm_make_request(KVM_REQ_EVENT, apic->vcpu);
 }
-EXPORT_SYMBOL_GPL(kvm_apic_set_eoi_accelerated);
 
 static void apic_send_ipi(struct kvm_lapic *apic, u32 icr_low, u32 icr_high)
 {
@@ -1355,7 +1348,6 @@ int kvm_lapic_reg_read(struct kvm_lapic *apic, u32 offset, int len,
 	}
 	return 0;
 }
-EXPORT_SYMBOL_GPL(kvm_lapic_reg_read);
 
 static int apic_mmio_in_range(struct kvm_lapic *apic, gpa_t addr)
 {
@@ -1533,7 +1525,6 @@ void kvm_wait_lapic_expire(struct kvm_vcpu *vcpu)
 	if (lapic_timer_int_injected(vcpu))
 		__kvm_wait_lapic_expire(vcpu);
 }
-EXPORT_SYMBOL_GPL(kvm_wait_lapic_expire);
 
 static void kvm_apic_inject_pending_timer_irqs(struct kvm_lapic *apic)
 {
@@ -1698,7 +1689,6 @@ bool kvm_lapic_hv_timer_in_use(struct kvm_vcpu *vcpu)
 
 	return vcpu->arch.apic->lapic_timer.hv_timer_in_use;
 }
-EXPORT_SYMBOL_GPL(kvm_lapic_hv_timer_in_use);
 
 static void cancel_hv_timer(struct kvm_lapic *apic)
 {
@@ -1799,13 +1789,11 @@ void kvm_lapic_expired_hv_timer(struct kvm_vcpu *vcpu)
 out:
 	preempt_enable();
 }
-EXPORT_SYMBOL_GPL(kvm_lapic_expired_hv_timer);
 
 void kvm_lapic_switch_to_hv_timer(struct kvm_vcpu *vcpu)
 {
 	restart_apic_timer(vcpu->arch.apic);
 }
-EXPORT_SYMBOL_GPL(kvm_lapic_switch_to_hv_timer);
 
 void kvm_lapic_switch_to_sw_timer(struct kvm_vcpu *vcpu)
 {
@@ -1817,7 +1805,6 @@ void kvm_lapic_switch_to_sw_timer(struct kvm_vcpu *vcpu)
 		start_sw_timer(apic);
 	preempt_enable();
 }
-EXPORT_SYMBOL_GPL(kvm_lapic_switch_to_sw_timer);
 
 void kvm_lapic_restart_hv_timer(struct kvm_vcpu *vcpu)
 {
@@ -1987,7 +1974,6 @@ int kvm_lapic_reg_write(struct kvm_lapic *apic, u32 reg, u32 val)
 
 	return ret;
 }
-EXPORT_SYMBOL_GPL(kvm_lapic_reg_write);
 
 static int apic_mmio_write(struct kvm_vcpu *vcpu, struct kvm_io_device *this,
 			    gpa_t address, int len, const void *data)
@@ -2026,7 +2012,6 @@ void kvm_lapic_set_eoi(struct kvm_vcpu *vcpu)
 {
 	kvm_lapic_reg_write(vcpu->arch.apic, APIC_EOI, 0);
 }
-EXPORT_SYMBOL_GPL(kvm_lapic_set_eoi);
 
 /* emulate APIC access in a trap manner */
 void kvm_apic_write_nodecode(struct kvm_vcpu *vcpu, u32 offset)
@@ -2041,7 +2026,6 @@ void kvm_apic_write_nodecode(struct kvm_vcpu *vcpu, u32 offset)
 	/* TODO: optimize to just emulate side effect w/o one more write */
 	kvm_lapic_reg_write(vcpu->arch.apic, offset, val);
 }
-EXPORT_SYMBOL_GPL(kvm_apic_write_nodecode);
 
 void kvm_free_lapic(struct kvm_vcpu *vcpu)
 {
arch/x86/kvm/mmu.c +0 -24
index 80124d00c504..d2b56dba7c77 100644
@@ -308,7 +308,6 @@ void kvm_mmu_set_mmio_spte_mask(u64 mmio_mask, u64 mmio_value, u64 access_mask)
 	shadow_mmio_mask = mmio_mask | SPTE_SPECIAL_MASK;
 	shadow_mmio_access_mask = access_mask;
 }
-EXPORT_SYMBOL_GPL(kvm_mmu_set_mmio_spte_mask);
 
 static bool is_mmio_spte(u64 spte)
 {
@@ -474,7 +473,6 @@ void kvm_mmu_set_mask_ptes(u64 user_mask, u64 accessed_mask,
 	shadow_acc_track_mask = acc_track_mask;
 	shadow_me_mask = me_mask;
 }
-EXPORT_SYMBOL_GPL(kvm_mmu_set_mask_ptes);
 
 static u8 kvm_get_shadow_phys_bits(void)
 {
@@ -1702,7 +1700,6 @@ void kvm_mmu_clear_dirty_pt_masked(struct kvm *kvm,
 		mask &= mask - 1;
 	}
 }
-EXPORT_SYMBOL_GPL(kvm_mmu_clear_dirty_pt_masked);
 
 /**
  * kvm_arch_mmu_enable_log_dirty_pt_masked - enable dirty logging for selected
@@ -2853,7 +2850,6 @@ int kvm_mmu_unprotect_page(struct kvm *kvm, gfn_t gfn)
 
 	return r;
 }
-EXPORT_SYMBOL_GPL(kvm_mmu_unprotect_page);
 
 static void kvm_unsync_page(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp)
 {
@@ -3621,7 +3617,6 @@ void kvm_mmu_free_roots(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,
 	kvm_mmu_commit_zap_page(vcpu->kvm, &invalid_list);
 	spin_unlock(&vcpu->kvm->mmu_lock);
 }
-EXPORT_SYMBOL_GPL(kvm_mmu_free_roots);
 
 static int mmu_check_root(struct kvm_vcpu *vcpu, gfn_t root_gfn)
 {
@@ -3846,7 +3841,6 @@ void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu)
 	kvm_mmu_audit(vcpu, AUDIT_POST_SYNC);
 	spin_unlock(&vcpu->kvm->mmu_lock);
 }
-EXPORT_SYMBOL_GPL(kvm_mmu_sync_roots);
 
 static gpa_t nonpaging_gva_to_gpa(struct kvm_vcpu *vcpu, gva_t vaddr,
 				  u32 access, struct x86_exception *exception)
@@ -4114,7 +4108,6 @@ int kvm_handle_page_fault(struct kvm_vcpu *vcpu, u64 error_code,
 	}
 	return r;
 }
-EXPORT_SYMBOL_GPL(kvm_handle_page_fault);
 
 static bool
 check_hugepage_cache_consistency(struct kvm_vcpu *vcpu, gfn_t gfn, int level)
@@ -4294,7 +4287,6 @@ void kvm_mmu_new_cr3(struct kvm_vcpu *vcpu, gpa_t new_cr3, bool skip_tlb_flush)
 	__kvm_mmu_new_cr3(vcpu, new_cr3, kvm_mmu_calc_root_page_role(vcpu),
 			  skip_tlb_flush);
 }
-EXPORT_SYMBOL_GPL(kvm_mmu_new_cr3);
 
 static unsigned long get_cr3(struct kvm_vcpu *vcpu)
 {
@@ -4534,7 +4526,6 @@ reset_shadow_zero_bits_mask(struct kvm_vcpu *vcpu, struct kvm_mmu *context)
 	}
 
 }
-EXPORT_SYMBOL_GPL(reset_shadow_zero_bits_mask);
 
 static inline bool boot_cpu_is_amd(void)
 {
@@ -4954,7 +4945,6 @@ void kvm_init_shadow_mmu(struct kvm_vcpu *vcpu)
 	context->mmu_role.as_u64 = new_role.as_u64;
 	reset_shadow_zero_bits_mask(vcpu, context);
 }
-EXPORT_SYMBOL_GPL(kvm_init_shadow_mmu);
 
 static union kvm_mmu_role
 kvm_calc_shadow_ept_root_page_role(struct kvm_vcpu *vcpu, bool accessed_dirty,
@@ -5018,7 +5008,6 @@ void kvm_init_shadow_ept_mmu(struct kvm_vcpu *vcpu, bool execonly,
 	reset_rsvds_bits_mask_ept(vcpu, context, execonly);
 	reset_ept_shadow_zero_bits_mask(vcpu, context, execonly);
 }
-EXPORT_SYMBOL_GPL(kvm_init_shadow_ept_mmu);
 
 static void init_kvm_softmmu(struct kvm_vcpu *vcpu)
 {
@@ -5098,7 +5087,6 @@ void kvm_init_mmu(struct kvm_vcpu *vcpu, bool reset_roots)
 	else
 		init_kvm_softmmu(vcpu);
 }
-EXPORT_SYMBOL_GPL(kvm_init_mmu);
 
 static union kvm_mmu_page_role
 kvm_mmu_calc_root_page_role(struct kvm_vcpu *vcpu)
@@ -5118,7 +5106,6 @@ void kvm_mmu_reset_context(struct kvm_vcpu *vcpu)
 	kvm_mmu_unload(vcpu);
 	kvm_init_mmu(vcpu, true);
 }
-EXPORT_SYMBOL_GPL(kvm_mmu_reset_context);
 
 int kvm_mmu_load(struct kvm_vcpu *vcpu)
 {
@@ -5136,7 +5123,6 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu)
 out:
 	return r;
 }
-EXPORT_SYMBOL_GPL(kvm_mmu_load);
 
 void kvm_mmu_unload(struct kvm_vcpu *vcpu)
 {
@@ -5145,7 +5131,6 @@ void kvm_mmu_unload(struct kvm_vcpu *vcpu)
 	kvm_mmu_free_roots(vcpu, &vcpu->arch.guest_mmu, KVM_MMU_ROOTS_ALL);
 	WARN_ON(VALID_PAGE(vcpu->arch.guest_mmu.root_hpa));
 }
-EXPORT_SYMBOL_GPL(kvm_mmu_unload);
 
 static void mmu_pte_write_new_pte(struct kvm_vcpu *vcpu,
 				  struct kvm_mmu_page *sp, u64 *spte,
@@ -5357,7 +5342,6 @@ int kvm_mmu_unprotect_page_virt(struct kvm_vcpu *vcpu, gva_t gva)
 
 	return r;
 }
-EXPORT_SYMBOL_GPL(kvm_mmu_unprotect_page_virt);
 
 static int make_mmu_pages_available(struct kvm_vcpu *vcpu)
 {
@@ -5464,7 +5448,6 @@ int kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gva_t cr2, u64 error_code,
 		BUG();
 	}
 }
-EXPORT_SYMBOL_GPL(kvm_mmu_page_fault);
 
 void kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva)
 {
@@ -5495,7 +5478,6 @@ void kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva)
 	kvm_x86_tlb_flush_gva(vcpu, gva);
 	++vcpu->stat.invlpg;
 }
-EXPORT_SYMBOL_GPL(kvm_mmu_invlpg);
 
 void kvm_mmu_invpcid_gva(struct kvm_vcpu *vcpu, gva_t gva, unsigned long pcid)
 {
@@ -5527,19 +5509,16 @@ void kvm_mmu_invpcid_gva(struct kvm_vcpu *vcpu, gva_t gva, unsigned long pcid)
 	 * for them.
 	 */
 }
-EXPORT_SYMBOL_GPL(kvm_mmu_invpcid_gva);
 
 void kvm_enable_tdp(void)
 {
 	tdp_enabled = true;
 }
-EXPORT_SYMBOL_GPL(kvm_enable_tdp);
 
 void kvm_disable_tdp(void)
 {
 	tdp_enabled = false;
 }
-EXPORT_SYMBOL_GPL(kvm_disable_tdp);
 
 
 /* The return value indicates if tlb flush on all vcpus is needed. */
@@ -5920,7 +5899,6 @@ void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm,
 		kvm_flush_remote_tlbs_with_address(kvm, memslot->base_gfn,
 				memslot->npages);
 }
-EXPORT_SYMBOL_GPL(kvm_mmu_slot_leaf_clear_dirty);
 
 void kvm_mmu_slot_largepage_remove_write_access(struct kvm *kvm,
 					struct kvm_memory_slot *memslot)
@@ -5939,7 +5917,6 @@ void kvm_mmu_slot_largepage_remove_write_access(struct kvm *kvm,
 		kvm_flush_remote_tlbs_with_address(kvm, memslot->base_gfn,
 				memslot->npages);
 }
-EXPORT_SYMBOL_GPL(kvm_mmu_slot_largepage_remove_write_access);
 
 void kvm_mmu_slot_set_dirty(struct kvm *kvm,
 			    struct kvm_memory_slot *memslot)
@@ -5957,7 +5934,6 @@ void kvm_mmu_slot_set_dirty(struct kvm *kvm,
 		kvm_flush_remote_tlbs_with_address(kvm, memslot->base_gfn,
 				memslot->npages);
 }
-EXPORT_SYMBOL_GPL(kvm_mmu_slot_set_dirty);
 
 static void __kvm_mmu_zap_all(struct kvm *kvm, bool mmio_only)
 {
arch/x86/kvm/mtrr.c +0 -2
index 25ce3edd1872..477f7141f793 100644
@@ -91,7 +91,6 @@ bool kvm_mtrr_valid(struct kvm_vcpu *vcpu, u32 msr, u64 data)
 
 	return true;
 }
-EXPORT_SYMBOL_GPL(kvm_mtrr_valid);
 
 static bool mtrr_is_enabled(struct kvm_mtrr *mtrr_state)
 {
@@ -686,7 +685,6 @@ u8 kvm_mtrr_get_guest_memory_type(struct kvm_vcpu *vcpu, gfn_t gfn)
 
 	return type;
 }
-EXPORT_SYMBOL_GPL(kvm_mtrr_get_guest_memory_type);
 
 bool kvm_mtrr_check_gfn_range_consistency(struct kvm_vcpu *vcpu, gfn_t gfn,
 					  int page_num)
arch/x86/kvm/pmu.c +0 -3
index 144e5d0c25ff..0ac70bad4b31 100644
@@ -200,7 +200,6 @@ void reprogram_gp_counter(struct kvm_pmc *pmc, u64 eventsel)
 			      (eventsel & HSW_IN_TX),
 			      (eventsel & HSW_IN_TX_CHECKPOINTED));
 }
-EXPORT_SYMBOL_GPL(reprogram_gp_counter);
 
 void reprogram_fixed_counter(struct kvm_pmc *pmc, u8 ctrl, int idx)
 {
@@ -230,7 +229,6 @@ void reprogram_fixed_counter(struct kvm_pmc *pmc, u8 ctrl, int idx)
 			      !(en_field & 0x1), /* exclude kernel */
 			      pmi, false, false);
 }
-EXPORT_SYMBOL_GPL(reprogram_fixed_counter);
 
 void reprogram_counter(struct kvm_pmu *pmu, int pmc_idx)
 {
@@ -248,7 +246,6 @@ void reprogram_counter(struct kvm_pmu *pmu, int pmc_idx)
 		reprogram_fixed_counter(pmc, ctrl, idx);
 	}
 }
-EXPORT_SYMBOL_GPL(reprogram_counter);
 
 void kvm_pmu_handle_event(struct kvm_vcpu *vcpu)
 {
arch/x86/kvm/x86.c +0 -106

573 lines not on this page · raw message

#15 lore raw

PATCH 9/14 KVM: monolithic: remove exports from KVM common code

In reply to Andrea Arcangeli

The exports would be duplicated across kvm-amd and kvm-intel if they're kept and that causes various harmless warnings.

The warnings aren't particularly concerning because the two modules can't load at the same time, but it's cleaner to remove the warnings by removing the exports.

This commit might break non-x86 archs, but it should be simple to make them monolithic too (if they're not already).

In the unlikely case there's a legit reason not to go monolithic in any arch and to keep kvm.ko around, we'll need a way to retain the exports. In which case this commit would need to be reversed and the exports in the kvm common code should then be done only conditionally to a new opt-in per-arch CONFIG option.

The following warning remains for now to be able to load the kvmgt driver. These remaining warnings can be handled later.

WARNING: arch/x86/kvm/kvm-amd: 'kvm_get_kvm' exported twice. Previous export was in arch/x86/kvm/kvm-intel.ko
WARNING: arch/x86/kvm/kvm-amd: 'kvm_put_kvm' exported twice. Previous export was in arch/x86/kvm/kvm-intel.ko
WARNING: arch/x86/kvm/kvm-amd: 'gfn_to_memslot' exported twice. Previous export was in arch/x86/kvm/kvm-intel.ko
WARNING: arch/x86/kvm/kvm-amd: 'kvm_is_visible_gfn' exported twice. Previous export was in arch/x86/kvm/kvm-intel.ko
WARNING: arch/x86/kvm/kvm-amd: 'gfn_to_pfn' exported twice. Previous export was in arch/x86/kvm/kvm-intel.ko
WARNING: arch/x86/kvm/kvm-amd: 'kvm_read_guest' exported twice. Previous export was in arch/x86/kvm/kvm-intel.ko
WARNING: arch/x86/kvm/kvm-amd: 'kvm_write_guest' exported twice. Previous export was in arch/x86/kvm/kvm-intel.ko

---
 virt/kvm/eventfd.c  |  1 -
 virt/kvm/kvm_main.c | 65 ---------------------------------------------
 2 files changed, 66 deletions(-)
virt/kvm/eventfd.c +0 -1
index 67b6fc153e9c..4c1a8abd1458 100644
@@ -462,7 +462,6 @@ bool kvm_irq_has_notifier(struct kvm *kvm, unsigned irqchip, unsigned pin)
 
 	return false;
 }
-EXPORT_SYMBOL_GPL(kvm_irq_has_notifier);
 
 void kvm_notify_acked_gsi(struct kvm *kvm, int gsi)
 {
virt/kvm/kvm_main.c +0 -65

438 lines not on this page · raw message

#16 lore raw

PATCH 10/14 KVM: monolithic: x86: drop the kvm_pmu_ops structure

In reply to Andrea Arcangeli

Cleanup after the structure was finally left completely unused.

---
 arch/x86/include/asm/kvm_host.h |  3 ---
 arch/x86/kvm/pmu.h              | 20 --------------------
 arch/x86/kvm/pmu_amd.c          | 15 ---------------
 arch/x86/kvm/svm.c              |  1 -
 arch/x86/kvm/vmx/pmu_intel.c    | 15 ---------------
 arch/x86/kvm/vmx/vmx.c          |  2 --
 6 files changed, 56 deletions(-)
arch/x86/include/asm/kvm_host.h +0 -3
index 75affbf7861b..0481648852c0 100644
@@ -1348,9 +1348,6 @@ struct kvm_x86_ops {
 					   gfn_t offset, unsigned long mask);
 	int (*write_log_dirty)(struct kvm_vcpu *vcpu);
 
-	/* pmu operations of sub-arch */
-	const struct kvm_pmu_ops *pmu_ops;
-
 	/*
 	 * Architecture specific hooks for vCPU blocking due to
 	 * HLT instruction.
arch/x86/kvm/pmu.h +0 -20
index 82f07e3492df..c74d4ab30f66 100644
@@ -36,23 +36,6 @@ extern void kvm_x86_pmu_refresh(struct kvm_vcpu *vcpu);
 extern void kvm_x86_pmu_init(struct kvm_vcpu *vcpu);
 extern void kvm_x86_pmu_reset(struct kvm_vcpu *vcpu);
 
-struct kvm_pmu_ops {
-	unsigned (*find_arch_event)(struct kvm_pmu *pmu, u8 event_select,
-				    u8 unit_mask);
-	unsigned (*find_fixed_event)(int idx);
-	bool (*pmc_is_enabled)(struct kvm_pmc *pmc);
-	struct kvm_pmc *(*pmc_idx_to_pmc)(struct kvm_pmu *pmu, int pmc_idx);
-	struct kvm_pmc *(*msr_idx_to_pmc)(struct kvm_vcpu *vcpu, unsigned idx,
-					  u64 *mask);
-	int (*is_valid_msr_idx)(struct kvm_vcpu *vcpu, unsigned idx);
-	bool (*is_valid_msr)(struct kvm_vcpu *vcpu, u32 msr);
-	int (*get_msr)(struct kvm_vcpu *vcpu, u32 msr, u64 *data);
-	int (*set_msr)(struct kvm_vcpu *vcpu, struct msr_data *msr_info);
-	void (*refresh)(struct kvm_vcpu *vcpu);
-	void (*init)(struct kvm_vcpu *vcpu);
-	void (*reset)(struct kvm_vcpu *vcpu);
-};
-
 static inline u64 pmc_bitmask(struct kvm_pmc *pmc)
 {
 	struct kvm_pmu *pmu = pmc_to_pmu(pmc);
@@ -138,7 +121,4 @@ void kvm_pmu_destroy(struct kvm_vcpu *vcpu);
 int kvm_vm_ioctl_set_pmu_event_filter(struct kvm *kvm, void __user *argp);
 
 bool is_vmware_backdoor_pmc(u32 pmc_idx);
-
-extern struct kvm_pmu_ops intel_pmu_ops;
-extern struct kvm_pmu_ops amd_pmu_ops;
 #endif /* __KVM_X86_PMU_H */
arch/x86/kvm/pmu_amd.c +0 -15
index 7ea588023949..1b09ae337516 100644
@@ -300,18 +300,3 @@ void kvm_x86_pmu_reset(struct kvm_vcpu *vcpu)
 		pmc->counter = pmc->eventsel = 0;
 	}
 }
-
-struct kvm_pmu_ops amd_pmu_ops = {
-	.find_arch_event = kvm_x86_pmu_find_arch_event,
-	.find_fixed_event = kvm_x86_pmu_find_fixed_event,
-	.pmc_is_enabled = kvm_x86_pmu_pmc_is_enabled,
-	.pmc_idx_to_pmc = kvm_x86_pmu_pmc_idx_to_pmc,
-	.msr_idx_to_pmc = kvm_x86_pmu_msr_idx_to_pmc,
-	.is_valid_msr_idx = kvm_x86_pmu_is_valid_msr_idx,
-	.is_valid_msr = kvm_x86_pmu_is_valid_msr,
-	.get_msr = kvm_x86_pmu_get_msr,
-	.set_msr = kvm_x86_pmu_set_msr,
-	.refresh = kvm_x86_pmu_refresh,
-	.init = kvm_x86_pmu_init,
-	.reset = kvm_x86_pmu_reset,
-};
arch/x86/kvm/svm.c +0 -1
index 057ba1f8d7b3..50c57112c0ce 100644
@@ -7305,7 +7305,6 @@ static struct kvm_x86_ops svm_x86_ops __ro_after_init = {
 
 	.sched_in = kvm_x86_sched_in,
 
-	.pmu_ops = &amd_pmu_ops,
 	.deliver_posted_interrupt = kvm_x86_deliver_posted_interrupt,
 	.dy_apicv_has_pending_interrupt = kvm_x86_dy_apicv_has_pending_interrupt,
 	.update_pi_irte = kvm_x86_update_pi_irte,
arch/x86/kvm/vmx/pmu_intel.c +0 -15
index 758d6dbdbed2..530ca9942ecd 100644
@@ -356,18 +356,3 @@ void kvm_x86_pmu_reset(struct kvm_vcpu *vcpu)
 	pmu->fixed_ctr_ctrl = pmu->global_ctrl = pmu->global_status =
 		pmu->global_ovf_ctrl = 0;
 }
-
-struct kvm_pmu_ops intel_pmu_ops = {
-	.find_arch_event = kvm_x86_pmu_find_arch_event,
-	.find_fixed_event = kvm_x86_pmu_find_fixed_event,
-	.pmc_is_enabled = kvm_x86_pmu_pmc_is_enabled,
-	.pmc_idx_to_pmc = kvm_x86_pmu_pmc_idx_to_pmc,
-	.msr_idx_to_pmc = kvm_x86_pmu_msr_idx_to_pmc,
-	.is_valid_msr_idx = kvm_x86_pmu_is_valid_msr_idx,
-	.is_valid_msr = kvm_x86_pmu_is_valid_msr,
-	.get_msr = kvm_x86_pmu_get_msr,
-	.set_msr = kvm_x86_pmu_set_msr,
-	.refresh = kvm_x86_pmu_refresh,
-	.init = kvm_x86_pmu_init,
-	.reset = kvm_x86_pmu_reset,
-};
arch/x86/kvm/vmx/vmx.c +0 -2
index faccffc4709e..6e995e37a8c8 100644
@@ -7805,8 +7805,6 @@ static struct kvm_x86_ops vmx_x86_ops __ro_after_init = {
 	.pre_block = kvm_x86_pre_block,
 	.post_block = kvm_x86_post_block,
 
-	.pmu_ops = &intel_pmu_ops,
-
 	.update_pi_irte = kvm_x86_update_pi_irte,
 
 #ifdef CONFIG_X86_64
#17 lore raw

PATCH 11/14 KVM: x86: optimize more exit handlers in vmx.c

In reply to Andrea Arcangeli

Eliminate wasteful call/ret non RETPOLINE case and unnecessary fentry dynamic tracing hooking points.

---
 arch/x86/kvm/vmx/vmx.c | 30 +++++-------------------------
 1 file changed, 5 insertions(+), 25 deletions(-)
arch/x86/kvm/vmx/vmx.c +5 -25
index 6e995e37a8c8..de3ae2246205 100644
@@ -4589,7 +4589,7 @@ static int handle_exception_nmi(struct kvm_vcpu *vcpu)
 	return 0;
 }
 
-static int handle_external_interrupt(struct kvm_vcpu *vcpu)
+static __always_inline int handle_external_interrupt(struct kvm_vcpu *vcpu)
 {
 	++vcpu->stat.irq_exits;
 	return 1;
@@ -4860,21 +4860,6 @@ void kvm_x86_set_dr7(struct kvm_vcpu *vcpu, unsigned long val)
 	vmcs_writel(GUEST_DR7, val);
 }
 
-static int handle_cpuid(struct kvm_vcpu *vcpu)
-{
-	return kvm_emulate_cpuid(vcpu);
-}
-
-static int handle_rdmsr(struct kvm_vcpu *vcpu)
-{
-	return kvm_emulate_rdmsr(vcpu);
-}
-
-static int handle_wrmsr(struct kvm_vcpu *vcpu)
-{
-	return kvm_emulate_wrmsr(vcpu);
-}
-
 static int handle_tpr_below_threshold(struct kvm_vcpu *vcpu)
 {
 	kvm_apic_update_ppr(vcpu);
@@ -4891,11 +4876,6 @@ static int handle_interrupt_window(struct kvm_vcpu *vcpu)
 	return 1;
 }
 
-static int handle_halt(struct kvm_vcpu *vcpu)
-{
-	return kvm_emulate_halt(vcpu);
-}
-
 static int handle_vmcall(struct kvm_vcpu *vcpu)
 {
 	return kvm_emulate_hypercall(vcpu);
@@ -5487,11 +5467,11 @@ static int (*kvm_vmx_exit_handlers[])(struct kvm_vcpu *vcpu) = {
 	[EXIT_REASON_IO_INSTRUCTION]          = handle_io,
 	[EXIT_REASON_CR_ACCESS]               = handle_cr,
 	[EXIT_REASON_DR_ACCESS]               = handle_dr,
-	[EXIT_REASON_CPUID]                   = handle_cpuid,
-	[EXIT_REASON_MSR_READ]                = handle_rdmsr,
-	[EXIT_REASON_MSR_WRITE]               = handle_wrmsr,
+	[EXIT_REASON_CPUID]                   = kvm_emulate_cpuid,
+	[EXIT_REASON_MSR_READ]                = kvm_emulate_rdmsr,
+	[EXIT_REASON_MSR_WRITE]               = kvm_emulate_wrmsr,
 	[EXIT_REASON_PENDING_INTERRUPT]       = handle_interrupt_window,
-	[EXIT_REASON_HLT]                     = handle_halt,
+	[EXIT_REASON_HLT]                     = kvm_emulate_halt,
 	[EXIT_REASON_INVD]		      = handle_invd,
 	[EXIT_REASON_INVLPG]		      = handle_invlpg,
 	[EXIT_REASON_RDPMC]                   = handle_rdpmc,
#18 lore raw

PATCH 12/14 KVM: retpolines: x86: eliminate retpoline from vmx.c exit handlers

In reply to Andrea Arcangeli

It's enough to check the exit value and issue a direct call to avoid the retpoline for all the common vmexit reasons.

Reducing this list to only EXIT_REASON_MSR_WRITE, EXIT_REASON_PREEMPTION_TIMER, EXIT_REASON_EPT_MISCONFIG, EXIT_REASON_IO_INSTRUCTION increases the computation time of the hrtimer guest testcase on Haswell i5-4670T CPU @ 2.30GHz by 7% with the default spectre v2 mitigation enabled in the host and guest. On skylake as opposed there's no measurable difference with the short list. To put things in prospective on Haswell the same hrtimer workload (note: it never calls cpuid and it never attempts to trigger more vmexit on purpose) in guest takes 16.3% longer to compute on upstream KVM running in the host than with the KVM mono v1 patchset applied to the host kernel, while on skylake the same takes only 5.4% more time (both with the default mitigations enabled in guest and host).

It's also unclear why EXIT_REASON_IO_INSTRUCTION should be included.

Of course CONFIG_RETPOLINE already forbids gcc not to do indirect jumps while compiling all switch() statements, however switch() would still allow the compiler to bisect the value, however it seems to run slower if something and the reason is that it's better to prioritize and do the minimal possible number of checks for the most common vmexit.

The halt and pause loop exiting may be slow paths from the point of the guest, but not necessarily so from the point of the host. There can be a flood of halt exit reasons (in fact that's why the cpuidle guest haltpoll support was recently merged and we can't rely on it here because there are older kernels and other OS that must also perform optimally). All it takes is a pipe ping pong with a different host CPU and the host CPUs running at full capacity.

The same consideration applies to the pause loop exiting exit reason, if there's heavy host overcommit that collides heavily in a spinlock the same may happen.

In the common case of a fully idle host, the halt and pause loop exiting can't help, but adding them doesn't hurt the common case and the expectation here is that if they would ever become measurable, it would be because they are increasing (and not decreasing) performance.

---
 arch/x86/kvm/vmx/vmx.c | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/vmx.c +22 -2
index de3ae2246205..2bd57a7d2be1 100644
@@ -5846,9 +5846,29 @@ int kvm_x86_handle_exit(struct kvm_vcpu *vcpu)
 	}
 
 	if (exit_reason < kvm_vmx_max_exit_handlers
-	    && kvm_vmx_exit_handlers[exit_reason])
+	    && kvm_vmx_exit_handlers[exit_reason]) {
+#ifdef CONFIG_RETPOLINE
+		if (exit_reason == EXIT_REASON_MSR_WRITE)
+			return kvm_emulate_wrmsr(vcpu);
+		else if (exit_reason == EXIT_REASON_PREEMPTION_TIMER)
+			return handle_preemption_timer(vcpu);
+		else if (exit_reason == EXIT_REASON_PENDING_INTERRUPT)
+			return handle_interrupt_window(vcpu);
+		else if (exit_reason == EXIT_REASON_EXTERNAL_INTERRUPT)
+			return handle_external_interrupt(vcpu);
+		else if (exit_reason == EXIT_REASON_HLT)
+			return kvm_emulate_halt(vcpu);
+		else if (exit_reason == EXIT_REASON_PAUSE_INSTRUCTION)
+			return handle_pause(vcpu);
+		else if (exit_reason == EXIT_REASON_MSR_READ)
+			return kvm_emulate_rdmsr(vcpu);
+		else if (exit_reason == EXIT_REASON_CPUID)
+			return kvm_emulate_cpuid(vcpu);
+		else if (exit_reason == EXIT_REASON_EPT_MISCONFIG)
+			return handle_ept_misconfig(vcpu);
+#endif
 		return kvm_vmx_exit_handlers[exit_reason](vcpu);
-	else {
+	} else {
 		vcpu_unimpl(vcpu, "vmx: unexpected exit reason 0x%x\n",
 				exit_reason);
 		dump_vmcs();
maintainer
#19 lore raw

Re: [PATCH 12/14] KVM: retpolines: x86: eliminate retpoline from vmx.c exit handlers

Quoted 14 lines · Andrea Arcangeli

Reducing this list to only EXIT_REASON_MSR_WRITE, EXIT_REASON_PREEMPTION_TIMER, EXIT_REASON_EPT_MISCONFIG, EXIT_REASON_IO_INSTRUCTION increases the computation time of the hrtimer guest testcase on Haswell i5-4670T CPU @ 2.30GHz by 7% with the default spectre v2 mitigation enabled in the host and guest. On skylake as opposed there's no measurable difference with the short list. To put things in prospective on Haswell the same hrtimer workload (note: it never calls cpuid and it never attempts to trigger more vmexit on purpose) in guest takes 16.3% longer to compute on upstream KVM running in the host than with the KVM mono v1 patchset applied to the host kernel, while on skylake the same takes only 5.4% more time (both with the default mitigations enabled in guest and host).

Quoted 1 line

It's also unclear why EXIT_REASON_IO_INSTRUCTION should be included.

If you're including EXIT_REASON_EPT_MISCONFIG (MMIO access) then you should include EXIT_REASON_IO_INSTRUCTION too. Depending on the devices that are in the guest, the doorbell register might be MMIO or PIO.

Quoted 8 lines
arch/x86/kvm/vmx/vmx.c +8 -0
+		if (exit_reason == EXIT_REASON_MSR_WRITE)
+			return kvm_emulate_wrmsr(vcpu);
+		else if (exit_reason == EXIT_REASON_PREEMPTION_TIMER)
+			return handle_preemption_timer(vcpu);
+		else if (exit_reason == EXIT_REASON_PENDING_INTERRUPT)
+			return handle_interrupt_window(vcpu);
+		else if (exit_reason == EXIT_REASON_EXTERNAL_INTERRUPT)
+			return handle_external_interrupt(vcpu);

arch/x86/kvm/vmx/vmx.c int kvm_x86_handle_exit(struct kvm_vcpu *vcpu) · line 5868

Quoted 10 lines
arch/x86/kvm/vmx/vmx.c +10 -0
+		else if (exit_reason == EXIT_REASON_HLT)
+			return kvm_emulate_halt(vcpu);
+		else if (exit_reason == EXIT_REASON_PAUSE_INSTRUCTION)
+			return handle_pause(vcpu);
+		else if (exit_reason == EXIT_REASON_MSR_READ)
+			return kvm_emulate_rdmsr(vcpu);
+		else if (exit_reason == EXIT_REASON_CPUID)
+			return kvm_emulate_cpuid(vcpu);
+		else if (exit_reason == EXIT_REASON_EPT_MISCONFIG)
+			return handle_ept_misconfig(vcpu);

So, the difference between my suggested list (which I admit is just based on conjecture, not benchmarking) is that you add EXIT_REASON_PAUSE_INSTRUCTION, EXIT_REASON_PENDING_INTERRUPT, EXIT_REASON_EXTERNAL_INTERRUPT, EXIT_REASON_HLT, EXIT_REASON_MSR_READ, EXIT_REASON_CPUID.

Which of these make a difference for the hrtimer testcase? It's of course totally fine to use benchmarks to prove that my intuition was bad---but you must also use them to show why your intuition is right. :)

Paolo

#20 lore raw

Re: [PATCH 12/14] KVM: retpolines: x86: eliminate retpoline from vmx.c exit handlers

Quoted 3 lines · Paolo Bonzini

If you're including EXIT_REASON_EPT_MISCONFIG (MMIO access) then you should include EXIT_REASON_IO_INSTRUCTION too. Depending on the devices that are in the guest, the doorbell register might be MMIO or PIO.

The fact outb/inb devices exists isn't the question here. The question you should clarify is: which of the PIO devices is performance critical as much as MMIO with virtio/vhost? I mean even on real hardware those devices aren't performance critical. I didn't run into PIO drivers with properly configured guests.

Quoted 9 lines

So, the difference between my suggested list (which I admit is just based on conjecture, not benchmarking) is that you add EXIT_REASON_PAUSE_INSTRUCTION, EXIT_REASON_PENDING_INTERRUPT, EXIT_REASON_EXTERNAL_INTERRUPT, EXIT_REASON_HLT, EXIT_REASON_MSR_READ, EXIT_REASON_CPUID.

Which of these make a difference for the hrtimer testcase? It's of course totally fine to use benchmarks to prove that my intuition was bad---but you must also use them to show why your intuition is right. :)

The hrtimer flood hits on this:

           MSR_WRITE     338793    56.54%     5.51%      0.33us     34.44us      0.44us ( +-   0.20% )
   PENDING_INTERRUPT     168431    28.11%     2.52%      0.36us     32.06us      0.40us ( +-   0.28% )
    PREEMPTION_TIMER      91723    15.31%     1.32%      0.34us     30.51us      0.39us ( +-   0.41% )
  EXTERNAL_INTERRUPT        234     0.04%     0.00%      0.25us      5.53us      0.43us ( +-   5.67% )
                 HLT         65     0.01%    90.64%      0.49us 319933.79us  37562.71us ( +-  21.68% )
            MSR_READ          6     0.00%     0.00%      0.67us      1.96us      1.06us ( +-  17.97% )
       EPT_MISCONFIG          6     0.00%     0.01%      3.09us    105.50us     26.76us ( +-  62.10% )

PENDING_INTERRUPT is the big missing thing in your list. It probably accounts for the bulk of slowdown from your list. However I could imagine other loads with higher external interrupt/hlt/rdmsr than the hrtimer one so I didn't drop those. Other loads are hitting on a flood of HLT and from host standpoint it's no a slow path. Not all OS have the cpuidle haltpoll governor to mitigate the HLT frequency.

I'm pretty sure HLT/EXTERNAL_INTERRUPT/PENDING_INTERRUPT should be included.

The least useful are PAUSE, CPUID and MSR_READ, we could considering dropping some of those (in the short term cpuid helps for benchmarking to more accurately measure the performance improvement of not hitting the retpoline there). I simply could imagine some load hitting frequently on those too so I didn't drop them.

I also wonder if VMCALL should be added, certain loads hit on fairly frequent VMCALL, but none of the one I benchmarked.

maintainer
#21 lore raw

Re: [PATCH 12/14] KVM: retpolines: x86: eliminate retpoline from vmx.c exit handlers

Quoted 8 lines · Andrea Arcangeli
Quoted 3 lines · Paolo Bonzini

If you're including EXIT_REASON_EPT_MISCONFIG (MMIO access) then you should include EXIT_REASON_IO_INSTRUCTION too. Depending on the devices that are in the guest, the doorbell register might be MMIO or PIO.

The fact outb/inb devices exists isn't the question here. The question you should clarify is: which of the PIO devices is performance critical as much as MMIO with virtio/vhost?

virtio 0.9 uses PIO.

Quoted 1 line

I mean even on real hardware those devices aren't performance critical.

On virtual machines they're actually faster than MMIO because they don't need to go through page table walks.

Quoted 20 lines
Quoted 9 lines

So, the difference between my suggested list (which I admit is just based on conjecture, not benchmarking) is that you add EXIT_REASON_PAUSE_INSTRUCTION, EXIT_REASON_PENDING_INTERRUPT, EXIT_REASON_EXTERNAL_INTERRUPT, EXIT_REASON_HLT, EXIT_REASON_MSR_READ, EXIT_REASON_CPUID.

Which of these make a difference for the hrtimer testcase? It's of course totally fine to use benchmarks to prove that my intuition was bad---but you must also use them to show why your intuition is right. :)

The hrtimer flood hits on this:

           MSR_WRITE     338793    56.54%     5.51%      0.33us     34.44us      0.44us ( +-   0.20% )
   PENDING_INTERRUPT     168431    28.11%     2.52%      0.36us     32.06us      0.40us ( +-   0.28% )
    PREEMPTION_TIMER      91723    15.31%     1.32%      0.34us     30.51us      0.39us ( +-   0.41% )
  EXTERNAL_INTERRUPT        234     0.04%     0.00%      0.25us      5.53us      0.43us ( +-   5.67% )
                 HLT         65     0.01%    90.64%      0.49us 319933.79us  37562.71us ( +-  21.68% )
            MSR_READ          6     0.00%     0.00%      0.67us      1.96us      1.06us ( +-  17.97% )
       EPT_MISCONFIG          6     0.00%     0.01%      3.09us    105.50us     26.76us ( +-  62.10% )
Quoted 2 lines

PENDING_INTERRUPT is the big missing thing in your list. It probably accounts for the bulk of slowdown from your list.

Makes sense.

Quoted 2 lines

However I could imagine other loads with higher external interrupt/hlt/rdmsr than the hrtimer one so I didn't drop those.

External interrupts should only tick at 1 Hz on nohz_full kernels, and even at 1000 Hz (if physical CPUs are not isolated) it should not really matter. We can include it since it has such a short handler so the cost of retpolines is in % much more than other exits.

HLT is certainly a slow path, the guest only invokes if things such as NAPI interrupt mitigation have failed. As long as the guest stays in halted state for a microsecond or so, the cost of retpoline will all but disappear.

RDMSR again shouldn't be there, guests sometimes read the PMTimer (which is an I/O port) or TSC but for example do not really ever read the APIC TMCCT.

Quoted 4 lines

I'm pretty sure HLT/EXTERNAL_INTERRUPT/PENDING_INTERRUPT should be included.
I also wonder if VMCALL should be added, certain loads hit on fairly frequent VMCALL, but none of the one I benchmarked.

I agree for external interrupt and pending interrupt, and VMCALL is fine too. In addition I'd add I/O instructions which are useful for some guests and also for benchmarking (e.g. vmexit.flat has both IN and OUT tests).

Paolo

#22 lore raw

Re: [PATCH 12/14] KVM: retpolines: x86: eliminate retpoline from vmx.c exit handlers

Quoted 11 lines · Paolo Bonzini
Quoted 8 lines · Andrea Arcangeli
Quoted 3 lines · Paolo Bonzini

If you're including EXIT_REASON_EPT_MISCONFIG (MMIO access) then you should include EXIT_REASON_IO_INSTRUCTION too. Depending on the devices that are in the guest, the doorbell register might be MMIO or PIO.

The fact outb/inb devices exists isn't the question here. The question you should clarify is: which of the PIO devices is performance critical as much as MMIO with virtio/vhost?

virtio 0.9 uses PIO.

0.9 is a 12 years old protocol replaced several years ago. Anybody who needs high performance won't be running it, the others can't perform well to begin with, so I'm not sure exactly how it's relevant in this microoptimization context. We're not optimizing for emulated devices or other old stuff either.

Quoted 2 lines

On virtual machines they're actually faster than MMIO because they don't need to go through page table walks.

And how does it help that they're faster if current virtio stopped using them and nothing else recent uses PIO?

Quoted 1 line

HLT is certainly a slow path, the guest only invokes if things such as

Your idea that HLT is a certainly is a slow path is only correct if you assume the host is IDLE, but the host is never idle if you use virt for consolidation.

From the point of view of the host, HLT is like every other vmexit.

Quoted 3 lines

NAPI interrupt mitigation have failed. As long as the guest stays in halted state for a microsecond or so, the cost of retpoline will all but disappear.

The only thing that matters is the number of HLT vmexit per second and you just need to measure the number of HLT vmexits to tell it's needed.

I've several workloads including eBPF tracing, not related to interrupts (that in turn cannot be mitigated by NAPI) that schedule frequently and hit 100k+ of HLT vmexits per second and the host is all but idle. There's no need of hardware interrupt to wake up tasks and schedule in the guest, scheduler IPIs and timers are more than enough.

The only thing that can mitigate that is the cpuidle haltpoll driver, but it hit upstream a few months ago, all most recent enterprise guest OS won't have it yet.

All it matters is how many vmexits per second there are, everything else including "why" they happen and what those vmexists means for the guest, is irrelevant, or it would be relevant only if the host was guaranteed to be idle but there's no such guarantee.

If the host is using all idle CPUs to compute in the background (i.e. building the kernel) with SCHED_IDLE the HLT retpoline cost will not be any different than any other vmexit retpoline cost and easy +100k HLT exit per second certainly puts it in the measurable territory.

Here's a random example:

             VM-EXIT    Samples  Samples%     Time%    Min Time    Max Time         Avg time 

                 HLT     101128    75.33%    99.66%      0.43us 901000.66us    310.88us ( +-   8.46% )
              VMCALL      14089    10.50%     0.10%      1.32us     84.99us      2.14us ( +-   0.90% )
           MSR_WRITE       8246     6.14%     0.03%      0.33us     32.79us      1.05us ( +-   1.51% )
       EPT_VIOLATION       6312     4.70%     0.18%      0.50us  26426.07us      8.90us ( +-  48.58% )
    PREEMPTION_TIMER       1730     1.29%     0.01%      0.55us     26.81us      1.60us ( +-   3.48% )
  EXTERNAL_INTERRUPT       1329     0.99%     0.03%      0.27us    944.88us      6.04us ( +-  20.52% )
       EPT_MISCONFIG        982     0.73%     0.01%      0.42us    137.68us      2.05us ( +-   9.88% )
   PENDING_INTERRUPT        308     0.23%     0.00%      0.44us      4.32us      0.73us ( +-   2.57% )
   PAUSE_INSTRUCTION         58     0.04%     0.00%      0.32us     18.55us      1.48us ( +-  23.12% )
            MSR_READ         35     0.03%     0.00%      0.78us      5.55us      2.07us ( +-  10.74% )
               CPUID         24     0.02%     0.00%      0.27us      2.20us      0.59us ( +-  13.43% )
# careful despite the verifier promise that eBPF shouldn't be kernel
# crashing this may be kernel crashing because there's no verifier at
# all that verifies that the eBPF function calls available depending
# on the hooking point can actually be invoked from the kernel hooking
# points they're invoked from. this is why I tested it in a VM
bpftrace -e 'kprobe:*interrupt* { @ = count() }'

Other example with a pipe loop that just bounces a byte across a pipe with two processes:

             VM-EXIT    Samples  Samples%     Time%    Min Time    Max Time         Avg time 

           MSR_WRITE     498945    80.49%     4.10%      0.33us     42.73us      0.44us ( +-   0.12% )
                 HLT     118474    19.11%    95.88%      0.33us 707693.05us     43.56us ( +-  24.23% )
    PREEMPTION_TIMER       1004     0.16%     0.01%      0.38us     25.47us      0.67us ( +-   5.69% )
   PENDING_INTERRUPT        894     0.14%     0.01%      0.37us     20.98us      0.49us ( +-   4.94% )
  EXTERNAL_INTERRUPT        518     0.08%     0.00%      0.26us     20.59us      0.51us ( +-   8.09% )
            MSR_READ          8     0.00%     0.00%      0.66us      1.37us      0.92us ( +-   9.19% )
       EPT_MISCONFIG          6     0.00%     0.00%      3.18us     32.71us     12.60us ( +-  43.58% )
   PAUSE_INSTRUCTION          3     0.00%     0.00%      0.59us      1.69us      1.07us ( +-  30.38% )

We wouldn't need to apply the cpuidle-haltpoll driver if HLT wasn't such a frequent vmexit that deserves to have its retpoline cost, not multiplied by 100000 times per second.

Over time if everything will turn out to use the cpuidle-haltpoll driver by default (that however can increase the host CPU usage on laptops) we can consider removing the HLT optimization, we're not remotely there yet.

Quoted 3 lines

RDMSR again shouldn't be there, guests sometimes read the PMTimer (which is an I/O port) or TSC but for example do not really ever read the APIC TMCCT.

We can try to drop RDMSR, and see if it's measurable. I already tried to re-add some of those retpolines but it was slower and this was the fastest combination that I got, I don't recall if I tried with RDMSR and PAUSE alone but I can try again.

Quoted 9 lines
Quoted 4 lines

I'm pretty sure HLT/EXTERNAL_INTERRUPT/PENDING_INTERRUPT should be included.
I also wonder if VMCALL should be added, certain loads hit on fairly frequent VMCALL, but none of the one I benchmarked.

I agree for external interrupt and pending interrupt, and VMCALL is fine too. In addition I'd add I/O instructions which are useful for some guests and also for benchmarking (e.g. vmexit.flat has both IN and OUT tests).

Isn't it faster to use cpuid for benchmarking? I mean we don't want to pay for more than one branch for benchmarking (even cpuid is questionable in the long term, but for now it's handy to have), and unlike inb/outb, cpuid runs occasionally in all real life workloads (including in guest userland) so between inb/outb, I'd rather prefer to use cpuid as the benchmark vector because at least it has a chance to help real workloads a bit too.

Thanks,
Andrea

maintainer
#23 lore raw

Re: [PATCH 12/14] KVM: retpolines: x86: eliminate retpoline from vmx.c exit handlers

Quoted 7 lines · Andrea Arcangeli
Quoted 6 lines · Paolo Bonzini
Quoted 5 lines · Andrea Arcangeli
Quoted 3 lines · Paolo Bonzini

If you're including EXIT_REASON_EPT_MISCONFIG (MMIO access) then you should include EXIT_REASON_IO_INSTRUCTION too. Depending on the devices that are in the guest, the doorbell register might be MMIO or PIO.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

The fact outb/inb devices exists isn't the question here. The question you should clarify is: which of the PIO devices is performance critical as much as MMIO with virtio/vhost?

virtio 0.9 uses PIO.

0.9 is a 12 years old protocol replaced several years ago.

Oh come on. 0.9 is not 12-years old. virtio 1.0 is 3.5 years old (March 2016). Anything older than 2017 is going to use 0.9.

Quoted 10 lines

Your idea that HLT is a certainly is a slow path is only correct if you assume the host is IDLE, but the host is never idle if you use virt for consolidation.

I've several workloads including eBPF tracing, not related to interrupts (that in turn cannot be mitigated by NAPI) that schedule frequently and hit 100k+ of HLT vmexits per second and the host is all but idle. There's no need of hardware interrupt to wake up tasks and schedule in the guest, scheduler IPIs and timers are more than enough.

Quoted 4 lines

All it matters is how many vmexits per second there are, everything else including "why" they happen and what those vmexists means for the guest, is irrelevant, or it would be relevant only if the host was guaranteed to be idle but there's no such guarantee.

Your tables give:

Samples Samples% Time% Min Time Max time Avg time

HLT     101128    75.33%    99.66%    0.43us    901000.66us    310.88us
HLT     118474    19.11%    95.88%    0.33us    707693.05us    43.56us

If "avg time" means the average time to serve an HLT vmexit, I don't understand how you can have an average time of 0.3ms (1/3000th of a second) and 100000 samples per second. Can you explain that to me?

Anyway, if the average time is indeed 310us and 43us, it is orders of magnitude more than the time spent executing a retpoline. That time will be spent in an indirect branch miss (retpoline) instead of doing while(!kvm_vcpu_check_block()), but it doesn't change anything.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

I'm pretty sure HLT/EXTERNAL_INTERRUPT/PENDING_INTERRUPT should be included.
I also wonder if VMCALL should be added, certain loads hit on fairly frequent VMCALL, but none of the one I benchmarked.

Quoted 8 lines
Quoted 4 lines

I agree for external interrupt and pending interrupt, and VMCALL is fine too. In addition I'd add I/O instructions which are useful for some guests and also for benchmarking (e.g. vmexit.flat has both IN and OUT tests).

Isn't it faster to use cpuid for benchmarking? I mean we don't want to pay for more than one branch for benchmarking (even cpuid is questionable in the long term, but for now it's handy to have),

outl is more or less the same as cpuid and vmcall. You can measure it with vmexit.flat. inl is slower.

Quoted 4 lines

and unlike inb/outb, cpuid runs occasionally in all real life workloads (including in guest userland) so between inb/outb, I'd rather prefer to use cpuid as the benchmark vector because at least it has a chance to help real workloads a bit too.

Again: what is the real workload that does thousands of CPUIDs per second?

Paolo

#24 lore raw

Re: [PATCH 12/14] KVM: retpolines: x86: eliminate retpoline from vmx.c exit handlers

Quoted 2 lines · Paolo Bonzini

Oh come on. 0.9 is not 12-years old. virtio 1.0 is 3.5 years old (March 2016). Anything older than 2017 is going to use 0.9.

Sorry if I got the date wrong, but still I don't see the point in optimizing for legacy virtio. I can't justify forcing everyone to execute that additional branch for inb/outb, in the attempt to make legacy virtio faster that nobody should use in combination with bleeding edge KVM in the host.

Quoted 9 lines

Your tables give:

Samples Samples% Time% Min Time Max time Avg time

HLT     101128    75.33%    99.66%    0.43us    901000.66us    310.88us
HLT     118474    19.11%    95.88%    0.33us    707693.05us    43.56us

If "avg time" means the average time to serve an HLT vmexit, I don't understand how you can have an average time of 0.3ms (1/3000th of a second) and 100000 samples per second. Can you explain that to me?

I described it wrong, the bpftrace record was a sleep 5, not a sleep
1. The pipe loop was sure a sleep 1.

I just wanted to show how even on things where you wouldn't even expected to get HLT like the bpftrace that is pure guest CPU load, you still get 100k of them (over 5 sec).

The issue is that in production you get a flood more of those with hundred of CPUs, so the exact number doesn't move the needle.

Quoted 4 lines

Anyway, if the average time is indeed 310us and 43us, it is orders of magnitude more than the time spent executing a retpoline. That time will be spent in an indirect branch miss (retpoline) instead of doing while(!kvm_vcpu_check_block()), but it doesn't change anything.

Doesn't cpuidle haltpoll disable that loop? Ideally there should be HLT vmexits then but I don't know how much fewer. This just needs to be frequent enough that the branch cost pay itself off, but the sure thing is that HLT vmexit will not go away unless you execute mwait in guest mode by isolating the CPU in the host.

Quoted 1 line

Again: what is the real workload that does thousands of CPUIDs per second?

None, but there are always background CPUID vmexits while there are never inb/outb vmexits.

So the cpuid retpoline removal has a slight chance to pay for the cost of the branch, the inb/outb retpoline removal cannot pay off the cost of the branch.

This is why I prefer cpuid as benchmark gadget for the short term unless inb/outb offers other benchmark related benefits.

Thanks,
Andrea

maintainer
#25 lore raw

Re: [PATCH 12/14] KVM: retpolines: x86: eliminate retpoline from vmx.c exit handlers

Quoted 9 lines · Andrea Arcangeli
Quoted 2 lines · Paolo Bonzini

Oh come on. 0.9 is not 12-years old. virtio 1.0 is 3.5 years old (March 2016). Anything older than 2017 is going to use 0.9.

Sorry if I got the date wrong, but still I don't see the point in optimizing for legacy virtio. I can't justify forcing everyone to execute that additional branch for inb/outb, in the attempt to make legacy virtio faster that nobody should use in combination with bleeding edge KVM in the host.

Yet you would add CPUID to the list even though it is not even there in your benchmarks, and is *never* invoked in a hot path by *any* sane program? Some OSes have never gotten virtio 1.0 drivers. OpenBSD only got it earlier this year.

Quoted 6 lines
Quoted 6 lines

Your tables give:

Samples Samples% Time% Min Time Max time Avg time

HLT     101128    75.33%    99.66%    0.43us    901000.66us    310.88us
HLT     118474    19.11%    95.88%    0.33us    707693.05us    43.56us
Quoted 6 lines
Quoted 3 lines

If "avg time" means the average time to serve an HLT vmexit, I don't understand how you can have an average time of 0.3ms (1/3000th of a second) and 100000 samples per second. Can you explain that to me?

I described it wrong, the bpftrace record was a sleep 5, not a sleep
1. The pipe loop was sure a sleep 1.

It still doesn't add up. 0.3ms / 5 is 1/15000th of a second; 43us is 1/25000th of a second. Do you have multiple vCPU perhaps?

Quoted 5 lines

The issue is that in production you get a flood more of those with hundred of CPUs, so the exact number doesn't move the needle. This just needs to be frequent enough that the branch cost pay itself off, but the sure thing is that HLT vmexit will not go away unless you execute mwait in guest mode by isolating the CPU in the host.

The number of vmexits doesn't count (for HLT). What counts is how long they take to be serviced, and as long as it's 1us or more the optimization is pointless.

Consider these pictures

         w/o optimization                   with optimization
         ----------------------             -------------------------
0us      vmexit                             vmexit
500ns    retpoline                          call vmexit handler directly
600ns    retpoline                          kvm_vcpu_check_block()
700ns    retpoline                          kvm_vcpu_check_block()
800ns    kvm_vcpu_check_block()             kvm_vcpu_check_block()
900ns    kvm_vcpu_check_block()             kvm_vcpu_check_block()

...
39900ns kvm_vcpu_check_block() kvm_vcpu_check_block()

<interrupt arrives>

40000ns kvm_vcpu_check_block() kvm_vcpu_check_block()

Unless the interrupt arrives exactly in the few nanoseconds that it takes to execute the retpoline, a direct handling of HLT vmexits makes *absolutely no difference*.

Quoted 8 lines
Quoted 1 line

Again: what is the real workload that does thousands of CPUIDs per second?

None, but there are always background CPUID vmexits while there are never inb/outb vmexits.

So the cpuid retpoline removal has a slight chance to pay for the cost of the branch, the inb/outb retpoline removal cannot pay off the cost of the branch.

Please stop considering only the exact configuration of your benchmarks.
There are known, valid configurations where outb is a very hot vmexit.

Thanks,

Paolo

#26 lore raw

Re: [PATCH 12/14] KVM: retpolines: x86: eliminate retpoline from vmx.c exit handlers

Quoted 2 lines · Paolo Bonzini

Yet you would add CPUID to the list even though it is not even there in your benchmarks, and is *never* invoked in a hot path by *any* sane

I justified CPUID as a "short term" benchmark gadget, it's one of those it shouldn't be a problem at all to remove, I couldn't possibly be against removing it. I only pointed out the fact cpuid on any modern linux guest is going to run more frequently than any inb/outb so if I had to pick a benchmark gadget, that remains my favorite one.

Quoted 2 lines

program? Some OSes have never gotten virtio 1.0 drivers. OpenBSD only got it earlier this year.

If the target is an optimization to a cranky OS that can't upgrade virtio to obtain the full performance benefit from the retpoline removal too (I don't know the specifics by just reading the above) then it's a better argument. At least it sounds fair enough not to unfair penalize the cranky OS forced to run obsolete protocols that nobody can update or has the time to update.

I mean, until you said there's some OS that cannot upgrade to virtio 1.0, I thought it was perfectly fine to say "if you want to run a guest with the full benefit of virtio 1.0 on KVM, you should upgrade to virtio 1.0 and not stick to whatever 3 year old protocol, then also the inb/outb retpoline will go away if you upgrade the host because the inb/outb will go away in the first place".

Quoted 2 lines

It still doesn't add up. 0.3ms / 5 is 1/15000th of a second; 43us is 1/25000th of a second. Do you have multiple vCPU perhaps?

Why would I run any test on UP guests? Rather then spending time doing the math on my results, it's probably quicker that you run it yourself:

https://lkml.kernel.org/r/[email protected]/

Marcelo should have better reproducers for frequent HLT that is a real workload we have to pass, I reported the first two random things I had around that reported fairly frequent HLT. The pipe loop load is similar to local network I/O.

Quoted 17 lines

The number of vmexits doesn't count (for HLT). What counts is how long they take to be serviced, and as long as it's 1us or more the optimization is pointless.

Consider these pictures

         w/o optimization                   with optimization
         ----------------------             -------------------------
0us      vmexit                             vmexit
500ns    retpoline                          call vmexit handler directly
600ns    retpoline                          kvm_vcpu_check_block()
700ns    retpoline                          kvm_vcpu_check_block()
800ns    kvm_vcpu_check_block()             kvm_vcpu_check_block()
900ns    kvm_vcpu_check_block()             kvm_vcpu_check_block()

...
39900ns kvm_vcpu_check_block() kvm_vcpu_check_block()

Quoted 9 lines

<interrupt arrives>

40000ns kvm_vcpu_check_block() kvm_vcpu_check_block()

Unless the interrupt arrives exactly in the few nanoseconds that it takes to execute the retpoline, a direct handling of HLT vmexits makes *absolutely no difference*.

You keep focusing on what happens if the host is completely idle (in which case guest HLT is a slow path) and you keep ignoring the case that the host isn't completely idle (in which case guest HLT is not a slow path).

Please note the single_task_running() check which immediately breaks the kvm_vcpu_check_block() loop if there's even a single other task that can be scheduled in the runqueue of the host CPU.

What happen when the host is not idle is quoted below:

         w/o optimization                   with optimization
         ----------------------             -------------------------
0us      vmexit                             vmexit
500ns    retpoline                          call vmexit handler directly
600ns    retpoline                          kvm_vcpu_check_block()
700ns    retpoline                          schedule()

800ns kvm_vcpu_check_block()
900ns schedule()
...

Disclaimer: the numbers on the left are arbitrary and I just cut and pasted them from yours, no idea how far off they are.

To be clear, I would find it very reasonable to be requested to proof the benefit of the HLT optimization with benchmarks specifics for that single one liner, but until then, the idea that we can drop the retpoline optimization from the HLT vmexit by just thinking about it, still doesn't make sense to me, because by thinking about it I come to the opposite conclusion.

The lack of single_task_running() in the guest driver is also why the guest cpuidle haltpoll risks to waste some CPU with host overcommit or with the host loaded at full capacity and why we may not assume it to be universally enabled.

Thanks,
Andrea

maintainer
#27 lore raw

Re: [PATCH 12/14] KVM: retpolines: x86: eliminate retpoline from vmx.c exit handlers

Quoted 5 lines · Andrea Arcangeli
Quoted 2 lines

It still doesn't add up. 0.3ms / 5 is 1/15000th of a second; 43us is 1/25000th of a second. Do you have multiple vCPU perhaps?

Why would I run any test on UP guests? Rather then spending time doing the math on my results, it's probably quicker that you run it yourself:

I don't know, but if you don't say how many vCPUs you have, I cannot do the math and review the patch.

Quoted 16 lines
Quoted 3 lines

The number of vmexits doesn't count (for HLT). What counts is how long they take to be serviced, and as long as it's 1us or more the optimization is pointless.

Please note the single_task_running() check which immediately breaks the kvm_vcpu_check_block() loop if there's even a single other task that can be scheduled in the runqueue of the host CPU.

What happen when the host is not idle is quoted below:

         w/o optimization                   with optimization
         ----------------------             -------------------------
0us      vmexit                             vmexit
500ns    retpoline                          call vmexit handler directly
600ns    retpoline                          kvm_vcpu_check_block()
700ns    retpoline                          schedule()
Quoted 6 lines

800ns kvm_vcpu_check_block()
900ns schedule()
...

Disclaimer: the numbers on the left are arbitrary and I just cut and pasted them from yours, no idea how far off they are.

Yes, of course. But the idea is the same: yes, because of the retpoline you run the guest for perhaps 300ns more before schedule()ing, but does that really matter? 300ns * 20000 times/second is a 0.6% performance impact, and 300ns is already very generous. I am not sure it would be measurable at all.

Paolo

Quoted 15 lines

To be clear, I would find it very reasonable to be requested to proof the benefit of the HLT optimization with benchmarks specifics for that single one liner, but until then, the idea that we can drop the retpoline optimization from the HLT vmexit by just thinking about it, still doesn't make sense to me, because by thinking about it I come to the opposite conclusion.

The lack of single_task_running() in the guest driver is also why the guest cpuidle haltpoll risks to waste some CPU with host overcommit or with the host loaded at full capacity and why we may not assume it to be universally enabled.

Thanks,
Andrea

#28 lore raw

PATCH 13/14 KVM: retpolines: x86: eliminate retpoline from svm.c exit handlers

In reply to Andrea Arcangeli

It's enough to check the exit value and issue a direct call to avoid the retpoline for all the common vmexit reasons.

After this commit is applied, here the most common retpolines executed under a high resolution timer workload in the guest on a SVM host:

[..]
@[

    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    ktime_get_update_offsets_now+70
    hrtimer_interrupt+131
    smp_apic_timer_interrupt+106
    apic_timer_interrupt+15
    start_sw_timer+359
    restart_apic_timer+85
    kvm_set_msr_common+1497
    msr_interception+142
    vcpu_enter_guest+684
    kvm_arch_vcpu_ioctl_run+261
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 1940
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_r12+33
    force_qs_rnp+217
    rcu_gp_kthread+1270
    kthread+268
    ret_from_fork+34
]: 4644
@[]: 25095
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    lapic_next_event+28
    clockevents_program_event+148
    hrtimer_start_range_ns+528
    start_sw_timer+356
    restart_apic_timer+85
    kvm_set_msr_common+1497
    msr_interception+142
    vcpu_enter_guest+684
    kvm_arch_vcpu_ioctl_run+261
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 41474
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    clockevents_program_event+148
    hrtimer_start_range_ns+528
    start_sw_timer+356
    restart_apic_timer+85
    kvm_set_msr_common+1497
    msr_interception+142
    vcpu_enter_guest+684
    kvm_arch_vcpu_ioctl_run+261
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 41474
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    ktime_get+58
    clockevents_program_event+84
    hrtimer_start_range_ns+528
    start_sw_timer+356
    restart_apic_timer+85
    kvm_set_msr_common+1497
    msr_interception+142
    vcpu_enter_guest+684
    kvm_arch_vcpu_ioctl_run+261
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 41887
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    lapic_next_event+28
    clockevents_program_event+148
    hrtimer_try_to_cancel+168
    hrtimer_cancel+21
    kvm_set_lapic_tscdeadline_msr+43
    kvm_set_msr_common+1497
    msr_interception+142
    vcpu_enter_guest+684
    kvm_arch_vcpu_ioctl_run+261
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 42723
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    clockevents_program_event+148
    hrtimer_try_to_cancel+168
    hrtimer_cancel+21
    kvm_set_lapic_tscdeadline_msr+43
    kvm_set_msr_common+1497
    msr_interception+142
    vcpu_enter_guest+684
    kvm_arch_vcpu_ioctl_run+261
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 42766
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    ktime_get+58
    clockevents_program_event+84
    hrtimer_try_to_cancel+168
    hrtimer_cancel+21
    kvm_set_lapic_tscdeadline_msr+43
    kvm_set_msr_common+1497
    msr_interception+142
    vcpu_enter_guest+684
    kvm_arch_vcpu_ioctl_run+261
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 42848
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    ktime_get+58
    start_sw_timer+279
    restart_apic_timer+85
    kvm_set_msr_common+1497
    msr_interception+142
    vcpu_enter_guest+684
    kvm_arch_vcpu_ioctl_run+261
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68

]: 499845

@total: 1780243

SVM has no TSC based programmable preemption timer so it is invoking ktime_get() frequently.

---
 arch/x86/kvm/svm.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
arch/x86/kvm/svm.c +14 -0
index 50c57112c0ce..4d8370fcd212 100644
@@ -4989,6 +4989,20 @@ int kvm_x86_handle_exit(struct kvm_vcpu *vcpu)
 		return 0;
 	}
 
+#ifdef CONFIG_RETPOLINE
+	if (exit_code == SVM_EXIT_MSR)
+		return msr_interception(svm);
+	else if (exit_code == SVM_EXIT_VINTR)
+		return interrupt_window_interception(svm);
+	else if (exit_code == SVM_EXIT_INTR)
+		return intr_interception(svm);
+	else if (exit_code == SVM_EXIT_HLT)
+		return halt_interception(svm);
+	else if (exit_code == SVM_EXIT_NPF)
+		return npf_interception(svm);
+	else if (exit_code == SVM_EXIT_CPUID)
+		return cpuid_interception(svm);
+#endif
 	return svm_exit_handlers[exit_code](svm);
 }
 
#29 lore raw

PATCH 14/14 x86: retpolines: eliminate retpoline from msr event handlers

In reply to Andrea Arcangeli

It's enough to check the value and issue the direct call.

After this commit is applied, here the most common retpolines executed under a high resolution timer workload in the guest on a VMX host:

[..]
@[

    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68
]: 267
@[]: 2256
@[
    trace_retpoline+1
    __trace_retpoline+30
    __x86_indirect_thunk_rax+33
    __kvm_wait_lapic_expire+284
    vmx_vcpu_run.part.97+1091
    vcpu_enter_guest+377
    kvm_arch_vcpu_ioctl_run+261
    kvm_vcpu_ioctl+559
    do_vfs_ioctl+164
    ksys_ioctl+96
    __x64_sys_ioctl+22
    do_syscall_64+89
    entry_SYSCALL_64_after_hwframe+68

]: 2390
@[]: 33410

@total: 315707

Note the highest hit above is __delay so probably not worth optimizing even if it would be more frequent than 2k hits per sec.

---
 arch/x86/events/intel/core.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
arch/x86/events/intel/core.c +11 -0
index 27ee47a7be66..65b383d5e062 100644
@@ -3323,8 +3323,19 @@ static int intel_pmu_hw_config(struct perf_event *event)
 	return 0;
 }
 
+#ifdef CONFIG_RETPOLINE
+static struct perf_guest_switch_msr *core_guest_get_msrs(int *nr);
+static struct perf_guest_switch_msr *intel_guest_get_msrs(int *nr);
+#endif
+
 struct perf_guest_switch_msr *perf_guest_get_msrs(int *nr)
 {
+#ifdef CONFIG_RETPOLINE
+	if (x86_pmu.guest_get_msrs == intel_guest_get_msrs)
+		return intel_guest_get_msrs(nr);
+	else if (x86_pmu.guest_get_msrs == core_guest_get_msrs)
+		return core_guest_get_msrs(nr);
+#endif
 	if (x86_pmu.guest_get_msrs)
 		return x86_pmu.guest_get_msrs(nr);
 	*nr = 0;