KVM: Register cpuhp/syscore callbacks when enabling virt
Subsystems (12)
Sean Christopherson
maintainerv4 cover KVM: Register cpuhp/syscore callbacks when enabling virt
Register KVM's cpuhp and syscore callbacks when enabling virtualization in hardware, as the sole purpose of said callbacks is to disable and re-enable virtualization as needed.
The primary motivation for this series is to simplify dealing with enabling virtualization for Intel's TDX, which needs to enable virtualization when kvm-intel.ko is loaded, i.e. long before the first VM is created.
That said, this is a nice cleanup on its own. By registering the callbacks on-demand, the callbacks themselves don't need to check kvm_usage_count, because their very existence implies a non-zero count.
Patch 1 (re)adds a dedicated lock for kvm_usage_count. Hopefully it's a temporary lock, as we can likely fix the cpus_read_lock() vs. kvm_lock mess for good by switching vm_list to an (S)RCU-protected list.
v4:
- Collect more reviews.
- Fix documentation for kvm_usage_lock. [Kai]
- Add a blurb in locking.rst to call out that cpus_read_lock() vs kvm_lock is a general problem.
- Add a paragraph in the locking patch to suggest switching vm_list to an (S)RCU-protected list, so that walkers don't need to take kvm_lock.
- Add an example call chain for the other cpus_read_lock() vs kvm_lock issues that are lurking.
- Enable the module param by default from the get-go. [Paolo]
- Rename even more APIs/symbols (all of 'em I could find). [Paolo]
- Clarify that it's the 0=>1 VM creation that's problematic, not simply the creation of the very first VM. [Paolo]
- Document enable_virt_at_load, and call out the (dis)advantages of enabling virtualization when KVM is loaded.
- Drop the WARN on kvm_usage_count being elevated at the end of kvm_uninit_virtualization(), as it's annoyingly difficult to keep the WARN and still allow arch code (i.e. TDX) to enable virtualization during setup.
v3:
- https://lore.kernel.org/all/20240608000639.3295768-1-seanjc at google.com
- Collect reviews/acks.
- Switch to kvm_usage_lock in a dedicated patch, Cc'd for stable at . [Chao]
- Enable virt at load by default. [Chao]
- Add comments to document how kvm_arch_{en,dis}able_virtualization() fit into the overall flow. [Kai]
v2:
- https://lore.kernel.org/all/20240522022827.1690416-1-seanjc at google.com
- Use a dedicated mutex to avoid lock inversion issues between kvm_lock and the cpuhp lock.
- Register emergency disable callbacks on-demand. [Kai]
- Drop an unintended s/junk/ign rename. [Kai]
- Decrement kvm_usage_count on failure. [Chao]
v1: https://lore.kernel.org/all/20240425233951.3344485-1-seanjc at google.com
Sean Christopherson (10):
KVM: Use dedicated mutex to protect kvm_usage_count to avoid deadlock
KVM: Register cpuhp and syscore callbacks when enabling hardware
KVM: Rename symbols related to enabling virtualization hardware
KVM: Rename arch hooks related to per-CPU virtualization enabling
KVM: MIPS: Rename virtualization {en,dis}abling APIs to match common
KVM
KVM: x86: Rename virtualization {en,dis}abling APIs to match common
KVM
KVM: Add a module param to allow enabling virtualization when KVM is
loaded
KVM: Add arch hooks for enabling/disabling virtualization x86/reboot: Unconditionally define cpu_emergency_virt_cb typedef
KVM: x86: Register "emergency disable" callbacks when virt is enabled
.../admin-guide/kernel-parameters.txt | 17 ++ Documentation/virt/kvm/locking.rst | 31 +- arch/arm64/kvm/arm.c | 6 +- arch/loongarch/kvm/main.c | 4 +- arch/mips/include/asm/kvm_host.h | 4 +- arch/mips/kvm/mips.c | 8 +- arch/mips/kvm/vz.c | 8 +- arch/riscv/kvm/main.c | 4 +- arch/x86/include/asm/kvm-x86-ops.h | 4 +- arch/x86/include/asm/kvm_host.h | 7 +- arch/x86/include/asm/reboot.h | 2 +- arch/x86/kvm/svm/svm.c | 19 +- arch/x86/kvm/vmx/main.c | 6 +- arch/x86/kvm/vmx/vmx.c | 10 +- arch/x86/kvm/vmx/x86_ops.h | 5 +- arch/x86/kvm/x86.c | 26 +- include/linux/kvm_host.h | 18 +- virt/kvm/kvm_main.c | 270 +++++++++--------- 18 files changed, 251 insertions(+), 198 deletions(-)
base-commit: 15e1c3d65975524c5c792fcd59f7d89f00402261
Signature · 1 line
2.46.0.469.g59c65b2a67-goog
Sean Christopherson
maintainerPATCH v4 1/10 KVM: Use dedicated mutex to protect kvm_usage_count to avoid deadlock
Use a dedicated mutex to guard kvm_usage_count to fix a potential deadlock on x86 due to a chain of locks and SRCU synchronizations. Translating the below lockdep splat, CPU1 #6 will wait on CPU0 #1, CPU0 #8 will wait on CPU2 #3, and CPU2 #7 will wait on CPU1 #4 (if there's a writer, due to the fairness of r/w semaphores).
CPU0 CPU1 CPU2
1 lock(&kvm->slots_lock);
2 lock(&vcpu->mutex);
3 lock(&kvm->srcu);
4 lock(cpu_hotplug_lock); 5 lock(kvm_lock); 6 lock(&kvm->slots_lock);
7 lock(cpu_hotplug_lock);
8 sync(&kvm->srcu);
Note, there are likely more potential deadlocks in KVM x86, e.g. the same pattern of taking cpu_hotplug_lock outside of kvm_lock likely exists with __kvmclock_cpufreq_notifier():
cpuhp_cpufreq_online()
|
-> cpufreq_online()
|
-> cpufreq_gov_performance_limits()
|
-> __cpufreq_driver_target()
|
-> __target_index()
|
-> cpufreq_freq_transition_begin()
|
-> cpufreq_notify_transition()
|
-> ... __kvmclock_cpufreq_notifier()But, actually triggering such deadlocks is beyond rare due to the combination of dependencies and timings involved. E.g. the cpufreq notifier is only used on older CPUs without a constant TSC, mucking with the NX hugepage mitigation while VMs are running is very uncommon, and doing so while also onlining/offlining a CPU (necessary to generate contention on cpu_hotplug_lock) would be even more unusual.
The most robust solution to the general cpu_hotplug_lock issue is likely to switch vm_list to be an RCU-protected list, e.g. so that x86's cpufreq notifier doesn't to take kvm_lock. For now, settle for fixing the most blatant deadlock, as switching to an RCU-protected list is a much more involved change, but add a comment in locking.rst to call out that care needs to be taken when walking holding kvm_lock and walking vm_list.
======================================================
WARNING: possible circular locking dependency detected
6.10.0-smp--c257535a0c9d-pip #330 Tainted: G S O
------------------------------------------------------
tee/35048 is trying to acquire lock:
ff6a80eced71e0a8 (&kvm->slots_lock){+.+.}-{3:3}, at: set_nx_huge_pages+0x179/0x1e0 [kvm]
but task is already holding lock:
ffffffffc07abb08 (kvm_lock){+.+.}-{3:3}, at: set_nx_huge_pages+0x14a/0x1e0 [kvm]
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #3 (kvm_lock){+.+.}-{3:3}:
__mutex_lock+0x6a/0xb40
mutex_lock_nested+0x1f/0x30
kvm_dev_ioctl+0x4fb/0xe50 [kvm]
__se_sys_ioctl+0x7b/0xd0
__x64_sys_ioctl+0x21/0x30
x64_sys_call+0x15d0/0x2e60
do_syscall_64+0x83/0x160
entry_SYSCALL_64_after_hwframe+0x76/0x7e-> #2 (cpu_hotplug_lock){++++}-{0:0}:
cpus_read_lock+0x2e/0xb0
static_key_slow_inc+0x16/0x30
kvm_lapic_set_base+0x6a/0x1c0 [kvm]
kvm_set_apic_base+0x8f/0xe0 [kvm]
kvm_set_msr_common+0x9ae/0xf80 [kvm]
vmx_set_msr+0xa54/0xbe0 [kvm_intel]
__kvm_set_msr+0xb6/0x1a0 [kvm]
kvm_arch_vcpu_ioctl+0xeca/0x10c0 [kvm]
kvm_vcpu_ioctl+0x485/0x5b0 [kvm]
__se_sys_ioctl+0x7b/0xd0
__x64_sys_ioctl+0x21/0x30
x64_sys_call+0x15d0/0x2e60
do_syscall_64+0x83/0x160
entry_SYSCALL_64_after_hwframe+0x76/0x7e-> #1 (&kvm->srcu){.+.+}-{0:0}:
__synchronize_srcu+0x44/0x1a0
synchronize_srcu_expedited+0x21/0x30
kvm_swap_active_memslots+0x110/0x1c0 [kvm]
kvm_set_memslot+0x360/0x620 [kvm]
__kvm_set_memory_region+0x27b/0x300 [kvm]
kvm_vm_ioctl_set_memory_region+0x43/0x60 [kvm]
kvm_vm_ioctl+0x295/0x650 [kvm]
__se_sys_ioctl+0x7b/0xd0
__x64_sys_ioctl+0x21/0x30
x64_sys_call+0x15d0/0x2e60
do_syscall_64+0x83/0x160
entry_SYSCALL_64_after_hwframe+0x76/0x7e-> #0 (&kvm->slots_lock){+.+.}-{3:3}:
__lock_acquire+0x15ef/0x2e30
lock_acquire+0xe0/0x260
__mutex_lock+0x6a/0xb40
mutex_lock_nested+0x1f/0x30
set_nx_huge_pages+0x179/0x1e0 [kvm]
param_attr_store+0x93/0x100
module_attr_store+0x22/0x40
sysfs_kf_write+0x81/0xb0
kernfs_fop_write_iter+0x133/0x1d0
vfs_write+0x28d/0x380
ksys_write+0x70/0xe0
__x64_sys_write+0x1f/0x30
x64_sys_call+0x281b/0x2e60
do_syscall_64+0x83/0x160
entry_SYSCALL_64_after_hwframe+0x76/0x7e--- Documentation/virt/kvm/locking.rst | 32 +++++++++++++++++++++--------- virt/kvm/kvm_main.c | 31 +++++++++++++++-------------- 2 files changed, 39 insertions(+), 24 deletions(-)
Documentation/virt/kvm/locking.rst +23 -9
@@ -9,7 +9,7 @@ KVM Lock Overview The acquisition orders for mutexes are as follows: -- cpus_read_lock() is taken outside kvm_lock +- cpus_read_lock() is taken outside kvm_lock and kvm_usage_lock - kvm->lock is taken outside vcpu->mutex
@@ -24,6 +24,12 @@ The acquisition orders for mutexes are as follows: are taken on the waiting side when modifying memslots, so MMU notifiers must not take either kvm->slots_lock or kvm->slots_arch_lock. +cpus_read_lock() vs kvm_lock: +- Taking cpus_read_lock() outside of kvm_lock is problematic, despite that + being the official ordering, as it is quite easy to unknowingly trigger + cpus_read_lock() while holding kvm_lock. Use caution when walking vm_list, + e.g. avoid complex operations when possible. + For SRCU: - ``synchronize_srcu(&kvm->srcu)`` is called inside critical sections
@@ -227,10 +233,17 @@ time it will be set using the Dirty tracking mechanism described above. :Type: mutex :Arch: any :Protects: - vm_list - - kvm_usage_count + +``kvm_usage_lock`` +^^^^^^^^^^^^^^^^^^ + +:Type: mutex +:Arch: any +:Protects: - kvm_usage_count - hardware virtualization enable/disable -:Comment: KVM also disables CPU hotplug via cpus_read_lock() during - enable/disable. +:Comment: Exists because using kvm_lock leads to deadlock (see earlier comment + on cpus_read_lock() vs kvm_lock). Note, KVM also disables CPU hotplug via + cpus_read_lock() when enabling/disabling virtualization. ``kvm->mn_invalidate_lock`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -290,11 +303,12 @@ time it will be set using the Dirty tracking mechanism described above. wakeup. ``vendor_module_lock`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +^^^^^^^^^^^^^^^^^^^^^^ :Type: mutex :Arch: x86 :Protects: loading a vendor module (kvm_amd or kvm_intel) -:Comment: Exists because using kvm_lock leads to deadlock. cpu_hotplug_lock is - taken outside of kvm_lock, e.g. in KVM's CPU online/offline callbacks, and - many operations need to take cpu_hotplug_lock when loading a vendor module, - e.g. updating static calls. +:Comment: Exists because using kvm_lock leads to deadlock. kvm_lock is taken + in notifiers, e.g. __kvmclock_cpufreq_notifier(), that may be invoked while + cpu_hotplug_lock is held, e.g. from cpufreq_boost_trigger_state(), and many + operations need to take cpu_hotplug_lock when loading a vendor module, e.g. + updating static calls.
virt/kvm/kvm_main.c +16 -15
@@ -5576,6 +5576,7 @@ __visible bool kvm_rebooting;EXPORT_SYMBOL_GPL (kvm_rebooting); static DEFINE_PER_CPU(bool, hardware_enabled); +static DEFINE_MUTEX (kvm_usage_lock);static int kvm_usage_count;static int __hardware_enable_nolock (void )
@@ -5608,10 +5609,10 @@ static int kvm_online_cpu(unsigned int cpu)* be enabled. Otherwise running VMs would encounter unrecoverable * errors when scheduled to this CPU. */ -mutex_lock (&kvm_lock); +mutex_lock (&kvm_usage_lock);if (kvm_usage_count) ret =__hardware_enable_nolock (); -mutex_unlock (&kvm_lock); +mutex_unlock (&kvm_usage_lock);return ret; }
@@ -5631,10 +5632,10 @@ static void hardware_disable_nolock(void *junk)static int kvm_offline_cpu (unsigned int cpu) { -mutex_lock (&kvm_lock); +mutex_lock (&kvm_usage_lock);if (kvm_usage_count)hardware_disable_nolock (NULL ); -mutex_unlock (&kvm_lock); +mutex_unlock (&kvm_usage_lock);return 0 ; }
@@ -5650,9 +5651,9 @@ static void hardware_disable_all_nolock(void)static void hardware_disable_all (void ) {cpus_read_lock (); -mutex_lock (&kvm_lock); +mutex_lock (&kvm_usage_lock);hardware_disable_all_nolock (); -mutex_unlock (&kvm_lock); +mutex_unlock (&kvm_usage_lock);cpus_read_unlock (); }
@@ -5683,7 +5684,7 @@ static int hardware_enable_all(void)* enable hardware multiple times. */ cpus_read_lock (); -mutex_lock (&kvm_lock); +mutex_lock (&kvm_usage_lock); r =0 ;
@@ -5697,7 +5698,7 @@ static int hardware_enable_all(void) } } -mutex_unlock (&kvm_lock); +mutex_unlock (&kvm_usage_lock);cpus_read_unlock ();return r;
@@ -5725,13 +5726,13 @@ static int kvm_suspend(void) {/* * Secondary CPUs and CPU hotplug are disabled across the suspend/resume -* callbacks, i.e. no need to acquire kvm_lock to ensure the usage count -* is stable. Assert that kvm_lock is not held to ensure the system -* isn't suspended while KVM is enabling hardware. Hardware enabling -* can be preempted, but the task cannot be frozen until it has dropped -* all locks (userspace tasks are frozen via a fake signal). +* callbacks, i.e. no need to acquire kvm_usage_lock to ensure the usage +* count is stable. Assert that kvm_usage_lock is not held to ensure +* the system isn't suspended while KVM is enabling hardware. Hardware +* enabling can be preempted, but the task cannot be frozen until it has +* dropped all locks (userspace tasks are frozen via a fake signal). */ -lockdep_assert_not_held (&kvm_lock); +lockdep_assert_not_held (&kvm_usage_lock);lockdep_assert_irqs_disabled ();if (kvm_usage_count)
@@ -5741,7 +5742,7 @@ static int kvm_suspend(void)static void kvm_resume (void ) { -lockdep_assert_not_held (&kvm_lock); +lockdep_assert_not_held (&kvm_usage_lock);lockdep_assert_irqs_disabled ();if (kvm_usage_count)
Signature · 1 line
2.46.0.469.g59c65b2a67-goog
Re: [PATCH v4 01/10] KVM: Use dedicated mutex to protect kvm_usage_count to avoid deadlock
Quoted 4 lines
Reviewed-by Kai Huang <[email protected]>Acked-by Kai Huang <[email protected]>
Hmm I must have done a lot for me to receive two credits (and for most patches in this series) :-)
I think one Reviewed-by tag is good enough :-)
Sean Christopherson
maintainerRe: [PATCH v4 01/10] KVM: Use dedicated mutex to protect kvm_usage_count to avoid deadlock
Quoted 9 lines · Kai Huang
Quoted 4 lines
Reviewed-by Kai Huang <[email protected]>Acked-by Kai Huang <[email protected]>Hmm I must have done a lot for me to receive two credits (and for most patches in this series) :-)
I think one Reviewed-by tag is good enough :-)
Heh, indeed. b4 has made me very lazy; I just b4 am the patches and let b4 grab all the trailers. I'm guessing something went awry in that flow (or maybe you acked a previous version or something?)
Anyways, one of Paolo or I can clean this up when applying, assuming we remember to do so...
Re: [PATCH v4 01/10] KVM: Use dedicated mutex to protect kvm_usage_count to avoid deadlock
Quoted 6 lines · Sean Christopherson
Quoted 5 lines · Kai Huang
Quoted 4 lines
Reviewed-by Kai Huang <[email protected]>Acked-by Kai Huang <[email protected]>
Quoted 7 lines
Quoted 4 lines
Hmm I must have done a lot for me to receive two credits (and for most patches in this series) :-)
I think one Reviewed-by tag is good enough :-)
Heh, indeed. b4 has made me very lazy; I just
b4 amthe patches and let b4 grab all the trailers.
That's what I was thinking about too.
Quoted 2 lines
I'm guessing something went awry in that flow (or maybe you acked a previous version or something?)
Anyway will pay more attention to this. Thanks!
Sean Christopherson
maintainerPATCH v4 2/10 KVM: Register cpuhp and syscore callbacks when enabling hardware
In reply to Sean ChristophersonRegister KVM's cpuhp and syscore callback when enabling virtualization in hardware instead of registering the callbacks during initialization, and let the CPU up/down framework invoke the inner enable/disable functions. Registering the callbacks during initialization makes things more complex than they need to be, as KVM needs to be very careful about handling races between enabling CPUs being onlined/offlined and hardware being enabled/disabled.
Intel TDX support will require KVM to enable virtualization during KVM initialization, i.e. will add another wrinkle to things, at which point sorting out the potential races with kvm_usage_count would become even more complex.
Note, using the cpuhp framework has a subtle behavioral change: enabling will be done serially across all CPUs, whereas KVM currently sends an IPI to all CPUs in parallel. While serializing virtualization enabling could create undesirable latency, the issue is limited to the 0=>1 transition of VM creation. And even that can be mitigated, e.g. by letting userspace force virtualization to be enabled when KVM is initialized.
--- Documentation/virt/kvm/locking.rst | 9 +- virt/kvm/kvm_main.c | 174 ++++++++++------------------- 2 files changed, 66 insertions(+), 117 deletions(-)
Documentation/virt/kvm/locking.rst +5 -4
@@ -9,7 +9,9 @@ KVM Lock Overview The acquisition orders for mutexes are as follows: -- cpus_read_lock() is taken outside kvm_lock and kvm_usage_lock +- cpus_read_lock() is taken outside kvm_lock + +- kvm_usage_lock is taken outside cpus_read_lock() - kvm->lock is taken outside vcpu->mutex
@@ -241,9 +243,8 @@ time it will be set using the Dirty tracking mechanism described above. :Arch: any :Protects: - kvm_usage_count - hardware virtualization enable/disable -:Comment: Exists because using kvm_lock leads to deadlock (see earlier comment - on cpus_read_lock() vs kvm_lock). Note, KVM also disables CPU hotplug via - cpus_read_lock() when enabling/disabling virtualization. +:Comment: Exists to allow taking cpus_read_lock() while kvm_usage_count is + protected, which simplifies the virtualization enabling logic. ``kvm->mn_invalidate_lock`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^
virt/kvm/kvm_main.c +61 -113
248 lines not on this page · raw message
Signature · 1 line
2.46.0.469.g59c65b2a67-goog
Sean Christopherson
maintainerPATCH v4 3/10 KVM: Rename symbols related to enabling virtualization hardware
In reply to Sean ChristophersonRename the various functions (and a variable) that enable virtualization to prepare for upcoming changes, and to clean up artifacts of KVM's previous behavior, which required manually juggling locks around kvm_usage_count.
Drop the "nolock" qualifier from per-CPU functions now that there are no "nolock" implementations of the "all" variants, i.e. now that calling a non-nolock function from a nolock function isn't confusing (unlike this sentence).
Drop "all" from the outer helpers as they no longer manually iterate over all CPUs, and because it might not be obvious what "all" refers to.
In lieu of the above qualifiers, append "_cpu" to the end of the functions that are per-CPU helpers for the outer APIs.
Opportunistically prepend "kvm" to all functions to help make it clear that they are KVM helpers, but mostly because there's no reason not to.
Lastly, use "virtualization" instead of "hardware", because while the functions do enable virtualization in hardware, there are a _lot_ of things that KVM enables in hardware.
Defer renaming the arch hooks to future patches, purely to reduce the amount of churn in a single commit.
--- virt/kvm/kvm_main.c | 42 +++++++++++++++++++++--------------------- 1 file changed, 21 insertions(+), 21 deletions(-)
virt/kvm/kvm_main.c +21 -21
@@ -136,8 +136,8 @@ static int kvm_no_compat_open(struct inode *inode, struct file *file)#define KVM_COMPAT (c) .compat_ioctl = kvm_no_compat_ioctl, \ .open = kvm_no_compat_open#endif -static int hardware_enable_all (void ); -static void hardware_disable_all (void ); +static int kvm_enable_virtualization (void ); +static void kvm_disable_virtualization (void );static void kvm_io_bus_destroy (struct kvm_io_bus *bus);
@@ -1220,7 +1220,7 @@ static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)if (r) goto out_err_no_arch_destroy_vm; - r =hardware_enable_all (); + r =kvm_enable_virtualization ();if (r) goto out_err_no_disable;
@@ -1263,7 +1263,7 @@ static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)mmu_notifier_unregister (&kvm->mmu_notifier, current->mm);#endif out_err_no_mmu_notifier: -hardware_disable_all (); +kvm_disable_virtualization (); out_err_no_disable:kvm_arch_destroy_vm (kvm); out_err_no_arch_destroy_vm:
@@ -1360,7 +1360,7 @@ static void kvm_destroy_vm(struct kvm *kvm)#endif kvm_arch_free_vm (kvm);preempt_notifier_dec (); -hardware_disable_all (); +kvm_disable_virtualization ();mmdrop (mm); }
@@ -5575,13 +5575,13 @@ static struct miscdevice kvm_dev = { __visible bool kvm_rebooting;EXPORT_SYMBOL_GPL (kvm_rebooting); -static DEFINE_PER_CPU(bool, hardware_enabled); +static DEFINE_PER_CPU(bool, virtualization_enabled);static DEFINE_MUTEX (kvm_usage_lock);static int kvm_usage_count; -static int hardware_enable_nolock (void ) +static int kvm_enable_virtualization_cpu (void ) { -if (__this_cpu_read (hardware_enabled)) +if (__this_cpu_read (virtualization_enabled))return 0 ;if (kvm_arch_hardware_enable ()) {
@@ -5590,7 +5590,7 @@ static int hardware_enable_nolock(void)return -EIO ; } -__this_cpu_write (hardware_enabled, true); +__this_cpu_write (virtualization_enabled, true);return 0 ; }
@@ -5601,22 +5601,22 @@ static int kvm_online_cpu(unsigned int cpu)* be enabled. Otherwise running VMs would encounter unrecoverable * errors when scheduled to this CPU. */ -return hardware_enable_nolock (); +return kvm_enable_virtualization_cpu (); } -static void hardware_disable_nolock (void *junk) +static void kvm_disable_virtualization_cpu (void *ign) { -if (!__this_cpu_read (hardware_enabled)) +if (!__this_cpu_read (virtualization_enabled))return ;kvm_arch_hardware_disable (); -__this_cpu_write (hardware_enabled, false); +__this_cpu_write (virtualization_enabled, false); }static int kvm_offline_cpu (unsigned int cpu) { -hardware_disable_nolock (NULL ); +kvm_disable_virtualization_cpu (NULL );return 0 ; }
@@ -5635,7 +5635,7 @@ static void kvm_shutdown(void)*/ pr_info ("kvm: exiting hardware virtualization\n" ); kvm_rebooting = true; -on_each_cpu (hardware_disable_nolock,NULL ,1 ); +on_each_cpu (kvm_disable_virtualization_cpu,NULL ,1 ); }static int kvm_suspend (void )
@@ -5651,7 +5651,7 @@ static int kvm_suspend(void)lockdep_assert_not_held (&kvm_usage_lock);lockdep_assert_irqs_disabled (); -hardware_disable_nolock (NULL ); +kvm_disable_virtualization_cpu (NULL );return 0 ; }
@@ -5660,7 +5660,7 @@ static void kvm_resume(void)lockdep_assert_not_held (&kvm_usage_lock);lockdep_assert_irqs_disabled (); -WARN_ON_ONCE (hardware_enable_nolock ()); +WARN_ON_ONCE (kvm_enable_virtualization_cpu ()); }static struct syscore_ops kvm_syscore_ops = {
@@ -5669,7 +5669,7 @@ static struct syscore_ops kvm_syscore_ops = { .shutdown = kvm_shutdown, }; -static int hardware_enable_all (void ) +static int kvm_enable_virtualization (void ) {int r;
@@ -5712,7 +5712,7 @@ static int hardware_enable_all(void)return r; } -static void hardware_disable_all (void ) +static void kvm_disable_virtualization (void ) {guard (mutex)(&kvm_usage_lock);
@@ -5723,12 +5723,12 @@ static void hardware_disable_all(void) cpuhp_remove_state(CPUHP_AP_KVM_ONLINE); }#else /* CONFIG_KVM_GENERIC_HARDWARE_ENABLING */ -static int hardware_enable_all (void ) +static int kvm_enable_virtualization (void ) {return 0 ; } -static void hardware_disable_all (void ) +static void kvm_disable_virtualization (void ) { }
Signature · 1 line
2.46.0.469.g59c65b2a67-goog
Sean Christopherson
maintainerPATCH v4 4/10 KVM: Rename arch hooks related to per-CPU virtualization enabling
In reply to Sean ChristophersonRename the per-CPU hooks used to enable virtualization in hardware to align with the KVM-wide helpers in kvm_main.c, and to better capture that the callbacks are invoked on every online CPU.
No functional change intended.
--- arch/arm64/kvm/arm.c | 6 +++--- arch/loongarch/kvm/main.c | 4 ++-- arch/mips/kvm/mips.c | 4 ++-- arch/riscv/kvm/main.c | 4 ++-- arch/x86/kvm/x86.c | 6 +++--- include/linux/kvm_host.h | 4 ++-- virt/kvm/kvm_main.c | 4 ++-- 7 files changed, 16 insertions(+), 16 deletions(-)
arch/arm64/kvm/arm.c +3 -3
@@ -2163,7 +2163,7 @@ static void cpu_hyp_uninit(void *discard) } } -int kvm_arch_hardware_enable (void ) +int kvm_arch_enable_virtualization_cpu (void ) {/* * Most calls to this function are made with migration
@@ -2183,7 +2183,7 @@ int kvm_arch_hardware_enable(void)return 0 ; } -void kvm_arch_hardware_disable (void ) +void kvm_arch_disable_virtualization_cpu (void ) {kvm_timer_cpu_down ();kvm_vgic_cpu_down ();
@@ -2379,7 +2379,7 @@ static int __init do_pkvm_init(u32 hyp_va_bits)/* * The stub hypercalls are now disabled, so set our local flag to -* prevent a later re-init attempt in kvm_arch_hardware_enable(). +* prevent a later re-init attempt in kvm_arch_enable_virtualization_cpu(). */ __this_cpu_write (kvm_hyp_initialized,1 );preempt_enable ();
arch/loongarch/kvm/main.c +2 -2
@@ -261,7 +261,7 @@ long kvm_arch_dev_ioctl(struct file *filp,return -ENOIOCTLCMD ; } -int kvm_arch_hardware_enable (void ) +int kvm_arch_enable_virtualization_cpu (void ) {unsigned long env, gcfg =0 ;
@@ -300,7 +300,7 @@ int kvm_arch_hardware_enable(void)return 0 ; } -void kvm_arch_hardware_disable (void ) +void kvm_arch_disable_virtualization_cpu (void ) {write_csr_gcfg (0 );write_csr_gstat (0 );
arch/mips/kvm/mips.c +2 -2
@@ -125,12 +125,12 @@ int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu)return 1 ; } -int kvm_arch_hardware_enable (void ) +int kvm_arch_enable_virtualization_cpu (void ) {return kvm_mips_callbacks->hardware_enable (); } -void kvm_arch_hardware_disable (void ) +void kvm_arch_disable_virtualization_cpu (void ) { kvm_mips_callbacks->hardware_disable (); }
arch/riscv/kvm/main.c +2 -2
@@ -20,7 +20,7 @@ long kvm_arch_dev_ioctl(struct file *filp,return -EINVAL ; } -int kvm_arch_hardware_enable (void ) +int kvm_arch_enable_virtualization_cpu (void ) {csr_write (CSR_HEDELEG ,KVM_HEDELEG_DEFAULT );csr_write (CSR_HIDELEG ,KVM_HIDELEG_DEFAULT );
@@ -35,7 +35,7 @@ int kvm_arch_hardware_enable(void)return 0 ; } -void kvm_arch_hardware_disable (void ) +void kvm_arch_disable_virtualization_cpu (void ) {kvm_riscv_aia_disable ();
arch/x86/kvm/x86.c +3 -3
@@ -355,7 +355,7 @@ static void kvm_on_user_return(struct user_return_notifier *urn)/* * Disabling irqs at this point since the following code could be -* interrupted and executed through kvm_arch_hardware_disable() +* interrupted and executed through kvm_arch_disable_virtualization_cpu() */ local_irq_save (flags);if (msrs->registered) {
@@ -12512,7 +12512,7 @@ void kvm_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector) }EXPORT_SYMBOL_GPL (kvm_vcpu_deliver_sipi_vector); -int kvm_arch_hardware_enable (void ) +int kvm_arch_enable_virtualization_cpu (void ) {struct kvm *kvm;struct kvm_vcpu *vcpu;
@@ -12608,7 +12608,7 @@ int kvm_arch_hardware_enable(void)return 0 ; } -void kvm_arch_hardware_disable (void ) +void kvm_arch_disable_virtualization_cpu (void ) {kvm_x86_call (hardware_disable)();drop_user_return_notifiers ();
include/linux/kvm_host.h +2 -2
@@ -1514,8 +1514,8 @@ static inline void kvm_create_vcpu_debugfs(struct kvm_vcpu *vcpu) {}#endif #ifdef CONFIG_KVM_GENERIC_HARDWARE_ENABLING -int kvm_arch_hardware_enable (void ); -void kvm_arch_hardware_disable (void ); +int kvm_arch_enable_virtualization_cpu (void ); +void kvm_arch_disable_virtualization_cpu (void );#endif int kvm_arch_vcpu_runnable (struct kvm_vcpu *vcpu);bool kvm_arch_vcpu_in_kernel (struct kvm_vcpu *vcpu);
virt/kvm/kvm_main.c +2 -2
@@ -5584,7 +5584,7 @@ static int kvm_enable_virtualization_cpu(void)if (__this_cpu_read (virtualization_enabled))return 0 ; -if (kvm_arch_hardware_enable ()) { +if (kvm_arch_enable_virtualization_cpu ()) {pr_info ("kvm: enabling virtualization on CPU%d failed\n" ,raw_smp_processor_id ());return -EIO ;
@@ -5609,7 +5609,7 @@ static void kvm_disable_virtualization_cpu(void *ign)if (!__this_cpu_read (virtualization_enabled))return ; -kvm_arch_hardware_disable (); +kvm_arch_disable_virtualization_cpu ();__this_cpu_write (virtualization_enabled, false); }
Signature · 1 line
2.46.0.469.g59c65b2a67-goog
Re: [PATCH v4 04/10] KVM: Rename arch hooks related to per-CPU virtualization enabling
Quoted 9 lines · Sean Christopherson
Rename the per-CPU hooks used to enable virtualization in hardware to align with the KVM-wide helpers in
kvm_main.c, and to better capture that the callbacks are invoked on every online CPU.No functional change intended.
Suggested-by Paolo Bonzini <[email protected]>Signed-off-by Sean Christopherson <[email protected]>
Sean Christopherson
maintainerPATCH v4 5/10 KVM: MIPS: Rename virtualization {en,dis}abling APIs to match common KVM
In reply to Sean ChristophersonRename MIPS's trampoline hooks for virtualization enabling to match the recently renamed arch hooks.
No functional change intended.
--- arch/mips/include/asm/kvm_host.h | 4 ++-- arch/mips/kvm/mips.c | 4 ++-- arch/mips/kvm/vz.c | 8 ++++---- 3 files changed, 8 insertions(+), 8 deletions(-)
arch/mips/include/asm/kvm_host.h +2 -2
@@ -728,8 +728,8 @@ struct kvm_mips_callbacks {int (*handle_fpe)(struct kvm_vcpu *vcpu);int (*handle_msa_disabled)(struct kvm_vcpu *vcpu);int (*handle_guest_exit)(struct kvm_vcpu *vcpu); -int (*hardware_enable)(void ); -void (*hardware_disable)(void ); +int (*enable_virtualization_cpu)(void ); +void (*disable_virtualization_cpu)(void );int (*check_extension)(struct kvm *kvm,long ext);int (*vcpu_init)(struct kvm_vcpu *vcpu);void (*vcpu_uninit)(struct kvm_vcpu *vcpu);
arch/mips/kvm/mips.c +2 -2
@@ -127,12 +127,12 @@ int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu)int kvm_arch_enable_virtualization_cpu (void ) { -return kvm_mips_callbacks->hardware_enable (); +return kvm_mips_callbacks->enable_virtualization_cpu (); }void kvm_arch_disable_virtualization_cpu (void ) { - kvm_mips_callbacks->hardware_disable (); + kvm_mips_callbacks->disable_virtualization_cpu (); }int kvm_arch_init_vm (struct kvm *kvm,unsigned long type)
arch/mips/kvm/vz.c +4 -4
@@ -2869,7 +2869,7 @@ static unsigned int kvm_vz_resize_guest_vtlb(unsigned int size)return ret +1 ; } -static int kvm_vz_hardware_enable (void ) +static int kvm_vz_enable_virtualization_cpu (void ) {unsigned int mmu_size, guest_mmu_size, ftlb_size;u64 guest_cvmctl, cvmvmconfig;
@@ -2983,7 +2983,7 @@ static int kvm_vz_hardware_enable(void)return 0 ; } -static void kvm_vz_hardware_disable (void ) +static void kvm_vz_disable_virtualization_cpu (void ) {u64 cvmvmconfig;unsigned int mmu_size;
@@ -3280,8 +3280,8 @@ static struct kvm_mips_callbacks kvm_vz_callbacks = { .handle_msa_disabled = kvm_trap_vz_handle_msa_disabled, .handle_guest_exit = kvm_trap_vz_handle_guest_exit, - .hardware_enable = kvm_vz_hardware_enable, - .hardware_disable = kvm_vz_hardware_disable, + .enable_virtualization_cpu = kvm_vz_enable_virtualization_cpu, + .disable_virtualization_cpu = kvm_vz_disable_virtualization_cpu, .check_extension = kvm_vz_check_extension, .vcpu_init = kvm_vz_vcpu_init, .vcpu_uninit = kvm_vz_vcpu_uninit,
Signature · 1 line
2.46.0.469.g59c65b2a67-goog
Sean Christopherson
maintainerPATCH v4 6/10 KVM: x86: Rename virtualization {en,dis}abling APIs to match common KVM
In reply to Sean ChristophersonRename x86's the per-CPU vendor hooks used to enable virtualization in hardware to align with the recently renamed arch hooks.
No functional change intended.
--- arch/x86/include/asm/kvm-x86-ops.h | 4 ++-- arch/x86/include/asm/kvm_host.h | 4 ++-- arch/x86/kvm/svm/svm.c | 18 +++++++++--------- arch/x86/kvm/vmx/main.c | 4 ++-- arch/x86/kvm/vmx/vmx.c | 10 +++++----- arch/x86/kvm/vmx/x86_ops.h | 4 ++-- arch/x86/kvm/x86.c | 10 +++++----- 7 files changed, 27 insertions(+), 27 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +2 -2
@@ -14,8 +14,8 @@ BUILD_BUG_ON(1)* be __static_call_return0. */ KVM_X86_OP(check_processor_compatibility) -KVM_X86_OP(hardware_enable) -KVM_X86_OP(hardware_disable) +KVM_X86_OP(enable_virtualization_cpu) +KVM_X86_OP(disable_virtualization_cpu) KVM_X86_OP(hardware_unsetup) KVM_X86_OP(has_emulated_msr) KVM_X86_OP(vcpu_after_set_cpuid)
arch/x86/include/asm/kvm_host.h +2 -2
@@ -1629,8 +1629,8 @@ struct kvm_x86_ops {int (*check_processor_compatibility)(void ); -int (*hardware_enable)(void ); -void (*hardware_disable)(void ); +int (*enable_virtualization_cpu)(void ); +void (*disable_virtualization_cpu)(void );void (*hardware_unsetup)(void );bool (*has_emulated_msr)(struct kvm *kvm,u32 index);void (*vcpu_after_set_cpuid)(struct kvm_vcpu *vcpu);
arch/x86/kvm/svm/svm.c +9 -9
@@ -592,14 +592,14 @@ static inline void kvm_cpu_svm_disable(void) } } -static void svm_emergency_disable (void ) +static void svm_emergency_disable_virtualization_cpu (void ) { kvm_rebooting = true;kvm_cpu_svm_disable (); } -static void svm_hardware_disable (void ) +static void svm_disable_virtualization_cpu (void ) {/* Make sure we clean up behind us */ if (tsc_scaling)
@@ -610,7 +610,7 @@ static void svm_hardware_disable(void)amd_pmu_disable_virt (); } -static int svm_hardware_enable (void ) +static int svm_enable_virtualization_cpu (void ) {struct svm_cpu_data *sd;
@@ -1533,7 +1533,7 @@ static void svm_prepare_switch_to_guest(struct kvm_vcpu *vcpu)* TSC_AUX is always virtualized for SEV-ES guests when the feature is * available. The user return MSR support is not required in this case * because TSC_AUX is restored on #VMEXIT from the host save area -* (which has been initialized in svm_hardware_enable()). +* (which has been initialized in svm_enable_virtualization_cpu()). */ if (likely (tsc_aux_uret_slot >=0 ) && (!boot_cpu_has (X86_FEATURE_V_TSC_AUX ) || !sev_es_guest (vcpu->kvm)))
@@ -3132,7 +3132,7 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)* feature is available. The user return MSR support is not * required in this case because TSC_AUX is restored on #VMEXIT * from the host save area (which has been initialized in -* svm_hardware_enable()). +* svm_enable_virtualization_cpu()). */ if (boot_cpu_has (X86_FEATURE_V_TSC_AUX ) &&sev_es_guest (vcpu->kvm))break ;
@@ -4980,8 +4980,8 @@ static struct kvm_x86_ops svm_x86_ops __initdata = { .check_processor_compatibility = svm_check_processor_compat, .hardware_unsetup = svm_hardware_unsetup, - .hardware_enable = svm_hardware_enable, - .hardware_disable = svm_hardware_disable, + .enable_virtualization_cpu = svm_enable_virtualization_cpu, + .disable_virtualization_cpu = svm_disable_virtualization_cpu, .has_emulated_msr = svm_has_emulated_msr, .vcpu_create = svm_vcpu_create,
@@ -5411,7 +5411,7 @@ static void __svm_exit(void) {kvm_x86_vendor_exit (); -cpu_emergency_unregister_virt_callback (svm_emergency_disable); +cpu_emergency_unregister_virt_callback (svm_emergency_disable_virtualization_cpu); }static int __init svm_init (void )
@@ -5427,7 +5427,7 @@ static int __init svm_init(void)if (r)return r; -cpu_emergency_register_virt_callback (svm_emergency_disable); +cpu_emergency_register_virt_callback (svm_emergency_disable_virtualization_cpu);/* * Common KVM initialization _must_ come last, after this, /dev/kvm is
arch/x86/kvm/vmx/main.c +2 -2
@@ -23,8 +23,8 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .hardware_unsetup = vmx_hardware_unsetup, - .hardware_enable = vmx_hardware_enable, - .hardware_disable = vmx_hardware_disable, + .enable_virtualization_cpu = vmx_enable_virtualization_cpu, + .disable_virtualization_cpu = vmx_disable_virtualization_cpu, .has_emulated_msr = vmx_has_emulated_msr, .vm_size = sizeof(struct kvm_vmx),
arch/x86/kvm/vmx/vmx.c +5 -5
@@ -755,7 +755,7 @@ static int kvm_cpu_vmxoff(void)return -EIO ; } -static void vmx_emergency_disable (void ) +static void vmx_emergency_disable_virtualization_cpu (void ) {int cpu =raw_smp_processor_id ();struct loaded_vmcs *v;
@@ -2844,7 +2844,7 @@ static int kvm_cpu_vmxon(u64 vmxon_pointer)return -EFAULT ; } -int vmx_hardware_enable (void ) +int vmx_enable_virtualization_cpu (void ) {int cpu =raw_smp_processor_id ();u64 phys_addr =__pa (per_cpu (vmxarea, cpu));
@@ -2881,7 +2881,7 @@ static void vmclear_local_loaded_vmcss(void)__loaded_vmcs_clear (v); } -void vmx_hardware_disable (void ) +void vmx_disable_virtualization_cpu (void ) {vmclear_local_loaded_vmcss ();
@@ -8584,7 +8584,7 @@ static void __vmx_exit(void) { allow_smaller_maxphyaddr = false; -cpu_emergency_unregister_virt_callback (vmx_emergency_disable); +cpu_emergency_unregister_virt_callback (vmx_emergency_disable_virtualization_cpu);vmx_cleanup_l1d_flush (); }
@@ -8632,7 +8632,7 @@ static int __init vmx_init(void)pi_init_cpu (cpu); } -cpu_emergency_register_virt_callback (vmx_emergency_disable); +cpu_emergency_register_virt_callback (vmx_emergency_disable_virtualization_cpu);vmx_check_vmcs12_offsets ();
arch/x86/kvm/vmx/x86_ops.h +2 -2
@@ -13,8 +13,8 @@ extern struct kvm_x86_init_ops vt_init_ops __initdata;void vmx_hardware_unsetup (void );int vmx_check_processor_compat (void ); -int vmx_hardware_enable (void ); -void vmx_hardware_disable (void ); +int vmx_enable_virtualization_cpu (void ); +void vmx_disable_virtualization_cpu (void );int vmx_vm_init (struct kvm *kvm);void vmx_vm_destroy (struct kvm *kvm);int vmx_vcpu_precreate (struct kvm *kvm);
arch/x86/kvm/x86.c +5 -5
@@ -9749,7 +9749,7 @@ int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops)guard (mutex)(&vendor_module_lock); -if (kvm_x86_ops.hardware_enable) { +if (kvm_x86_ops.enable_virtualization_cpu) {pr_err ("already loaded vendor module '%s'\n" , kvm_x86_ops.name);return -EEXIST ; }
@@ -9876,7 +9876,7 @@ int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops)return 0 ; out_unwind_ops: - kvm_x86_ops.hardware_enable =NULL ; + kvm_x86_ops.enable_virtualization_cpu =NULL ;kvm_x86_call (hardware_unsetup)(); out_mmu_exit:kvm_mmu_vendor_module_exit ();
@@ -9917,7 +9917,7 @@ void kvm_x86_vendor_exit(void)WARN_ON (static_branch_unlikely (&kvm_xen_enabled.key));#endif mutex_lock (&vendor_module_lock); - kvm_x86_ops.hardware_enable =NULL ; + kvm_x86_ops.enable_virtualization_cpu =NULL ;mutex_unlock (&vendor_module_lock); }EXPORT_SYMBOL_GPL (kvm_x86_vendor_exit);
@@ -12528,7 +12528,7 @@ int kvm_arch_enable_virtualization_cpu(void)if (ret)return ret; - ret =kvm_x86_call (hardware_enable)(); + ret =kvm_x86_call (enable_virtualization_cpu)();if (ret !=0 )return ret;
@@ -12610,7 +12610,7 @@ int kvm_arch_enable_virtualization_cpu(void)void kvm_arch_disable_virtualization_cpu (void ) { -kvm_x86_call (hardware_disable)(); +kvm_x86_call (disable_virtualization_cpu)();drop_user_return_notifiers (); }
Signature · 1 line
2.46.0.469.g59c65b2a67-goog
Re: [PATCH v4 06/10] KVM: x86: Rename virtualization {en,dis}abling APIs to match common KVM
Quoted 7 lines · Sean Christopherson
Rename x86's the per-CPU vendor hooks used to enable virtualization in hardware to align with the recently renamed arch hooks.
No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>
Sean Christopherson
maintainerPATCH v4 7/10 KVM: Add a module param to allow enabling virtualization when KVM is loaded
In reply to Sean ChristophersonAdd an on-by-default module param, enable_virt_at_load, to let userspace force virtualization to be enabled in hardware when KVM is initialized, i.e. just before /dev/kvm is exposed to userspace. Enabling virtualization during KVM initialization allows userspace to avoid the additional latency when creating/destroying the first/last VM (or more specifically, on the 0=>1 and 1=>0 edges of creation/destruction).
Now that KVM uses the cpuhp framework to do per-CPU enabling, the latency could be non-trivial as the cpuhup bringup/teardown is serialized across CPUs, e.g. the latency could be problematic for use case that need to spin up VMs quickly.
Prior to commit 10474ae8945c ("KVM: Activate Virtualization On Demand"), KVM _unconditionally_ enabled virtualization during load, i.e. there's no fundamental reason KVM needs to dynamically toggle virtualization. These days, the only known argument for not enabling virtualization is to allow KVM to be autoloaded without blocking other out-of-tree hypervisors, and such use cases can simply change the module param, e.g. via command line.
Note, the aforementioned commit also mentioned that enabling SVM (AMD's virtualization extensions) can result in "using invalid TLB entries". It's not clear whether the changelog was referring to a KVM bug, a CPU bug, or something else entirely. Regardless, leaving virtualization off by default is not a robust "fix", as any protection provided is lost the instant userspace creates the first VM.
--- .../admin-guide/kernel-parameters.txt | 17 +++++++++ virt/kvm/kvm_main.c | 35 +++++++++++++++++++ 2 files changed, 52 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +17 -0
@@ -2648,6 +2648,23 @@ Default is Y (on). + kvm.enable_virt_at_load=[KVM,ARM64,LOONGARCH,MIPS,RISCV,X86] + If enabled, KVM will enable virtualization in hardware + when KVM is loaded, and disable virtualization when KVM + is unloaded (if KVM is built as a module). + + If disabled, KVM will dynamically enable and disable + virtualization on-demand when creating and destroying + VMs, i.e. on the 0=>1 and 1=>0 transitions of the + number of VMs. + + Enabling virtualization at module lode avoids potential + latency for creation of the 0=>1 VM, as KVM serializes + virtualization enabling across all online CPUs. The + "cost" of enabling virtualization when KVM is loaded, + is that doing so may interfere with using out-of-tree + hypervisors that want to "own" virtualization hardware. + kvm.enable_vmware_backdoor=[KVM] Support VMware backdoor PV interface. Default is false (don't support).
virt/kvm/kvm_main.c +35 -0
@@ -5572,6 +5572,9 @@ static struct miscdevice kvm_dev = { };#ifdef CONFIG_KVM_GENERIC_HARDWARE_ENABLING +static bool enable_virt_at_load = true; +module_param (enable_virt_at_load, bool,0444 ); + __visible bool kvm_rebooting;EXPORT_SYMBOL_GPL (kvm_rebooting);
@@ -5722,15 +5725,39 @@ static void kvm_disable_virtualization(void)unregister_syscore_ops (&kvm_syscore_ops); cpuhp_remove_state(CPUHP_AP_KVM_ONLINE); } + +static int kvm_init_virtualization (void ) +{ +if (enable_virt_at_load) +return kvm_enable_virtualization (); + +return 0 ; +} + +static void kvm_uninit_virtualization (void ) +{ +if (enable_virt_at_load) +kvm_disable_virtualization (); +}#else /* CONFIG_KVM_GENERIC_HARDWARE_ENABLING */ static int kvm_enable_virtualization (void ) {return 0 ; } +static int kvm_init_virtualization (void ) +{ +return 0 ; +} +static void kvm_disable_virtualization (void ) { +} + +static void kvm_uninit_virtualization (void ) +{ + }#endif /* CONFIG_KVM_GENERIC_HARDWARE_ENABLING */
@@ -6475,6 +6502,10 @@ int kvm_init(unsigned vcpu_size, unsigned vcpu_align, struct module *module)kvm_gmem_init (module); + r =kvm_init_virtualization (); +if (r) + goto err_virt; +/* * Registration _must_ be the very last thing done, as this exposes * /dev/kvm to userspace, i.e. all infrastructure must be setup!
@@ -6488,6 +6519,8 @@ int kvm_init(unsigned vcpu_size, unsigned vcpu_align, struct module *module)return 0 ; err_register: +kvm_uninit_virtualization (); +err_virt:kvm_vfio_ops_exit (); err_vfio:kvm_async_pf_deinit ();
@@ -6513,6 +6546,8 @@ void kvm_exit(void)*/ misc_deregister (&kvm_dev); +kvm_uninit_virtualization (); +debugfs_remove_recursive (kvm_debugfs_dir);for_each_possible_cpu (cpu)free_cpumask_var (per_cpu (cpu_kick_mask, cpu));
Signature · 1 line
2.46.0.469.g59c65b2a67-goog
Sean Christopherson
maintainerPATCH v4 8/10 KVM: Add arch hooks for enabling/disabling virtualization
In reply to Sean ChristophersonAdd arch hooks that are invoked when KVM enables/disable virtualization. x86 will use the hooks to register an "emergency disable" callback, which is essentially an x86-specific shutdown notifier that is used when the kernel is doing an emergency reboot/shutdown/kexec.
Add comments for the declarations to help arch code understand exactly when the callbacks are invoked. Alternatively, the APIs themselves could communicate most of the same info, but kvm_arch_pre_enable_virtualization() and kvm_arch_post_disable_virtualization() are a bit cumbersome, and make it a bit less obvious that they are intended to be implemented as a pair.
--- include/linux/kvm_host.h | 14 ++++++++++++++ virt/kvm/kvm_main.c | 14 ++++++++++++++ 2 files changed, 28 insertions(+)
include/linux/kvm_host.h +14 -0
@@ -1514,6 +1514,20 @@ static inline void kvm_create_vcpu_debugfs(struct kvm_vcpu *vcpu) {}#endif #ifdef CONFIG_KVM_GENERIC_HARDWARE_ENABLING +/* +* kvm_arch_{enable,disable}_virtualization() are called on one CPU, under +* kvm_usage_lock, immediately after/before 0=>1 and 1=>0 transitions of +* kvm_usage_count, i.e. at the beginning of the generic hardware enabling +* sequence, and at the end of the generic hardware disabling sequence. +*/ +void kvm_arch_enable_virtualization (void ); +void kvm_arch_disable_virtualization (void ); +/* +* kvm_arch_{enable,disable}_virtualization_cpu() are called on "every" CPU to +* do the actual twiddling of hardware bits. The hooks are called on all +* online CPUs when KVM enables/disabled virtualization, and on a single CPU +* when that CPU is onlined/offlined (including for Resume/Suspend). +*/ int kvm_arch_enable_virtualization_cpu (void );void kvm_arch_disable_virtualization_cpu (void );#endif
virt/kvm/kvm_main.c +14 -0
@@ -5582,6 +5582,16 @@ static DEFINE_PER_CPU(bool, virtualization_enabled);static DEFINE_MUTEX (kvm_usage_lock);static int kvm_usage_count; +__weak void kvm_arch_enable_virtualization (void ) +{ + +} + +__weak void kvm_arch_disable_virtualization (void ) +{ + +} +static int kvm_enable_virtualization_cpu (void ) {if (__this_cpu_read (virtualization_enabled))
@@ -5681,6 +5691,8 @@ static int kvm_enable_virtualization(void)if (kvm_usage_count++)return 0 ; +kvm_arch_enable_virtualization (); + r =cpuhp_setup_state (CPUHP_AP_KVM_ONLINE ,"kvm/cpu:online" , kvm_online_cpu, kvm_offline_cpu);if (r)
@@ -5711,6 +5723,7 @@ static int kvm_enable_virtualization(void)unregister_syscore_ops (&kvm_syscore_ops);cpuhp_remove_state (CPUHP_AP_KVM_ONLINE ); err_cpuhp: +kvm_arch_disable_virtualization (); --kvm_usage_count;return r; }
@@ -5724,6 +5737,7 @@ static void kvm_disable_virtualization(void)unregister_syscore_ops (&kvm_syscore_ops);cpuhp_remove_state (CPUHP_AP_KVM_ONLINE ); +kvm_arch_disable_virtualization (); }static int kvm_init_virtualization (void )
Signature · 1 line
2.46.0.469.g59c65b2a67-goog
Sean Christopherson
maintainerPATCH v4 9/10 x86/reboot: Unconditionally define cpu_emergency_virt_cb typedef
In reply to Sean ChristophersonDefine cpu_emergency_virt_cb even if the kernel is being built without KVM support so that KVM can reference the typedef in asm/kvm_host.h without needing yet more #ifdefs.
No functional change intended.
--- arch/x86/include/asm/reboot.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/reboot.h +1 -1
@@ -25,8 +25,8 @@ void __noreturn machine_real_restart(unsigned int type);#define MRR_BIOS 0#define MRR_APM 1 -#if IS_ENABLED (CONFIG_KVM_INTEL ) ||IS_ENABLED (CONFIG_KVM_AMD )typedef void (cpu_emergency_virt_cb )(void ); +#if IS_ENABLED (CONFIG_KVM_INTEL ) ||IS_ENABLED (CONFIG_KVM_AMD )void cpu_emergency_register_virt_callback (cpu_emergency_virt_cb *callback);void cpu_emergency_unregister_virt_callback (cpu_emergency_virt_cb *callback);void cpu_emergency_disable_virtualization (void );
Signature · 1 line
2.46.0.469.g59c65b2a67-goog
Sean Christopherson
maintainerPATCH v4 10/10 KVM: x86: Register "emergency disable" callbacks when virt is enabled
In reply to Sean ChristophersonRegister the "disable virtualization in an emergency" callback just before KVM enables virtualization in hardware, as there is no functional need to keep the callbacks registered while KVM happens to be loaded, but is inactive, i.e. if KVM hasn't enabled virtualization.
Note, unregistering the callback every time the last VM is destroyed could have measurable latency due to the synchronize_rcu() needed to ensure all references to the callback are dropped before KVM is unloaded. But the latency should be a small fraction of the total latency of disabling virtualization across all CPUs, and userspace can set enable_virt_at_load to completely eliminate the runtime overhead.
Add a pointer in kvm_x86_ops to allow vendor code to provide its callback. There is no reason to force vendor code to do the registration, and either way KVM would need a new kvm_x86_ops hook.
--- arch/x86/include/asm/kvm_host.h | 3 +++ arch/x86/kvm/svm/svm.c | 5 +---- arch/x86/kvm/vmx/main.c | 2 ++ arch/x86/kvm/vmx/vmx.c | 6 +----- arch/x86/kvm/vmx/x86_ops.h | 1 + arch/x86/kvm/x86.c | 10 ++++++++++ 6 files changed, 18 insertions(+), 9 deletions(-)
arch/x86/include/asm/kvm_host.h +3 -0
@@ -36,6 +36,7 @@#include <asm/kvm_page_track.h> #include <asm/kvm_vcpu_regs.h> #include <asm/hyperv-tlfs.h> +#include <asm/reboot.h> #define __KVM_HAVE_ARCH_VCPU_DEBUGFS
@@ -1631,6 +1632,8 @@ struct kvm_x86_ops {int (*enable_virtualization_cpu)(void );void (*disable_virtualization_cpu)(void ); +cpu_emergency_virt_cb *emergency_disable_virtualization_cpu; +void (*hardware_unsetup)(void );bool (*has_emulated_msr)(struct kvm *kvm,u32 index);void (*vcpu_after_set_cpuid)(struct kvm_vcpu *vcpu);
arch/x86/kvm/svm/svm.c +1 -4
@@ -4982,6 +4982,7 @@ static struct kvm_x86_ops svm_x86_ops __initdata = { .hardware_unsetup = svm_hardware_unsetup, .enable_virtualization_cpu = svm_enable_virtualization_cpu, .disable_virtualization_cpu = svm_disable_virtualization_cpu, + .emergency_disable_virtualization_cpu = svm_emergency_disable_virtualization_cpu, .has_emulated_msr = svm_has_emulated_msr, .vcpu_create = svm_vcpu_create,
@@ -5410,8 +5411,6 @@ static struct kvm_x86_init_ops svm_init_ops __initdata = {static void __svm_exit (void ) {kvm_x86_vendor_exit (); - -cpu_emergency_unregister_virt_callback (svm_emergency_disable_virtualization_cpu); }static int __init svm_init (void )
@@ -5427,8 +5426,6 @@ static int __init svm_init(void)if (r)return r; -cpu_emergency_register_virt_callback (svm_emergency_disable_virtualization_cpu); -/* * Common KVM initialization _must_ come last, after this, /dev/kvm is * exposed to userspace!
arch/x86/kvm/vmx/main.c +2 -0
@@ -25,6 +25,8 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .enable_virtualization_cpu = vmx_enable_virtualization_cpu, .disable_virtualization_cpu = vmx_disable_virtualization_cpu, + .emergency_disable_virtualization_cpu = vmx_emergency_disable_virtualization_cpu, + .has_emulated_msr = vmx_has_emulated_msr, .vm_size = sizeof(struct kvm_vmx),
arch/x86/kvm/vmx/vmx.c +1 -5
@@ -755,7 +755,7 @@ static int kvm_cpu_vmxoff(void)return -EIO ; } -static void vmx_emergency_disable_virtualization_cpu (void ) +void vmx_emergency_disable_virtualization_cpu (void ) {int cpu =raw_smp_processor_id ();struct loaded_vmcs *v;
@@ -8584,8 +8584,6 @@ static void __vmx_exit(void) { allow_smaller_maxphyaddr = false; -cpu_emergency_unregister_virt_callback (vmx_emergency_disable_virtualization_cpu); -vmx_cleanup_l1d_flush (); }
@@ -8632,8 +8630,6 @@ static int __init vmx_init(void)pi_init_cpu (cpu); } -cpu_emergency_register_virt_callback (vmx_emergency_disable_virtualization_cpu); -vmx_check_vmcs12_offsets ();/*
arch/x86/kvm/vmx/x86_ops.h +1 -0
@@ -15,6 +15,7 @@ void vmx_hardware_unsetup(void);int vmx_check_processor_compat (void );int vmx_enable_virtualization_cpu (void );void vmx_disable_virtualization_cpu (void ); +void vmx_emergency_disable_virtualization_cpu (void );int vmx_vm_init (struct kvm *kvm);void vmx_vm_destroy (struct kvm *kvm);int vmx_vcpu_precreate (struct kvm *kvm);
arch/x86/kvm/x86.c +10 -0
@@ -12512,6 +12512,16 @@ void kvm_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector) }EXPORT_SYMBOL_GPL (kvm_vcpu_deliver_sipi_vector); +void kvm_arch_enable_virtualization (void ) +{ +cpu_emergency_register_virt_callback (kvm_x86_ops.emergency_disable_virtualization_cpu); +} + +void kvm_arch_disable_virtualization (void ) +{ +cpu_emergency_unregister_virt_callback (kvm_x86_ops.emergency_disable_virtualization_cpu); +} +int kvm_arch_enable_virtualization_cpu (void ) {struct kvm *kvm;
Signature · 1 line
2.46.0.469.g59c65b2a67-goog
Queued, thanks.
Paolo