KVM: TDX: Disable PMU virtualization for TDX VMs
Subsystems (4)
cover KVM: TDX: Disable PMU virtualization for TDX VMs
From: Vishal Annapurve <[email protected]>
TDX module virtualizes PMU for TDX VMs [1]. KVM has limited role to play in virtualizing PMU accesses and needs additional enlightenment to support all toggles provided by TDX module.
This series disables PMU virtualization within KVM for TDX VMs, which is a safe bet until the complete set of PMU controls are implemented within KVM.
[1] Section 15.2: https://cdrdv2.intel.com/v1/dl/getContent/733575
Vishal Annapurve (2):
KVM: x86: Move the default arch state init before vm_init() call
KVM: TDX: Disable pmu virtualization for TDX VMs
arch/x86/kvm/vmx/tdx.c | 6 ++++++ arch/x86/kvm/x86.c | 10 +++++----- 2 files changed, 11 insertions(+), 5 deletions(-)
Signature · 1 line
2.54.0.545.g6539524ca2-goog
PATCH 1/2 KVM: x86: Move the default arch state init before vm_init() call
From: Vishal Annapurve <[email protected]>
Move the default kvm state initialization before vendor specific vm_init() call to allow vendor specific code to override these defaults.
Next patch will use this update to allow TDX specific vm_init() to override enable_pmu toggle.
--- arch/x86/kvm/x86.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
arch/x86/kvm/x86.c +5 -5
@@ -13349,6 +13349,11 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) type ==KVM_X86_DEFAULT_VM || type ==KVM_X86_SW_PROTECTED_VM ; kvm->arch.disabled_quirks = kvm_caps.inapplicable_quirks & kvm_caps.supported_quirks; + kvm->arch.default_tsc_khz = max_tsc_khz ? : tsc_khz; + kvm->arch.apic_bus_cycle_ns =APIC_BUS_CYCLE_NS_DEFAULT ; + kvm->arch.guest_can_read_msr_platform_info = true; + kvm->arch.enable_pmu = enable_pmu; + ret =kvm_page_track_init (kvm);if (ret) goto out;
@@ -13372,11 +13377,6 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)pvclock_update_vm_gtod_copy (kvm);raw_spin_unlock_irqrestore (&kvm->arch.tsc_write_lock, flags); - kvm->arch.default_tsc_khz = max_tsc_khz ? : tsc_khz; - kvm->arch.apic_bus_cycle_ns =APIC_BUS_CYCLE_NS_DEFAULT ; - kvm->arch.guest_can_read_msr_platform_info = true; - kvm->arch.enable_pmu = enable_pmu; -#if IS_ENABLED (CONFIG_HYPERV )spin_lock_init (&kvm->arch.hv_root_tdp_lock); kvm->arch.hv_root_tdp =INVALID_PAGE ;
Signature · 1 line
2.54.0.545.g6539524ca2-goog
PATCH 2/2 KVM: TDX: Disable pmu virtualization for TDX VMs
In reply to FirstName LastNameFrom: Vishal Annapurve <[email protected]>
TDX module virtualizes PMU for TDX VMs[1]. Host has following toggles to control the PMU functionality exposed to TDX VMs:
1) Configure TD_PARAMS to allow guests to use performance monitoring.
2) Restrict the TD to a subset of the PEBS counters if supported.
3) Limit the TD to setup a certain perfmon events using basic/enhanced event filtering.
KVM will need to be enlightened to support these toggles. Explicitly disable PMU virtualization for TDX VMs by default until such a support lands.
[1] Section 15.2: https://cdrdv2.intel.com/v1/dl/getContent/733575
--- arch/x86/kvm/vmx/tdx.c | 6 ++++++ 1 file changed, 6 insertions(+)
arch/x86/kvm/vmx/tdx.c +6 -0
@@ -638,6 +638,12 @@ int tdx_vm_init(struct kvm *kvm) kvm->arch.has_private_mem = true; kvm->arch.disabled_quirks |=KVM_X86_QUIRK_IGNORE_GUEST_PAT ; +/* +* PMU support is provide by the TDX-Module (if enabled for the VM). +* From KVM's perspective, the VM doesn't have a virtual PMU. +*/ + kvm->arch.enable_pmu = false; +/* * Because guest TD is protected, VMM can't parse the instruction in TD. * Instead, guest uses MMIO hypercall. For unmodified device driver,
Signature · 1 line
2.54.0.545.g6539524ca2-goog
Sean Christopherson
maintainerRe: [PATCH 2/2] KVM: TDX: Disable pmu virtualization for TDX VMs
Quoted 25 lines · FirstName LastName
From: Vishal Annapurve <[email protected]>
TDX module virtualizes PMU for TDX VMs[1]. Host has following toggles to control the PMU functionality exposed to TDX VMs:
1) ConfigureTD_PARAMSto allow guests to use performance monitoring.
2) Restrict the TD to a subset of the PEBS counters if supported.
3) Limit the TD to setup a certain perfmon events using basic/enhanced event filtering.KVM will need to be enlightened to support these toggles. Explicitly disable PMU virtualization for TDX VMs by default until such a support lands.
[1] Section 15.2: https://cdrdv2.intel.com/v1/dl/getContent/733575
Suggested-by Sean Christopherson <[email protected]>Signed-off-by Vishal Annapurve <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 6 ++++++ 1 file changed, 6 insertions(+)arch/x86/kvm/vmx/tdx.c +0
-0
arch/x86/kvm/vmx/tdx.c int tdx_vm_init(struct kvm *kvm) · line 645
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +5
-0@@ -638,6 +638,12 @@ int tdx_vm_init(struct kvm *kvm) kvm->arch.has_private_mem = true; kvm->arch.disabled_quirks |=KVM_X86_QUIRK_IGNORE_GUEST_PAT ; +/* +* PMU support is provide by the TDX-Module (if enabled for the VM). +* From KVM's perspective, the VM doesn't have a virtual PMU. +*/ + kvm->arch.enable_pmu = false;
Gah, I forgot that KVM_CAP_PMU_CAPABILITY allows re-enabling PMU support (which is really quite annoying). Unless we want to risk breaking userspace, the best idea I can come up with is to add a has_protected_pmu flag, and then disallow KVM_CAP_PMU_CAPABILITY.
The question then becomes, do we keep patch 1 and also clear enable_pmu in tdx.c, or do we keep the ordering and have kvm_arch_init_vm() consume has_protected_pmu? Neither one is particularly awesome :-/
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1422,6 +1422,7 @@ struct kvm_arch {bool has_private_mem;bool has_protected_state;bool has_protected_eoi; +bool has_protected_pmu;bool pre_fault_allowed;struct hlist_head *mmu_page_hash;struct list_head active_mmu_pages;
arch/x86/kvm/vmx/tdx.c +1 -0
@@ -635,6 +635,7 @@ int tdx_vm_init(struct kvm *kvm)* i.e. all EOIs are accelerated and never trigger exits. */ kvm->arch.has_protected_eoi = true; + kvm->arch.has_protected_pmu = true; kvm->arch.has_private_mem = true; kvm->arch.disabled_quirks |=KVM_X86_QUIRK_IGNORE_GUEST_PAT ;
arch/x86/kvm/x86.c +2 -1
@@ -6910,7 +6910,8 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,break ;mutex_lock (&kvm->lock); -if (!kvm->created_vcpus && !kvm->arch.created_mediated_pmu) { +if (!kvm->created_vcpus && !kvm->arch.created_mediated_pmu && + !kvm->arch.has_protected_pmu) { kvm->arch.enable_pmu = !(cap->args[0 ] &KVM_PMU_CAP_DISABLE ); r =0 ; }
Re: [PATCH 2/2] KVM: TDX: Disable pmu virtualization for TDX VMs
Quoted 41 lines · Sean Christopherson
Quoted 25 lines · FirstName LastName
From: Vishal Annapurve <[email protected]>
TDX module virtualizes PMU for TDX VMs[1]. Host has following toggles to control the PMU functionality exposed to TDX VMs:
1) ConfigureTD_PARAMSto allow guests to use performance monitoring.
2) Restrict the TD to a subset of the PEBS counters if supported.
3) Limit the TD to setup a certain perfmon events using basic/enhanced event filtering.KVM will need to be enlightened to support these toggles. Explicitly disable PMU virtualization for TDX VMs by default until such a support lands.
[1] Section 15.2: https://cdrdv2.intel.com/v1/dl/getContent/733575
Suggested-by Sean Christopherson <[email protected]>Signed-off-by Vishal Annapurve <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 6 ++++++ 1 file changed, 6 insertions(+)arch/x86/kvm/vmx/tdx.c +0
-0arch/x86/kvm/vmx/tdx.c int tdx_vm_init(struct kvm *kvm) · line 645
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +5
-0@@ -638,6 +638,12 @@ int tdx_vm_init(struct kvm *kvm) kvm->arch.has_private_mem = true; kvm->arch.disabled_quirks |=KVM_X86_QUIRK_IGNORE_GUEST_PAT ; +/* +* PMU support is provide by the TDX-Module (if enabled for the VM). +* From KVM's perspective, the VM doesn't have a virtual PMU. +*/ + kvm->arch.enable_pmu = false;Gah, I forgot that
KVM_CAP_PMU_CAPABILITYallows re-enabling PMU support (which is really quite annoying). Unless we want to risk breaking userspace, the best idea I can come up with is to add ahas_protected_pmuflag, and then disallowKVM_CAP_PMU_CAPABILITY.
Quoted 3 lines
The question then becomes, do we keep patch 1 and also clear
enable_pmuintdx.c, or do we keep the ordering and havekvm_arch_init_vm()consumehas_protected_pmu? Neither one is particularly awesome :-/
I am inclined to send the v2 with the latter option i.e. keep the ordering the same and have kvm_arch_init_vm() consume has_protected_pmu. Let's see how that goes.
Quoted 39 lines
arch/x86/include/asm/kvm_host.h +1
-0@@ -1422,6 +1422,7 @@ struct kvm_arch {bool has_private_mem;bool has_protected_state;bool has_protected_eoi; +bool has_protected_pmu;bool pre_fault_allowed;struct hlist_head *mmu_page_hash;struct list_head active_mmu_pages;arch/x86/kvm/vmx/tdx.c +1
-0@@ -635,6 +635,7 @@ int tdx_vm_init(struct kvm *kvm)* i.e. all EOIs are accelerated and never trigger exits. */ kvm->arch.has_protected_eoi = true; + kvm->arch.has_protected_pmu = true; kvm->arch.has_private_mem = true; kvm->arch.disabled_quirks |=KVM_X86_QUIRK_IGNORE_GUEST_PAT ;arch/x86/kvm/x86.c +2
-1@@ -6910,7 +6910,8 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,break ;mutex_lock (&kvm->lock); -if (!kvm->created_vcpus && !kvm->arch.created_mediated_pmu) { +if (!kvm->created_vcpus && !kvm->arch.created_mediated_pmu && + !kvm->arch.has_protected_pmu) { kvm->arch.enable_pmu = !(cap->args[0 ] &KVM_PMU_CAP_DISABLE ); r =0 ; }
Re: [PATCH 2/2] KVM: TDX: Disable pmu virtualization for TDX VMs
In reply to Sean ChristophersonQuoted 3 lines · Sean Christopherson
The question then becomes, do we keep patch 1 and also clear
enable_pmuintdx.c, or do we keep the ordering and havekvm_arch_init_vm()consumehas_protected_pmu? Neither one is particularly awesome :-/
Maybe keeping patch 1 is slightly better? This allows the vm_init() to toggle all the values and flags that the default ones don't fit. If we go with latter, when there's similar cases where we need more flags, then kvm_arch_init_vm() needs to consume more flags.
This is also consistent with how KVM sets the existing kvm-
Quoted 1 line
arch.shadow_mmio_value: KVM sets the default in
kvm_arch_init_vm()->
kvm_mmu_init_vm(), and then TDX changes it to other values in tdx_vm_init().
Re: [PATCH 2/2] KVM: TDX: Disable pmu virtualization for TDX VMs
Quoted 10 lines · Kai
Quoted 3 lines · Sean Christopherson
The question then becomes, do we keep patch 1 and also clear
enable_pmuintdx.c, or do we keep the ordering and havekvm_arch_init_vm()consumehas_protected_pmu? Neither one is particularly awesome :-/Maybe keeping patch 1 is slightly better? This allows the
vm_init()to toggle all the values and flags that the default ones don't fit. If we go with latter, when there's similar cases where we need more flags, thenkvm_arch_init_vm()needs to consume more flags.
For now I have sent a v2 without moving around the default flags. It's definitely better to move them up as done in patch1, but I think its ok to keep the flow unchanged untill we have a more concrete reason to do the move.
Quoted 4 lines
This is also consistent with how KVM sets the existing kvm-
Quoted 1 line
arch.shadow_mmio_value: KVM sets the default in
kvm_arch_init_vm()->
kvm_mmu_init_vm(), and then TDX changes it to other values intdx_vm_init().
Sean Christopherson
maintainerRe: [PATCH 2/2] KVM: TDX: Disable pmu virtualization for TDX VMs
Quoted 7 lines · Vishal Annapurve
Quoted 6 lines · Kai
Quoted 3 lines · Sean Christopherson
The question then becomes, do we keep patch 1 and also clear
enable_pmuintdx.c, or do we keep the ordering and havekvm_arch_init_vm()consumehas_protected_pmu? Neither one is particularly awesome :-/
Quoted 7 lines
Quoted 4 lines
Maybe keeping patch 1 is slightly better? This allows the
vm_init()to toggle all the values and flags that the default ones don't fit. If we go with latter, when there's similar cases where we need more flags, thenkvm_arch_init_vm()needs to consume more flags.For now I have sent a v2 without moving around the default flags. It's definitely better to move them up as done in patch1,
Yes and no. No matter what we do, we're subtly relying on ordering between initialization in kvm_arch_init_vm() and kvm_x86_ops.vm_init(). I don't see an easy way around that.
For these fields:
kvm->arch.default_tsc_khz = max_tsc_khz ? : tsc_khz; kvm->arch.apic_bus_cycle_ns = APIC_BUS_CYCLE_NS_DEFAULT; kvm->arch.guest_can_read_msr_platform_info = true;
and any other fields that KVM initializes with hardcoded values, from globals, or from local variables (including @type), I 100% agree that they should be initialized before calling kvm_x86_ops.vm_init().
But with the proposed has_protected_pmu, I'm leaning towards keeping this:
kvm->arch.enable_pmu = enable_pmu && !kvm->arch.has_protected_pmu;
which obviously means keeping it after kvm_x86_ops.vm_init(). Because if we make vendor code responsible for clearing kvm->arch.enable_pmu, then (a) we may end up with duplicate code for SEV-ES+ (does SEV-ES/SNP allow for a virtual/emulated PMU?), and (b) the x86 code _looks_ wrong, because kvm_arch_init_vm() will ignore kvm->arch.has_protected_pmu, while the KVM_CAP_PMU_CAPABILITY case-statement does not.
Re: [PATCH 2/2] KVM: TDX: Disable pmu virtualization for TDX VMs
Quoted 31 lines · Sean Christopherson
Quoted 7 lines · Vishal Annapurve
Quoted 6 lines · Kai
Quoted 3 lines · Sean Christopherson
The question then becomes, do we keep patch 1 and also clear
enable_pmuintdx.c, or do we keep the ordering and havekvm_arch_init_vm()consumehas_protected_pmu? Neither one is particularly awesome :-/Quoted 7 lines
Quoted 4 lines
Maybe keeping patch 1 is slightly better? This allows the
vm_init()to toggle all the values and flags that the default ones don't fit. If we go with latter, when there's similar cases where we need more flags, thenkvm_arch_init_vm()needs to consume more flags.For now I have sent a v2 without moving around the default flags. It's definitely better to move them up as done in patch1,
Yes and no. No matter what we do, we're subtly relying on ordering between initialization in
kvm_arch_init_vm()andkvm_x86_ops.vm_init(). I don't see an easy way around that.For these fields:
kvm->arch.default_tsc_khz = max_tsc_khz ? : tsc_khz; kvm->arch.apic_bus_cycle_ns = APIC_BUS_CYCLE_NS_DEFAULT; kvm->arch.guest_can_read_msr_platform_info = true;and any other fields that KVM initializes with hardcoded values, from globals, or from local variables (including @type), I 100% agree that they should be initialized before calling
kvm_x86_ops.vm_init().
Quoted 7 lines
But with the proposed
has_protected_pmu, I'm leaning towards keeping this:kvm->arch.enable_pmu = enable_pmu && !kvm->arch.has_protected_pmu;which obviously means keeping it after
kvm_x86_ops.vm_init(). Because if we make vendor code responsible for clearingkvm->arch.enable_pmu, then (a) we may end up with duplicate code for SEV-ES+ (does SEV-ES/SNP allow for a virtual/emulated
Michael,
I believe has_protected_pmu should be set for SNP VMs as well based on my read of AMD SNP spec which suggests PMC virtualization managed by hardware for SNP VMs and not by KVM.
Quoted 3 lines
PMU?), and (b) the x86 code _looks_ wrong, because
kvm_arch_init_vm()will ignorekvm->arch.has_protected_pmu, while theKVM_CAP_PMU_CAPABILITYcase-statement does not.