Skip to contentSkip to conversation

KVM: TDX: Disable PMU virtualization for TDX VMs

PATCH Superseded FirstName LastName 3 participants 9 messages May 5 – May 13

Subsystems (4)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover KVM: TDX: Disable PMU virtualization for TDX VMs

From: Vishal Annapurve <[email protected]>

TDX module virtualizes PMU for TDX VMs [1]. KVM has limited role to play in virtualizing PMU accesses and needs additional enlightenment to support all toggles provided by TDX module.

This series disables PMU virtualization within KVM for TDX VMs, which is a safe bet until the complete set of PMU controls are implemented within KVM.

[1] Section 15.2: https://cdrdv2.intel.com/v1/dl/getContent/733575

Vishal Annapurve (2):
KVM: x86: Move the default arch state init before vm_init() call
KVM: TDX: Disable pmu virtualization for TDX VMs

 arch/x86/kvm/vmx/tdx.c |  6 ++++++
 arch/x86/kvm/x86.c     | 10 +++++-----
 2 files changed, 11 insertions(+), 5 deletions(-)
Signature · 1 line
2.54.0.545.g6539524ca2-goog
#1 lore raw

PATCH 1/2 KVM: x86: Move the default arch state init before vm_init() call

From: Vishal Annapurve <[email protected]>

Move the default kvm state initialization before vendor specific vm_init() call to allow vendor specific code to override these defaults.

Next patch will use this update to allow TDX specific vm_init() to override enable_pmu toggle.

---
 arch/x86/kvm/x86.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
arch/x86/kvm/x86.c +5 -5
index 0a1b63c63d1a..fda842adb58a 100644
@@ -13349,6 +13349,11 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 		type == KVM_X86_DEFAULT_VM || type == KVM_X86_SW_PROTECTED_VM;
 	kvm->arch.disabled_quirks = kvm_caps.inapplicable_quirks & kvm_caps.supported_quirks;
 
+	kvm->arch.default_tsc_khz = max_tsc_khz ? : tsc_khz;
+	kvm->arch.apic_bus_cycle_ns = APIC_BUS_CYCLE_NS_DEFAULT;
+	kvm->arch.guest_can_read_msr_platform_info = true;
+	kvm->arch.enable_pmu = enable_pmu;
+
 	ret = kvm_page_track_init(kvm);
 	if (ret)
 		goto out;
@@ -13372,11 +13377,6 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 	pvclock_update_vm_gtod_copy(kvm);
 	raw_spin_unlock_irqrestore(&kvm->arch.tsc_write_lock, flags);
 
-	kvm->arch.default_tsc_khz = max_tsc_khz ? : tsc_khz;
-	kvm->arch.apic_bus_cycle_ns = APIC_BUS_CYCLE_NS_DEFAULT;
-	kvm->arch.guest_can_read_msr_platform_info = true;
-	kvm->arch.enable_pmu = enable_pmu;
-
 #if IS_ENABLED(CONFIG_HYPERV)
 	spin_lock_init(&kvm->arch.hv_root_tdp_lock);
 	kvm->arch.hv_root_tdp = INVALID_PAGE;
Signature · 1 line
2.54.0.545.g6539524ca2-goog
#2 lore raw

PATCH 2/2 KVM: TDX: Disable pmu virtualization for TDX VMs

In reply to FirstName LastName

From: Vishal Annapurve <[email protected]>

TDX module virtualizes PMU for TDX VMs[1]. Host has following toggles to control the PMU functionality exposed to TDX VMs:
1) Configure TD_PARAMS to allow guests to use performance monitoring.
2) Restrict the TD to a subset of the PEBS counters if supported.
3) Limit the TD to setup a certain perfmon events using basic/enhanced event filtering.

KVM will need to be enlightened to support these toggles. Explicitly disable PMU virtualization for TDX VMs by default until such a support lands.

[1] Section 15.2: https://cdrdv2.intel.com/v1/dl/getContent/733575

---
 arch/x86/kvm/vmx/tdx.c | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/kvm/vmx/tdx.c +6 -0
index 1e47c194af53..01498c25942d 100644
@@ -638,6 +638,12 @@ int tdx_vm_init(struct kvm *kvm)
 	kvm->arch.has_private_mem = true;
 	kvm->arch.disabled_quirks |= KVM_X86_QUIRK_IGNORE_GUEST_PAT;
 
+	/*
+	 * PMU support is provide by the TDX-Module (if enabled for the VM).
+	 * From KVM's perspective, the VM doesn't have a virtual PMU.
+	 */
+	kvm->arch.enable_pmu = false;
+
 	/*
 	 * Because guest TD is protected, VMM can't parse the instruction in TD.
 	 * Instead, guest uses MMIO hypercall.  For unmodified device driver,
Signature · 1 line
2.54.0.545.g6539524ca2-goog
#3 lore raw

Re: [PATCH 2/2] KVM: TDX: Disable pmu virtualization for TDX VMs

Quoted 25 lines · FirstName LastName

From: Vishal Annapurve <[email protected]>

TDX module virtualizes PMU for TDX VMs[1]. Host has following toggles to control the PMU functionality exposed to TDX VMs:
1) Configure TD_PARAMS to allow guests to use performance monitoring.
2) Restrict the TD to a subset of the PEBS counters if supported.
3) Limit the TD to setup a certain perfmon events using basic/enhanced event filtering.

KVM will need to be enlightened to support these toggles. Explicitly disable PMU virtualization for TDX VMs by default until such a support lands.

[1] Section 15.2: https://cdrdv2.intel.com/v1/dl/getContent/733575

---
 arch/x86/kvm/vmx/tdx.c | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/kvm/vmx/tdx.c +0 -0
index 1e47c194af53..01498c25942d 100644

arch/x86/kvm/vmx/tdx.c int tdx_vm_init(struct kvm *kvm) · line 645

Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +5 -0
@@ -638,6 +638,12 @@ int tdx_vm_init(struct kvm *kvm)
 	kvm->arch.has_private_mem = true;
 	kvm->arch.disabled_quirks |= KVM_X86_QUIRK_IGNORE_GUEST_PAT;
 
+	/*
+	 * PMU support is provide by the TDX-Module (if enabled for the VM).
+	 * From KVM's perspective, the VM doesn't have a virtual PMU.
+	 */
+	kvm->arch.enable_pmu = false;

Gah, I forgot that KVM_CAP_PMU_CAPABILITY allows re-enabling PMU support (which is really quite annoying). Unless we want to risk breaking userspace, the best idea I can come up with is to add a has_protected_pmu flag, and then disallow KVM_CAP_PMU_CAPABILITY.

The question then becomes, do we keep patch 1 and also clear enable_pmu in tdx.c, or do we keep the ordering and have kvm_arch_init_vm() consume has_protected_pmu? Neither one is particularly awesome :-/

arch/x86/include/asm/kvm_host.h +1 -0
index c470e40a00aa..8371dcaaed1a 100644
@@ -1422,6 +1422,7 @@ struct kvm_arch {
        bool has_private_mem;
        bool has_protected_state;
        bool has_protected_eoi;
+       bool has_protected_pmu;
        bool pre_fault_allowed;
        struct hlist_head *mmu_page_hash;
        struct list_head active_mmu_pages;
arch/x86/kvm/vmx/tdx.c +1 -0
index 04ce321ebdf3..3ba295bd44f8 100644
@@ -635,6 +635,7 @@ int tdx_vm_init(struct kvm *kvm)
         * i.e. all EOIs are accelerated and never trigger exits.
         */
        kvm->arch.has_protected_eoi = true;
+       kvm->arch.has_protected_pmu = true;
        kvm->arch.has_private_mem = true;
        kvm->arch.disabled_quirks |= KVM_X86_QUIRK_IGNORE_GUEST_PAT;
 
arch/x86/kvm/x86.c +2 -1
index 0a1b63c63d1a..57d78255c80c 100644
@@ -6910,7 +6910,8 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
                        break;
 
                mutex_lock(&kvm->lock);
-               if (!kvm->created_vcpus && !kvm->arch.created_mediated_pmu) {
+               if (!kvm->created_vcpus && !kvm->arch.created_mediated_pmu &&
+                   !kvm->arch.has_protected_pmu) {
                        kvm->arch.enable_pmu = !(cap->args[0] & KVM_PMU_CAP_DISABLE);
                        r = 0;
                }
#4 lore raw

Re: [PATCH 2/2] KVM: TDX: Disable pmu virtualization for TDX VMs

Quoted 41 lines · Sean Christopherson
Quoted 25 lines · FirstName LastName

From: Vishal Annapurve <[email protected]>

TDX module virtualizes PMU for TDX VMs[1]. Host has following toggles to control the PMU functionality exposed to TDX VMs:
1) Configure TD_PARAMS to allow guests to use performance monitoring.
2) Restrict the TD to a subset of the PEBS counters if supported.
3) Limit the TD to setup a certain perfmon events using basic/enhanced event filtering.

KVM will need to be enlightened to support these toggles. Explicitly disable PMU virtualization for TDX VMs by default until such a support lands.

[1] Section 15.2: https://cdrdv2.intel.com/v1/dl/getContent/733575

---
 arch/x86/kvm/vmx/tdx.c | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/kvm/vmx/tdx.c +0 -0
index 1e47c194af53..01498c25942d 100644

arch/x86/kvm/vmx/tdx.c int tdx_vm_init(struct kvm *kvm) · line 645

Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +5 -0
@@ -638,6 +638,12 @@ int tdx_vm_init(struct kvm *kvm)
      kvm->arch.has_private_mem = true;
      kvm->arch.disabled_quirks |= KVM_X86_QUIRK_IGNORE_GUEST_PAT;
 
+     /*
+      * PMU support is provide by the TDX-Module (if enabled for the VM).
+      * From KVM's perspective, the VM doesn't have a virtual PMU.
+      */
+     kvm->arch.enable_pmu = false;

Gah, I forgot that KVM_CAP_PMU_CAPABILITY allows re-enabling PMU support (which is really quite annoying). Unless we want to risk breaking userspace, the best idea I can come up with is to add a has_protected_pmu flag, and then disallow KVM_CAP_PMU_CAPABILITY.

Quoted 3 lines

The question then becomes, do we keep patch 1 and also clear enable_pmu in tdx.c, or do we keep the ordering and have kvm_arch_init_vm() consume has_protected_pmu? Neither one is particularly awesome :-/

I am inclined to send the v2 with the latter option i.e. keep the ordering the same and have kvm_arch_init_vm() consume has_protected_pmu. Let's see how that goes.

Quoted 39 lines
arch/x86/include/asm/kvm_host.h +1 -0
index c470e40a00aa..8371dcaaed1a 100644
@@ -1422,6 +1422,7 @@ struct kvm_arch {
        bool has_private_mem;
        bool has_protected_state;
        bool has_protected_eoi;
+       bool has_protected_pmu;
        bool pre_fault_allowed;
        struct hlist_head *mmu_page_hash;
        struct list_head active_mmu_pages;
arch/x86/kvm/vmx/tdx.c +1 -0
index 04ce321ebdf3..3ba295bd44f8 100644
@@ -635,6 +635,7 @@ int tdx_vm_init(struct kvm *kvm)
         * i.e. all EOIs are accelerated and never trigger exits.
         */
        kvm->arch.has_protected_eoi = true;
+       kvm->arch.has_protected_pmu = true;
        kvm->arch.has_private_mem = true;
        kvm->arch.disabled_quirks |= KVM_X86_QUIRK_IGNORE_GUEST_PAT;
 
arch/x86/kvm/x86.c +2 -1
index 0a1b63c63d1a..57d78255c80c 100644
@@ -6910,7 +6910,8 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
                        break;
 
                mutex_lock(&kvm->lock);
-               if (!kvm->created_vcpus && !kvm->arch.created_mediated_pmu) {
+               if (!kvm->created_vcpus && !kvm->arch.created_mediated_pmu &&
+                   !kvm->arch.has_protected_pmu) {
                        kvm->arch.enable_pmu = !(cap->args[0] & KVM_PMU_CAP_DISABLE);
                        r = 0;
                }
#5 lore raw

Re: [PATCH 2/2] KVM: TDX: Disable pmu virtualization for TDX VMs

In reply to Sean Christopherson
Quoted 3 lines · Sean Christopherson

The question then becomes, do we keep patch 1 and also clear enable_pmu in tdx.c, or do we keep the ordering and have kvm_arch_init_vm() consume has_protected_pmu? Neither one is particularly awesome :-/

Maybe keeping patch 1 is slightly better? This allows the vm_init() to toggle all the values and flags that the default ones don't fit. If we go with latter, when there's similar cases where we need more flags, then kvm_arch_init_vm() needs to consume more flags.

This is also consistent with how KVM sets the existing kvm-

Quoted 1 line

arch.shadow_mmio_value: KVM sets the default in kvm_arch_init_vm() ->

kvm_mmu_init_vm(), and then TDX changes it to other values in tdx_vm_init().

#6 lore raw

Re: [PATCH 2/2] KVM: TDX: Disable pmu virtualization for TDX VMs

Quoted 10 lines · Kai
Quoted 3 lines · Sean Christopherson

The question then becomes, do we keep patch 1 and also clear enable_pmu in tdx.c, or do we keep the ordering and have kvm_arch_init_vm() consume has_protected_pmu? Neither one is particularly awesome :-/

Maybe keeping patch 1 is slightly better? This allows the vm_init() to toggle all the values and flags that the default ones don't fit. If we go with latter, when there's similar cases where we need more flags, then kvm_arch_init_vm() needs to consume more flags.

For now I have sent a v2 without moving around the default flags. It's definitely better to move them up as done in patch1, but I think its ok to keep the flow unchanged untill we have a more concrete reason to do the move.

Quoted 4 lines

This is also consistent with how KVM sets the existing kvm-

Quoted 1 line

arch.shadow_mmio_value: KVM sets the default in kvm_arch_init_vm() ->

kvm_mmu_init_vm(), and then TDX changes it to other values in tdx_vm_init().

#7 lore raw

Re: [PATCH 2/2] KVM: TDX: Disable pmu virtualization for TDX VMs

Quoted 7 lines · Vishal Annapurve
Quoted 6 lines · Kai
Quoted 3 lines · Sean Christopherson

The question then becomes, do we keep patch 1 and also clear enable_pmu in tdx.c, or do we keep the ordering and have kvm_arch_init_vm() consume has_protected_pmu? Neither one is particularly awesome :-/

Quoted 7 lines
Quoted 4 lines

Maybe keeping patch 1 is slightly better? This allows the vm_init() to toggle all the values and flags that the default ones don't fit. If we go with latter, when there's similar cases where we need more flags, then kvm_arch_init_vm() needs to consume more flags.

For now I have sent a v2 without moving around the default flags. It's definitely better to move them up as done in patch1,

Yes and no. No matter what we do, we're subtly relying on ordering between initialization in kvm_arch_init_vm() and kvm_x86_ops.vm_init(). I don't see an easy way around that.

For these fields:

	kvm->arch.default_tsc_khz = max_tsc_khz ? : tsc_khz;
	kvm->arch.apic_bus_cycle_ns = APIC_BUS_CYCLE_NS_DEFAULT;
	kvm->arch.guest_can_read_msr_platform_info = true;

and any other fields that KVM initializes with hardcoded values, from globals, or from local variables (including @type), I 100% agree that they should be initialized before calling kvm_x86_ops.vm_init().

But with the proposed has_protected_pmu, I'm leaning towards keeping this:

	kvm->arch.enable_pmu = enable_pmu && !kvm->arch.has_protected_pmu;

which obviously means keeping it after kvm_x86_ops.vm_init(). Because if we make vendor code responsible for clearing kvm->arch.enable_pmu, then (a) we may end up with duplicate code for SEV-ES+ (does SEV-ES/SNP allow for a virtual/emulated PMU?), and (b) the x86 code _looks_ wrong, because kvm_arch_init_vm() will ignore kvm->arch.has_protected_pmu, while the KVM_CAP_PMU_CAPABILITY case-statement does not.

#8 lore raw

Re: [PATCH 2/2] KVM: TDX: Disable pmu virtualization for TDX VMs

Quoted 31 lines · Sean Christopherson
Quoted 7 lines · Vishal Annapurve
Quoted 6 lines · Kai
Quoted 3 lines · Sean Christopherson

The question then becomes, do we keep patch 1 and also clear enable_pmu in tdx.c, or do we keep the ordering and have kvm_arch_init_vm() consume has_protected_pmu? Neither one is particularly awesome :-/

Quoted 7 lines
Quoted 4 lines

Maybe keeping patch 1 is slightly better? This allows the vm_init() to toggle all the values and flags that the default ones don't fit. If we go with latter, when there's similar cases where we need more flags, then kvm_arch_init_vm() needs to consume more flags.

For now I have sent a v2 without moving around the default flags. It's definitely better to move them up as done in patch1,

Yes and no. No matter what we do, we're subtly relying on ordering between initialization in kvm_arch_init_vm() and kvm_x86_ops.vm_init(). I don't see an easy way around that.

For these fields:

        kvm->arch.default_tsc_khz = max_tsc_khz ? : tsc_khz;
        kvm->arch.apic_bus_cycle_ns = APIC_BUS_CYCLE_NS_DEFAULT;
        kvm->arch.guest_can_read_msr_platform_info = true;

and any other fields that KVM initializes with hardcoded values, from globals, or from local variables (including @type), I 100% agree that they should be initialized before calling kvm_x86_ops.vm_init().

Quoted 7 lines

But with the proposed has_protected_pmu, I'm leaning towards keeping this:

        kvm->arch.enable_pmu = enable_pmu && !kvm->arch.has_protected_pmu;

which obviously means keeping it after kvm_x86_ops.vm_init(). Because if we make vendor code responsible for clearing kvm->arch.enable_pmu, then (a) we may end up with duplicate code for SEV-ES+ (does SEV-ES/SNP allow for a virtual/emulated

Michael,

I believe has_protected_pmu should be set for SNP VMs as well based on my read of AMD SNP spec which suggests PMC virtualization managed by hardware for SNP VMs and not by KVM.

Quoted 3 lines

PMU?), and (b) the x86 code _looks_ wrong, because kvm_arch_init_vm() will ignore kvm->arch.has_protected_pmu, while the KVM_CAP_PMU_CAPABILITY case-statement does not.