Skip to contentSkip to conversation

KVM: x86: Introduce has_protected_pmu state for TDX VMs

PATCH Superseded Vishal Annapurve 2 participants 4 messages May 7

Subsystems (4)

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 KVM: x86: Introduce has_protected_pmu state for TDX VMs

PMU state for TDX VMs is virtualized by TDX Module [1]. Host has following toggles to control the PMU functionality exposed to TDX VMs:
1) Configure TD_PARAMS to allow guests to use performance monitoring.
2) Restrict the TD to a subset of the PEBS counters if supported.
3) Limit the TD to setup a certain perfmon events using basic/enhanced event filtering.

KVM will need to be enlightened to support these toggles. Introduce has_protected_pmu state to track the pmu state for such scenarios and explicitly set the has_protected_pmu flag for TDX VMs.

If pmu state is protected:
1) Disable KVM's PMU virtualization framework as additional enlightenment is needed within KVM to control/manage the visibility of PMU state to such VMs.
2) Disallow userspace VMM from toggling PMU virtualization state using KVM_CAP_PMU_CAPABILITY.

[1] Section 15.2: https://cdrdv2.intel.com/v1/dl/getContent/733575

---
v2 -> v3:
- Squashed two patches into a single patch as per feedback from Sean.
- Dropped the cover letter.

v2: https://lore.kernel.org/kvm/[email protected]/#t

 arch/x86/include/asm/kvm_host.h | 1 +
 arch/x86/kvm/vmx/tdx.c          | 6 ++++++
 arch/x86/kvm/x86.c              | 8 ++++++--
 3 files changed, 13 insertions(+), 2 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
index c470e40a00aa..8371dcaaed1a 100644
@@ -1422,6 +1422,7 @@ struct kvm_arch {
 	bool has_private_mem;
 	bool has_protected_state;
 	bool has_protected_eoi;
+	bool has_protected_pmu;
 	bool pre_fault_allowed;
 	struct hlist_head *mmu_page_hash;
 	struct list_head active_mmu_pages;
arch/x86/kvm/vmx/tdx.c +6 -0
index 1e47c194af53..eb4b4518e6f0 100644
@@ -638,6 +638,12 @@ int tdx_vm_init(struct kvm *kvm)
 	kvm->arch.has_private_mem = true;
 	kvm->arch.disabled_quirks |= KVM_X86_QUIRK_IGNORE_GUEST_PAT;
 
+	/*
+	 * PMU support is provided by the TDX-Module (if enabled for the VM).
+	 * From KVM's perspective, the VM doesn't have a virtual PMU.
+	 */
+	kvm->arch.has_protected_pmu = true;
+
 	/*
 	 * Because guest TD is protected, VMM can't parse the instruction in TD.
 	 * Instead, guest uses MMIO hypercall.  For unmodified device driver,
arch/x86/kvm/x86.c +6 -2
index 0a1b63c63d1a..99a383455d46 100644
@@ -6910,7 +6910,8 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 			break;
 
 		mutex_lock(&kvm->lock);
-		if (!kvm->created_vcpus && !kvm->arch.created_mediated_pmu) {
+		if (!kvm->created_vcpus && !kvm->arch.created_mediated_pmu &&
+		    !kvm->arch.has_protected_pmu) {
 			kvm->arch.enable_pmu = !(cap->args[0] & KVM_PMU_CAP_DISABLE);
 			r = 0;
 		}
@@ -13375,7 +13376,10 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 	kvm->arch.default_tsc_khz = max_tsc_khz ? : tsc_khz;
 	kvm->arch.apic_bus_cycle_ns = APIC_BUS_CYCLE_NS_DEFAULT;
 	kvm->arch.guest_can_read_msr_platform_info = true;
-	kvm->arch.enable_pmu = enable_pmu;
+	if (kvm->arch.has_protected_pmu)
+		kvm->arch.enable_pmu = false;
+	else
+		kvm->arch.enable_pmu = enable_pmu;
 
 #if IS_ENABLED(CONFIG_HYPERV)
 	spin_lock_init(&kvm->arch.hv_root_tdp_lock);
Signature · 1 line
2.54.0.563.g4f69b47b94-goog
#1 lore raw
Quoted 27 lines · Vishal Annapurve

PMU state for TDX VMs is virtualized by TDX Module [1]. Host has following toggles to control the PMU functionality exposed to TDX VMs:
1) Configure TD_PARAMS to allow guests to use performance monitoring.
2) Restrict the TD to a subset of the PEBS counters if supported.
3) Limit the TD to setup a certain perfmon events using basic/enhanced event filtering.

KVM will need to be enlightened to support these toggles. Introduce has_protected_pmu state to track the pmu state for such scenarios and explicitly set the has_protected_pmu flag for TDX VMs.

If pmu state is protected:
1) Disable KVM's PMU virtualization framework as additional enlightenment is needed within KVM to control/manage the visibility of PMU state to such VMs.
2) Disallow userspace VMM from toggling PMU virtualization state using KVM_CAP_PMU_CAPABILITY.

[1] Section 15.2: https://cdrdv2.intel.com/v1/dl/getContent/733575

---
v2 -> v3:
- Squashed two patches into a single patch as per feedback from Sean.
- Dropped the cover letter.

...

arch/x86/kvm/x86.c int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) · line 13382

Quoted 9 lines
arch/x86/kvm/x86.c +4 -1
@@ -13375,7 +13376,10 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 	kvm->arch.default_tsc_khz = max_tsc_khz ? : tsc_khz;
 	kvm->arch.apic_bus_cycle_ns = APIC_BUS_CYCLE_NS_DEFAULT;
 	kvm->arch.guest_can_read_msr_platform_info = true;
-	kvm->arch.enable_pmu = enable_pmu;
+	if (kvm->arch.has_protected_pmu)
+		kvm->arch.enable_pmu = false;
+	else
+		kvm->arch.enable_pmu = enable_pmu;

Once more, with feeling :-)

	kvm->arch.enable_pmu = enable_pmu && !kvm->arch.has_protected_pmu;

No need for a v4, I'll fixup when applying (assuming this is indeed the directin we take).

Quoted 6 lines
arch/x86/kvm/x86.c +0 -0
 #if IS_ENABLED(CONFIG_HYPERV)
 	spin_lock_init(&kvm->arch.hv_root_tdp_lock);
Signature · 1 line
2.54.0.563.g4f69b47b94-goog
#2 lore raw
In reply to Vishal Annapurve

arch/x86/kvm/x86.c int kvm_vm_ioctl_enable_cap(struct kvm *kvm, · line 6914

Quoted 11 lines · Vishal Annapurve
arch/x86/kvm/x86.c +2 -1
index 0a1b63c63d1a..99a383455d46 100644
@@ -6910,7 +6910,8 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 			break;
 
 		mutex_lock(&kvm->lock);
-		if (!kvm->created_vcpus && !kvm->arch.created_mediated_pmu) {
+		if (!kvm->created_vcpus && !kvm->arch.created_mediated_pmu &&
+		    !kvm->arch.has_protected_pmu) {

An offlist Sashiko review pointed out that this will prevent doing KVM_PMU_CAP_DISABLE on a TDX VM, which could prove problematic if userspace is already explicitly disabling vPMU support for TDX. I was thinking that wouldn't be a problem since KVM already disallows KVM_PMU_CAP_DISABLE if enable_pmu=false, but in that case KVM doesn't advertise KVM_CAP_PMU_VALID_MASK:

	r = enable_pmu ? KVM_CAP_PMU_VALID_MASK : 0;

And unless I'm misreading QEMU code, this will indeed be a problem.

Argh. I was going to suggest something sightly fancier, but we rather stupidly didn't provide a flag for ENABLE, and so we can't feed has_protected_pmu into the valid mask. Lame.

The other super minor detail is that kvm->arch.has_protected_pmu isn't guarded by kvm->lock, and I really hope we never get to that point (e.g. for SNP).

So instead of disallowing KVM_CAP_PMU_CAPABILITY entirely, this?

x86/kvm/x86.c +7 -0
index dc69b8cebe0b..b833a2596430 100644
@@ -6911,6 +6911,13 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
                if (!enable_pmu || (cap->args[0] & ~KVM_CAP_PMU_VALID_MASK))
                        break;
 
+               if (kvm->arch.has_protected_pmu) {
+                       WARN_ON_ONCE(kvm->arch.enable_pmu);
+                       if (cap->args[0] == KVM_PMU_CAP_DISABLE)
+                               r = 0;
+                       break;
+               }
+
                mutex_lock(&kvm->lock);
                if (!kvm->created_vcpus && !kvm->arch.created_mediated_pmu) {
                        kvm->arch.enable_pmu = !(cap->args[0] & KVM_PMU_CAP_DISABLE);
Quoted 18 lines
arch/x86/kvm/x86.c +4 -1
 			kvm->arch.enable_pmu = !(cap->args[0] & KVM_PMU_CAP_DISABLE);
 			r = 0;
 		}
@@ -13375,7 +13376,10 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 	kvm->arch.default_tsc_khz = max_tsc_khz ? : tsc_khz;
 	kvm->arch.apic_bus_cycle_ns = APIC_BUS_CYCLE_NS_DEFAULT;
 	kvm->arch.guest_can_read_msr_platform_info = true;
-	kvm->arch.enable_pmu = enable_pmu;
+	if (kvm->arch.has_protected_pmu)
+		kvm->arch.enable_pmu = false;
+	else
+		kvm->arch.enable_pmu = enable_pmu;
 
 #if IS_ENABLED(CONFIG_HYPERV)
 	spin_lock_init(&kvm->arch.hv_root_tdp_lock);
Signature · 1 line
2.54.0.563.g4f69b47b94-goog
#3 lore raw
Quoted 42 lines · Sean Christopherson

arch/x86/kvm/x86.c int kvm_vm_ioctl_enable_cap(struct kvm *kvm, · line 6914

Quoted 11 lines · Vishal Annapurve
arch/x86/kvm/x86.c +2 -1
index 0a1b63c63d1a..99a383455d46 100644
@@ -6910,7 +6910,8 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
                      break;
 
              mutex_lock(&kvm->lock);
-             if (!kvm->created_vcpus && !kvm->arch.created_mediated_pmu) {
+             if (!kvm->created_vcpus && !kvm->arch.created_mediated_pmu &&
+                 !kvm->arch.has_protected_pmu) {

An offlist Sashiko review pointed out that this will prevent doing KVM_PMU_CAP_DISABLE on a TDX VM, which could prove problematic if userspace is already explicitly disabling vPMU support for TDX. I was thinking that wouldn't be a problem since KVM already disallows KVM_PMU_CAP_DISABLE if enable_pmu=false, but in that case KVM doesn't advertise KVM_CAP_PMU_VALID_MASK:

        r = enable_pmu ? KVM_CAP_PMU_VALID_MASK : 0;

And unless I'm misreading QEMU code, this will indeed be a problem.

Argh. I was going to suggest something sightly fancier, but we rather stupidly didn't provide a flag for ENABLE, and so we can't feed has_protected_pmu into the valid mask. Lame.

The other super minor detail is that kvm->arch.has_protected_pmu isn't guarded by kvm->lock, and I really hope we never get to that point (e.g. for SNP).

So instead of disallowing KVM_CAP_PMU_CAPABILITY entirely, this?

x86/kvm/x86.c +0 -0
index dc69b8cebe0b..b833a2596430 100644
@@ -6911,6 +6911,13 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
                if (!enable_pmu || (cap->args[0] & ~KVM_CAP_PMU_VALID_MASK))
                        break;
 

x86/kvm/x86.c int kvm_vm_ioctl_enable_cap(struct kvm *kvm, · line 6923

Quoted 10 lines
x86/kvm/x86.c +7 -0
+               if (kvm->arch.has_protected_pmu) {
+                       WARN_ON_ONCE(kvm->arch.enable_pmu);
+                       if (cap->args[0] == KVM_PMU_CAP_DISABLE)
+                               r = 0;
+                       break;
+               }
+
                mutex_lock(&kvm->lock);
                if (!kvm->created_vcpus && !kvm->arch.created_mediated_pmu) {
                        kvm->arch.enable_pmu = !(cap->args[0] & KVM_PMU_CAP_DISABLE);

Incorporated this in v4 [1].

[1] https://lore.kernel.org/kvm/[email protected]/.

Quoted 19 lines
Quoted 19 lines
arch/x86/kvm/x86.c +4 -2
                      kvm->arch.enable_pmu = !(cap->args[0] & KVM_PMU_CAP_DISABLE);
                      r = 0;
              }
@@ -13375,7 +13376,10 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
      kvm->arch.default_tsc_khz = max_tsc_khz ? : tsc_khz;
      kvm->arch.apic_bus_cycle_ns = APIC_BUS_CYCLE_NS_DEFAULT;
      kvm->arch.guest_can_read_msr_platform_info = true;
-     kvm->arch.enable_pmu = enable_pmu;
+     if (kvm->arch.has_protected_pmu)
+             kvm->arch.enable_pmu = false;
+     else
+             kvm->arch.enable_pmu = enable_pmu;
 
 #if IS_ENABLED(CONFIG_HYPERV)
      spin_lock_init(&kvm->arch.hv_root_tdp_lock);
--

2.54.0.563.g4f69b47b94-goog