Skip to contentSkip to conversation

KVM: VMX: Mark Intel PT virtualization as BROKEN

PATCH Applied Sean Christopherson 4 participants 11 messages Nov 1, 2024 – Dec 19, 2024

Subsystems (3)
lore
Share & tools
lore

Sean Christopherson

author maintainer
#0 lore raw

cover KVM: VMX: Mark Intel PT virtualization as BROKEN

Hide Intel PT virtualization behind BROKEN, as it has multiple fatal flaws, several which put the host at risk, and several of which are far too invasive to backport to stable trees.

I included one of the easier fixes from Adrian to help show just how broken PT virtualization is, e.g. to illustrate the apparent lack of usage.

Adrian Hunter (1):
KVM: VMX: Allow toggling bits in MSR_IA32_RTIT_CTL when enable bit is
cleared

Sean Christopherson (1):
KVM: VMX: Bury Intel PT virtualization (guest/host mode) behind
CONFIG_BROKEN

 arch/x86/kvm/vmx/vmx.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

base-commit: e466901b947d529f7b091a3b00b19d2bdee206ee

Signature · 1 line
2.47.0.163.g1226f6d8fa-goog
author maintainer
#1 lore raw

PATCH 1/2 KVM: VMX: Bury Intel PT virtualization (guest/host mode) behind CONFIG_BROKEN

Hide KVM's pt_mode module param behind CONFIG_BROKEN, i.e. disable support for virtualizing Intel PT via guest/host mode unless BROKEN=y. There are myriad bugs in the implementation, some of which are fatal to the guest, and others which put the stability and health of the host at risk.

For guest fatalities, the most glaring issue is that KVM fails to ensure tracing is disabled, and *stays* disabled prior to VM-Enter, which is necessary as hardware disallows loading (the guest's) RTIT_CTL if tracing is enabled (enforced via a VMX consistency check). Per the SDM:

If the logical processor is operating with Intel PT enabled (if IA32_RTIT_CTL.TraceEn = 1) at the time of VM entry, the "load IA32_RTIT_CTL" VM-entry control must be 0.

On the host side, KVM doesn't validate the guest CPUID configuration provided by userspace, and even worse, uses the guest configuration to decide what MSRs to save/load at VM-Enter and VM-Exit. E.g. configuring guest CPUID to enumerate more address ranges than are supported in hardware will result in KVM trying to passthrough, save, and load non-existent MSRs, which generates a variety of WARNs, ToPA ERRORs in the host, a potential deadlock, etc.

Fixes f99e3daf94ff ("KVM: x86: Add Intel PT virtualization work mode")
Cc Adrian Hunter <[email protected]>
---
 arch/x86/kvm/vmx/vmx.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/vmx.c +3 -1
index 6ed801ffe33f..087504fb1589 100644
@@ -217,9 +217,11 @@ module_param(ple_window_shrink, uint, 0444);
 static unsigned int ple_window_max        = KVM_VMX_DEFAULT_PLE_WINDOW_MAX;
 module_param(ple_window_max, uint, 0444);
 
-/* Default is SYSTEM mode, 1 for host-guest mode */
+/* Default is SYSTEM mode, 1 for host-guest mode (which is BROKEN) */
 int __read_mostly pt_mode = PT_MODE_SYSTEM;
+#ifdef CONFIG_BROKEN
 module_param(pt_mode, int, S_IRUGO);
+#endif
 
 struct x86_pmu_lbr __ro_after_init vmx_lbr_caps;
 
Signature · 1 line
2.47.0.163.g1226f6d8fa-goog
#2 lore raw

Re: [PATCH 1/2] KVM: VMX: Bury Intel PT virtualization (guest/host mode) behind CONFIG_BROKEN

Quoted 35 lines · Sean Christopherson

Hide KVM's pt_mode module param behind CONFIG_BROKEN, i.e. disable support for virtualizing Intel PT via guest/host mode unless BROKEN=y. There are myriad bugs in the implementation, some of which are fatal to the guest, and others which put the stability and health of the host at risk.

For guest fatalities, the most glaring issue is that KVM fails to ensure tracing is disabled, and *stays* disabled prior to VM-Enter, which is necessary as hardware disallows loading (the guest's) RTIT_CTL if tracing is enabled (enforced via a VMX consistency check). Per the SDM:

If the logical processor is operating with Intel PT enabled (if IA32_RTIT_CTL.TraceEn = 1) at the time of VM entry, the "load IA32_RTIT_CTL" VM-entry control must be 0.

On the host side, KVM doesn't validate the guest CPUID configuration provided by userspace, and even worse, uses the guest configuration to decide what MSRs to save/load at VM-Enter and VM-Exit. E.g. configuring guest CPUID to enumerate more address ranges than are supported in hardware will result in KVM trying to passthrough, save, and load non-existent MSRs, which generates a variety of WARNs, ToPA ERRORs in the host, a potential deadlock, etc.

Fixes f99e3daf94ff ("KVM: x86: Add Intel PT virtualization work mode")
Cc Adrian Hunter <[email protected]>
---
  arch/x86/kvm/vmx/vmx.c | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/vmx.c +0 -0
index 6ed801ffe33f..087504fb1589 100644

arch/x86/kvm/vmx/vmx.c module_param(ple_window_shrink, uint, 0444); · line 224

Quoted 9 lines
arch/x86/kvm/vmx/vmx.c +3 -1
@@ -217,9 +217,11 @@ module_param(ple_window_shrink, uint, 0444);
  static unsigned int ple_window_max        = KVM_VMX_DEFAULT_PLE_WINDOW_MAX;
  module_param(ple_window_max, uint, 0444);
 
-/* Default is SYSTEM mode, 1 for host-guest mode */
+/* Default is SYSTEM mode, 1 for host-guest mode (which is BROKEN) */
  int __read_mostly pt_mode = PT_MODE_SYSTEM;
+#ifdef CONFIG_BROKEN
  module_param(pt_mode, int, S_IRUGO);
+#endif

I like the patch, but I didn't find any other usercase of CONFIG_BROKEN in current Linux.

Quoted 2 lines
arch/x86/kvm/vmx/vmx.c +0 -0
  struct x86_pmu_lbr __ro_after_init vmx_lbr_caps;
 
author maintainer
#3 lore raw

Re: [PATCH 1/2] KVM: VMX: Bury Intel PT virtualization (guest/host mode) behind CONFIG_BROKEN

Quoted 31 lines · Xiaoyao Li
Quoted 30 lines · Sean Christopherson

Hide KVM's pt_mode module param behind CONFIG_BROKEN, i.e. disable support for virtualizing Intel PT via guest/host mode unless BROKEN=y. There are myriad bugs in the implementation, some of which are fatal to the guest, and others which put the stability and health of the host at risk.

For guest fatalities, the most glaring issue is that KVM fails to ensure tracing is disabled, and *stays* disabled prior to VM-Enter, which is necessary as hardware disallows loading (the guest's) RTIT_CTL if tracing is enabled (enforced via a VMX consistency check). Per the SDM:

If the logical processor is operating with Intel PT enabled (if IA32_RTIT_CTL.TraceEn = 1) at the time of VM entry, the "load IA32_RTIT_CTL" VM-entry control must be 0.

On the host side, KVM doesn't validate the guest CPUID configuration provided by userspace, and even worse, uses the guest configuration to decide what MSRs to save/load at VM-Enter and VM-Exit. E.g. configuring guest CPUID to enumerate more address ranges than are supported in hardware will result in KVM trying to passthrough, save, and load non-existent MSRs, which generates a variety of WARNs, ToPA ERRORs in the host, a potential deadlock, etc.

Fixes f99e3daf94ff ("KVM: x86: Add Intel PT virtualization work mode")
Cc Adrian Hunter <[email protected]>
---
  arch/x86/kvm/vmx/vmx.c | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)
Quoted 16 lines

arch/x86/kvm/vmx/vmx.c module_param(ple_window_shrink, uint, 0444); · line 223

Quoted 13 lines
arch/x86/kvm/vmx/vmx.c +3 -1
index 6ed801ffe33f..087504fb1589 100644
@@ -217,9 +217,11 @@ module_param(ple_window_shrink, uint, 0444);
  static unsigned int ple_window_max        = KVM_VMX_DEFAULT_PLE_WINDOW_MAX;
  module_param(ple_window_max, uint, 0444);
-/* Default is SYSTEM mode, 1 for host-guest mode */
+/* Default is SYSTEM mode, 1 for host-guest mode (which is BROKEN) */
  int __read_mostly pt_mode = PT_MODE_SYSTEM;
+#ifdef CONFIG_BROKEN
  module_param(pt_mode, int, S_IRUGO);
+#endif

I like the patch, but I didn't find any other usercase of CONFIG_BROKEN in current Linux.

Ya, BROKEN is typically used directly in Kconfigs, e.g. "depends on BROKEN". But I can't think of any reason using it in this way would be problematic.

Reviewed-by
#4 lore raw

Re: [PATCH 1/2] KVM: VMX: Bury Intel PT virtualization (guest/host mode) behind CONFIG_BROKEN

Quoted 49 lines · Sean Christopherson
Quoted 31 lines · Xiaoyao Li
Quoted 30 lines · Sean Christopherson

Hide KVM's pt_mode module param behind CONFIG_BROKEN, i.e. disable support for virtualizing Intel PT via guest/host mode unless BROKEN=y. There are myriad bugs in the implementation, some of which are fatal to the guest, and others which put the stability and health of the host at risk.

For guest fatalities, the most glaring issue is that KVM fails to ensure tracing is disabled, and *stays* disabled prior to VM-Enter, which is necessary as hardware disallows loading (the guest's) RTIT_CTL if tracing is enabled (enforced via a VMX consistency check). Per the SDM:

If the logical processor is operating with Intel PT enabled (if IA32_RTIT_CTL.TraceEn = 1) at the time of VM entry, the "load IA32_RTIT_CTL" VM-entry control must be 0.

On the host side, KVM doesn't validate the guest CPUID configuration provided by userspace, and even worse, uses the guest configuration to decide what MSRs to save/load at VM-Enter and VM-Exit. E.g. configuring guest CPUID to enumerate more address ranges than are supported in hardware will result in KVM trying to passthrough, save, and load non-existent MSRs, which generates a variety of WARNs, ToPA ERRORs in the host, a potential deadlock, etc.

Fixes f99e3daf94ff ("KVM: x86: Add Intel PT virtualization work mode")
Cc Adrian Hunter <[email protected]>
---
   arch/x86/kvm/vmx/vmx.c | 4 +++-
   1 file changed, 3 insertions(+), 1 deletion(-)
Quoted 16 lines

arch/x86/kvm/vmx/vmx.c module_param(ple_window_shrink, uint, 0444); · line 223

Quoted 13 lines
arch/x86/kvm/vmx/vmx.c +3 -1
index 6ed801ffe33f..087504fb1589 100644
@@ -217,9 +217,11 @@ module_param(ple_window_shrink, uint, 0444);
   static unsigned int ple_window_max        = KVM_VMX_DEFAULT_PLE_WINDOW_MAX;
   module_param(ple_window_max, uint, 0444);
-/* Default is SYSTEM mode, 1 for host-guest mode */
+/* Default is SYSTEM mode, 1 for host-guest mode (which is BROKEN) */
   int __read_mostly pt_mode = PT_MODE_SYSTEM;
+#ifdef CONFIG_BROKEN
   module_param(pt_mode, int, S_IRUGO);
+#endif

I like the patch, but I didn't find any other usercase of CONFIG_BROKEN in current Linux.

Quoted 2 lines

Ya, BROKEN is typically used directly in Kconfigs, e.g. "depends on BROKEN". But I can't think of any reason using it in this way would be problematic.

I see. Thanks for the information!

Tested-by on v1 1/2
#5 lore raw

Re: [PATCH 1/2] KVM: VMX: Bury Intel PT virtualization (guest/host mode) behind CONFIG_BROKEN

In reply to Sean Christopherson
Quoted 35 lines · Sean Christopherson

Hide KVM's pt_mode module param behind CONFIG_BROKEN, i.e. disable support for virtualizing Intel PT via guest/host mode unless BROKEN=y. There are myriad bugs in the implementation, some of which are fatal to the guest, and others which put the stability and health of the host at risk.

For guest fatalities, the most glaring issue is that KVM fails to ensure tracing is disabled, and *stays* disabled prior to VM-Enter, which is necessary as hardware disallows loading (the guest's) RTIT_CTL if tracing is enabled (enforced via a VMX consistency check). Per the SDM:

If the logical processor is operating with Intel PT enabled (if IA32_RTIT_CTL.TraceEn = 1) at the time of VM entry, the "load IA32_RTIT_CTL" VM-entry control must be 0.

On the host side, KVM doesn't validate the guest CPUID configuration provided by userspace, and even worse, uses the guest configuration to decide what MSRs to save/load at VM-Enter and VM-Exit. E.g. configuring guest CPUID to enumerate more address ranges than are supported in hardware will result in KVM trying to passthrough, save, and load non-existent MSRs, which generates a variety of WARNs, ToPA ERRORs in the host, a potential deadlock, etc.

Fixes f99e3daf94ff ("KVM: x86: Add Intel PT virtualization work mode")
Cc Adrian Hunter <[email protected]>
---
 arch/x86/kvm/vmx/vmx.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/vmx.c +0 -0
index 6ed801ffe33f..087504fb1589 100644

arch/x86/kvm/vmx/vmx.c module_param(ple_window_shrink, uint, 0444); · line 224

Quoted 9 lines
arch/x86/kvm/vmx/vmx.c +3 -1
@@ -217,9 +217,11 @@ module_param(ple_window_shrink, uint, 0444);
 static unsigned int ple_window_max        = KVM_VMX_DEFAULT_PLE_WINDOW_MAX;
 module_param(ple_window_max, uint, 0444);
 
-/* Default is SYSTEM mode, 1 for host-guest mode */
+/* Default is SYSTEM mode, 1 for host-guest mode (which is BROKEN) */
 int __read_mostly pt_mode = PT_MODE_SYSTEM;
+#ifdef CONFIG_BROKEN
 module_param(pt_mode, int, S_IRUGO);
+#endif

Side effects are:
1. If pt_mode is passed via modprobe, there will be a warning in kernel messages:
kvm_intel: unknown parameter 'pt_mode' ignored
2. The sysfs module parameter file pt_mode will be gone:

	# cat /sys/module/kvm_intel/parameters/pt_mode
	cat: /sys/module/kvm_intel/parameters/pt_mode: No such file or directory

Nevertheless:

Quoted 3 lines
arch/x86/kvm/vmx/vmx.c +0 -0
 struct x86_pmu_lbr __ro_after_init vmx_lbr_caps;
 
author maintainer
#6 lore raw

Re: [PATCH 1/2] KVM: VMX: Bury Intel PT virtualization (guest/host mode) behind CONFIG_BROKEN

Quoted 39 lines · Adrian Hunter
Quoted 38 lines · Sean Christopherson

Hide KVM's pt_mode module param behind CONFIG_BROKEN, i.e. disable support for virtualizing Intel PT via guest/host mode unless BROKEN=y. There are myriad bugs in the implementation, some of which are fatal to the guest, and others which put the stability and health of the host at risk.

For guest fatalities, the most glaring issue is that KVM fails to ensure tracing is disabled, and *stays* disabled prior to VM-Enter, which is necessary as hardware disallows loading (the guest's) RTIT_CTL if tracing is enabled (enforced via a VMX consistency check). Per the SDM:

If the logical processor is operating with Intel PT enabled (if IA32_RTIT_CTL.TraceEn = 1) at the time of VM entry, the "load IA32_RTIT_CTL" VM-entry control must be 0.

On the host side, KVM doesn't validate the guest CPUID configuration provided by userspace, and even worse, uses the guest configuration to decide what MSRs to save/load at VM-Enter and VM-Exit. E.g. configuring guest CPUID to enumerate more address ranges than are supported in hardware will result in KVM trying to passthrough, save, and load non-existent MSRs, which generates a variety of WARNs, ToPA ERRORs in the host, a potential deadlock, etc.

Fixes f99e3daf94ff ("KVM: x86: Add Intel PT virtualization work mode")
Cc Adrian Hunter <[email protected]>
---
 arch/x86/kvm/vmx/vmx.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/vmx.c +0 -0
index 6ed801ffe33f..087504fb1589 100644
@@ -217,9 +217,11 @@ module_param(ple_window_shrink, uint, 0444);
 static unsigned int ple_window_max        = KVM_VMX_DEFAULT_PLE_WINDOW_MAX;
 module_param(ple_window_max, uint, 0444);
 
Quoted 10 lines

arch/x86/kvm/vmx/vmx.c module_param(ple_window_shrink, uint, 0444); · line 224

Quoted 6 lines
arch/x86/kvm/vmx/vmx.c +3 -1
-/* Default is SYSTEM mode, 1 for host-guest mode */
+/* Default is SYSTEM mode, 1 for host-guest mode (which is BROKEN) */
 int __read_mostly pt_mode = PT_MODE_SYSTEM;
+#ifdef CONFIG_BROKEN
 module_param(pt_mode, int, S_IRUGO);
+#endif

Side effects are:
1. If pt_mode is passed via modprobe, there will be a warning in kernel messages:
kvm_intel: unknown parameter 'pt_mode' ignored

This is more of a feature in this case, as it's a non-fatal way of alerting the user that trying to enable PT virtualization won't work.

Quoted 3 lines

2. The sysfs module parameter file pt_mode will be gone:

	# cat /sys/module/kvm_intel/parameters/pt_mode
	cat: /sys/module/kvm_intel/parameters/pt_mode: No such file or directory

Hrm, this could be slightly more problematic, e.g. if userspace were asserting on the state of the parameter. But AFAIK, module params aren't considered ABI.

Paolo, any thoughts on how best to handle this?

maintainer Applied
#7 lore raw

Re: [PATCH 1/2] KVM: VMX: Bury Intel PT virtualization (guest/host mode) behind CONFIG_BROKEN

In reply to Sean Christopherson
Quoted 37 lines · Sean Christopherson

Hide KVM's pt_mode module param behind CONFIG_BROKEN, i.e. disable support for virtualizing Intel PT via guest/host mode unless BROKEN=y. There are myriad bugs in the implementation, some of which are fatal to the guest, and others which put the stability and health of the host at risk.

For guest fatalities, the most glaring issue is that KVM fails to ensure tracing is disabled, and *stays* disabled prior to VM-Enter, which is necessary as hardware disallows loading (the guest's) RTIT_CTL if tracing is enabled (enforced via a VMX consistency check). Per the SDM:

If the logical processor is operating with Intel PT enabled (if IA32_RTIT_CTL.TraceEn = 1) at the time of VM entry, the "load IA32_RTIT_CTL" VM-entry control must be 0.

On the host side, KVM doesn't validate the guest CPUID configuration provided by userspace, and even worse, uses the guest configuration to decide what MSRs to save/load at VM-Enter and VM-Exit. E.g. configuring guest CPUID to enumerate more address ranges than are supported in hardware will result in KVM trying to passthrough, save, and load non-existent MSRs, which generates a variety of WARNs, ToPA ERRORs in the host, a potential deadlock, etc.

Fixes f99e3daf94ff ("KVM: x86: Add Intel PT virtualization work mode")
Cc Adrian Hunter <[email protected]>
---
  arch/x86/kvm/vmx/vmx.c | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/vmx.c +0 -0
index 6ed801ffe33f..087504fb1589 100644
@@ -217,9 +217,11 @@ module_param(ple_window_shrink, uint, 0444);
  static unsigned int ple_window_max        = KVM_VMX_DEFAULT_PLE_WINDOW_MAX;
  module_param(ple_window_max, uint, 0444);

arch/x86/kvm/vmx/vmx.c module_param(ple_window_shrink, uint, 0444); · line 227

Quoted 10 lines
arch/x86/kvm/vmx/vmx.c +3 -1
 
-/* Default is SYSTEM mode, 1 for host-guest mode */
+/* Default is SYSTEM mode, 1 for host-guest mode (which is BROKEN) */
  int __read_mostly pt_mode = PT_MODE_SYSTEM;
+#ifdef CONFIG_BROKEN
  module_param(pt_mode, int, S_IRUGO);
+#endif
 
  struct x86_pmu_lbr __ro_after_init vmx_lbr_caps;
 

Applied, thanks.

Paolo

author maintainer
#8 lore raw

PATCH 2/2 KVM: VMX: Allow toggling bits in MSR_IA32_RTIT_CTL when enable bit is cleared

In reply to Sean Christopherson

From: Adrian Hunter <[email protected]>

Allow toggling other bits in MSR_IA32_RTIT_CTL if the enable bit is being cleared, the existing logic simply ignores the enable bit. E.g. KVM will incorrectly reject a write of '0' to stop tracing.

Fixes bf8c55d8dc09 ("KVM: x86: Implement Intel PT MSRs read/write emulation")
[sean: rework changelog, drop stable@]
---
 arch/x86/kvm/vmx/vmx.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/vmx.c +2 -1
index 087504fb1589..9b9d115c4824 100644
@@ -1636,7 +1636,8 @@ static int vmx_rtit_ctl_check(struct kvm_vcpu *vcpu, u64 data)
 	 * result in a #GP unless the same write also clears TraceEn.
 	 */
 	if ((vmx->pt_desc.guest.ctl & RTIT_CTL_TRACEEN) &&
-		((vmx->pt_desc.guest.ctl ^ data) & ~RTIT_CTL_TRACEEN))
+	    (data & RTIT_CTL_TRACEEN) &&
+	    data != vmx->pt_desc.guest.ctl)
 		return 1;
 
 	/*
Signature · 1 line
2.47.0.163.g1226f6d8fa-goog
Reviewed-by on v1 2/2
#9 lore raw

Re: [PATCH 2/2] KVM: VMX: Allow toggling bits in MSR_IA32_RTIT_CTL when enable bit is cleared

Quoted 5 lines · Sean Christopherson

From: Adrian Hunter <[email protected]>

Allow toggling other bits in MSR_IA32_RTIT_CTL if the enable bit is being cleared, the existing logic simply ignores the enable bit. E.g. KVM will incorrectly reject a write of '0' to stop tracing.

Quoted 22 lines
Fixes bf8c55d8dc09 ("KVM: x86: Implement Intel PT MSRs read/write emulation")
[sean: rework changelog, drop stable@]
---
  arch/x86/kvm/vmx/vmx.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/vmx.c +2 -1
index 087504fb1589..9b9d115c4824 100644
@@ -1636,7 +1636,8 @@ static int vmx_rtit_ctl_check(struct kvm_vcpu *vcpu, u64 data)
  	 * result in a #GP unless the same write also clears TraceEn.
  	 */
  	if ((vmx->pt_desc.guest.ctl & RTIT_CTL_TRACEEN) &&
-		((vmx->pt_desc.guest.ctl ^ data) & ~RTIT_CTL_TRACEEN))
+	    (data & RTIT_CTL_TRACEEN) &&
+	    data != vmx->pt_desc.guest.ctl)
  		return 1;
 
  	/*
author maintainer Applied
#10 lore raw
In reply to Sean Christopherson
Quoted 8 lines · Sean Christopherson

Hide Intel PT virtualization behind BROKEN, as it has multiple fatal flaws, several which put the host at risk, and several of which are far too invasive to backport to stable trees.

I included one of the easier fixes from Adrian to help show just how broken PT virtualization is, e.g. to illustrate the apparent lack of usage.

[...]

Applied patch 2 to kvm-x86 vmx (patch 1 already went in).

[1/2] KVM: VMX: Bury Intel PT virtualization (guest/host mode) behind CONFIG_BROKEN
(no commit info)
[2/2] KVM: VMX: Allow toggling bits in MSR_IA32_RTIT_CTL when enable bit is cleared

      https://github.com/kvm-x86/linux/commit/591ff4670c7b

--
https://github.com/kvm-x86/linux/tree/next