KVM: VMX: Mark Intel PT virtualization as BROKEN
Sean Christopherson
maintainercover KVM: VMX: Mark Intel PT virtualization as BROKEN
Hide Intel PT virtualization behind BROKEN, as it has multiple fatal flaws, several which put the host at risk, and several of which are far too invasive to backport to stable trees.
I included one of the easier fixes from Adrian to help show just how broken PT virtualization is, e.g. to illustrate the apparent lack of usage.
Adrian Hunter (1):
KVM: VMX: Allow toggling bits in MSR_IA32_RTIT_CTL when enable bit is
cleared
Sean Christopherson (1):
KVM: VMX: Bury Intel PT virtualization (guest/host mode) behind
CONFIG_BROKEN
arch/x86/kvm/vmx/vmx.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
base-commit: e466901b947d529f7b091a3b00b19d2bdee206ee
Signature · 1 line
2.47.0.163.g1226f6d8fa-goog
Sean Christopherson
maintainerPATCH 1/2 KVM: VMX: Bury Intel PT virtualization (guest/host mode) behind CONFIG_BROKEN
Hide KVM's pt_mode module param behind CONFIG_BROKEN, i.e. disable support for virtualizing Intel PT via guest/host mode unless BROKEN=y. There are myriad bugs in the implementation, some of which are fatal to the guest, and others which put the stability and health of the host at risk.
For guest fatalities, the most glaring issue is that KVM fails to ensure tracing is disabled, and *stays* disabled prior to VM-Enter, which is necessary as hardware disallows loading (the guest's) RTIT_CTL if tracing is enabled (enforced via a VMX consistency check). Per the SDM:
If the logical processor is operating with Intel PT enabled (if IA32_RTIT_CTL.TraceEn = 1) at the time of VM entry, the "load IA32_RTIT_CTL" VM-entry control must be 0.
On the host side, KVM doesn't validate the guest CPUID configuration provided by userspace, and even worse, uses the guest configuration to decide what MSRs to save/load at VM-Enter and VM-Exit. E.g. configuring guest CPUID to enumerate more address ranges than are supported in hardware will result in KVM trying to passthrough, save, and load non-existent MSRs, which generates a variety of WARNs, ToPA ERRORs in the host, a potential deadlock, etc.
--- arch/x86/kvm/vmx/vmx.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/vmx.c +3 -1
@@ -217,9 +217,11 @@ module_param(ple_window_shrink, uint, 0444);static unsigned int ple_window_max =KVM_VMX_DEFAULT_PLE_WINDOW_MAX ;module_param (ple_window_max, uint,0444 ); -/* Default is SYSTEM mode, 1 for host-guest mode */ +/* Default is SYSTEM mode, 1 for host-guest mode (which is BROKEN) */ int __read_mostly pt_mode =PT_MODE_SYSTEM ; +#ifdef CONFIG_BROKEN module_param (pt_mode, int,S_IRUGO ); +#endif struct x86_pmu_lbr __ro_after_init vmx_lbr_caps;
Signature · 1 line
2.47.0.163.g1226f6d8fa-goog
Re: [PATCH 1/2] KVM: VMX: Bury Intel PT virtualization (guest/host mode) behind CONFIG_BROKEN
Quoted 35 lines · Sean Christopherson
Hide KVM's
pt_modemodule param behindCONFIG_BROKEN, i.e. disable support for virtualizing Intel PT via guest/host mode unless BROKEN=y. There are myriad bugs in the implementation, some of which are fatal to the guest, and others which put the stability and health of the host at risk.For guest fatalities, the most glaring issue is that KVM fails to ensure tracing is disabled, and *stays* disabled prior to VM-Enter, which is necessary as hardware disallows loading (the guest's)
RTIT_CTLif tracing is enabled (enforced via a VMX consistency check). Per the SDM:If the logical processor is operating with Intel PT enabled (if IA32_RTIT_CTL.TraceEn = 1) at the time of VM entry, the "load
IA32_RTIT_CTL" VM-entry control must be 0.On the host side, KVM doesn't validate the guest CPUID configuration provided by userspace, and even worse, uses the guest configuration to decide what MSRs to save/load at VM-Enter and VM-Exit. E.g. configuring guest CPUID to enumerate more address ranges than are supported in hardware will result in KVM trying to passthrough, save, and load non-existent MSRs, which generates a variety of WARNs, ToPA ERRORs in the host, a potential deadlock, etc.
Fixes f99e3daf94ff ("KVM: x86: Add Intel PT virtualization work mode")Cc Adrian Hunter <[email protected]>Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/vmx.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)arch/x86/kvm/vmx/vmx.c +0
-0
arch/x86/kvm/vmx/vmx.c module_param(ple_window_shrink, uint, 0444); · line 224
Quoted 9 lines
arch/x86/kvm/vmx/vmx.c +3
-1@@ -217,9 +217,11 @@ module_param(ple_window_shrink, uint, 0444);static unsigned int ple_window_max =KVM_VMX_DEFAULT_PLE_WINDOW_MAX ;module_param (ple_window_max, uint,0444 ); -/* Default is SYSTEM mode, 1 for host-guest mode */ +/* Default is SYSTEM mode, 1 for host-guest mode (which is BROKEN) */ int __read_mostly pt_mode =PT_MODE_SYSTEM ; +#ifdef CONFIG_BROKEN module_param (pt_mode, int,S_IRUGO ); +#endif
I like the patch, but I didn't find any other usercase of CONFIG_BROKEN in current Linux.
Quoted 2 lines
arch/x86/kvm/vmx/vmx.c +0
-0struct x86_pmu_lbr __ro_after_init vmx_lbr_caps;
Sean Christopherson
maintainerRe: [PATCH 1/2] KVM: VMX: Bury Intel PT virtualization (guest/host mode) behind CONFIG_BROKEN
Quoted 31 lines · Xiaoyao Li
Quoted 30 lines · Sean Christopherson
Hide KVM's
pt_modemodule param behindCONFIG_BROKEN, i.e. disable support for virtualizing Intel PT via guest/host mode unless BROKEN=y. There are myriad bugs in the implementation, some of which are fatal to the guest, and others which put the stability and health of the host at risk.For guest fatalities, the most glaring issue is that KVM fails to ensure tracing is disabled, and *stays* disabled prior to VM-Enter, which is necessary as hardware disallows loading (the guest's)
RTIT_CTLif tracing is enabled (enforced via a VMX consistency check). Per the SDM:If the logical processor is operating with Intel PT enabled (if IA32_RTIT_CTL.TraceEn = 1) at the time of VM entry, the "load
IA32_RTIT_CTL" VM-entry control must be 0.On the host side, KVM doesn't validate the guest CPUID configuration provided by userspace, and even worse, uses the guest configuration to decide what MSRs to save/load at VM-Enter and VM-Exit. E.g. configuring guest CPUID to enumerate more address ranges than are supported in hardware will result in KVM trying to passthrough, save, and load non-existent MSRs, which generates a variety of WARNs, ToPA ERRORs in the host, a potential deadlock, etc.
Fixes f99e3daf94ff ("KVM: x86: Add Intel PT virtualization work mode")Cc Adrian Hunter <[email protected]>Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/vmx.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
Quoted 16 lines
arch/x86/kvm/vmx/vmx.c module_param(ple_window_shrink, uint, 0444); · line 223
Quoted 13 lines
arch/x86/kvm/vmx/vmx.c +3
-1@@ -217,9 +217,11 @@ module_param(ple_window_shrink, uint, 0444);static unsigned int ple_window_max =KVM_VMX_DEFAULT_PLE_WINDOW_MAX ;module_param (ple_window_max, uint,0444 ); -/* Default is SYSTEM mode, 1 for host-guest mode */ +/* Default is SYSTEM mode, 1 for host-guest mode (which is BROKEN) */ int __read_mostly pt_mode =PT_MODE_SYSTEM ; +#ifdef CONFIG_BROKEN module_param (pt_mode, int,S_IRUGO ); +#endif I like the patch, but I didn't find any other usercase of
CONFIG_BROKENin current Linux.
Ya, BROKEN is typically used directly in Kconfigs, e.g. "depends on BROKEN". But I can't think of any reason using it in this way would be problematic.
Xiaoyao Li
Reviewed-byRe: [PATCH 1/2] KVM: VMX: Bury Intel PT virtualization (guest/host mode) behind CONFIG_BROKEN
Quoted 49 lines · Sean Christopherson
Quoted 31 lines · Xiaoyao Li
Quoted 30 lines · Sean Christopherson
Hide KVM's
pt_modemodule param behindCONFIG_BROKEN, i.e. disable support for virtualizing Intel PT via guest/host mode unless BROKEN=y. There are myriad bugs in the implementation, some of which are fatal to the guest, and others which put the stability and health of the host at risk.For guest fatalities, the most glaring issue is that KVM fails to ensure tracing is disabled, and *stays* disabled prior to VM-Enter, which is necessary as hardware disallows loading (the guest's)
RTIT_CTLif tracing is enabled (enforced via a VMX consistency check). Per the SDM:If the logical processor is operating with Intel PT enabled (if IA32_RTIT_CTL.TraceEn = 1) at the time of VM entry, the "load
IA32_RTIT_CTL" VM-entry control must be 0.On the host side, KVM doesn't validate the guest CPUID configuration provided by userspace, and even worse, uses the guest configuration to decide what MSRs to save/load at VM-Enter and VM-Exit. E.g. configuring guest CPUID to enumerate more address ranges than are supported in hardware will result in KVM trying to passthrough, save, and load non-existent MSRs, which generates a variety of WARNs, ToPA ERRORs in the host, a potential deadlock, etc.
Fixes f99e3daf94ff ("KVM: x86: Add Intel PT virtualization work mode")Cc Adrian Hunter <[email protected]>Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/vmx.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)Quoted 16 lines
arch/x86/kvm/vmx/vmx.c module_param(ple_window_shrink, uint, 0444); · line 223
Quoted 13 lines
arch/x86/kvm/vmx/vmx.c +3
-1@@ -217,9 +217,11 @@ module_param(ple_window_shrink, uint, 0444);static unsigned int ple_window_max =KVM_VMX_DEFAULT_PLE_WINDOW_MAX ;module_param (ple_window_max, uint,0444 ); -/* Default is SYSTEM mode, 1 for host-guest mode */ +/* Default is SYSTEM mode, 1 for host-guest mode (which is BROKEN) */ int __read_mostly pt_mode =PT_MODE_SYSTEM ; +#ifdef CONFIG_BROKEN module_param (pt_mode, int,S_IRUGO ); +#endif I like the patch, but I didn't find any other usercase of
CONFIG_BROKENin current Linux.
Quoted 2 lines
Ya, BROKEN is typically used directly in Kconfigs, e.g. "depends on BROKEN". But I can't think of any reason using it in this way would be problematic.
I see. Thanks for the information!
Re: [PATCH 1/2] KVM: VMX: Bury Intel PT virtualization (guest/host mode) behind CONFIG_BROKEN
In reply to Sean ChristophersonQuoted 35 lines · Sean Christopherson
Hide KVM's
pt_modemodule param behindCONFIG_BROKEN, i.e. disable support for virtualizing Intel PT via guest/host mode unless BROKEN=y. There are myriad bugs in the implementation, some of which are fatal to the guest, and others which put the stability and health of the host at risk.For guest fatalities, the most glaring issue is that KVM fails to ensure tracing is disabled, and *stays* disabled prior to VM-Enter, which is necessary as hardware disallows loading (the guest's)
RTIT_CTLif tracing is enabled (enforced via a VMX consistency check). Per the SDM:If the logical processor is operating with Intel PT enabled (if IA32_RTIT_CTL.TraceEn = 1) at the time of VM entry, the "load
IA32_RTIT_CTL" VM-entry control must be 0.On the host side, KVM doesn't validate the guest CPUID configuration provided by userspace, and even worse, uses the guest configuration to decide what MSRs to save/load at VM-Enter and VM-Exit. E.g. configuring guest CPUID to enumerate more address ranges than are supported in hardware will result in KVM trying to passthrough, save, and load non-existent MSRs, which generates a variety of WARNs, ToPA ERRORs in the host, a potential deadlock, etc.
Fixes f99e3daf94ff ("KVM: x86: Add Intel PT virtualization work mode")Cc Adrian Hunter <[email protected]>Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/vmx.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)arch/x86/kvm/vmx/vmx.c +0
-0
arch/x86/kvm/vmx/vmx.c module_param(ple_window_shrink, uint, 0444); · line 224
Quoted 9 lines
arch/x86/kvm/vmx/vmx.c +3
-1@@ -217,9 +217,11 @@ module_param(ple_window_shrink, uint, 0444);static unsigned int ple_window_max =KVM_VMX_DEFAULT_PLE_WINDOW_MAX ;module_param (ple_window_max, uint,0444 ); -/* Default is SYSTEM mode, 1 for host-guest mode */ +/* Default is SYSTEM mode, 1 for host-guest mode (which is BROKEN) */ int __read_mostly pt_mode =PT_MODE_SYSTEM ; +#ifdef CONFIG_BROKEN module_param (pt_mode, int,S_IRUGO ); +#endif
Side effects are:
1. If pt_mode is passed via modprobe, there will be a warning in kernel messages:
kvm_intel: unknown parameter 'pt_mode' ignored
2. The sysfs module parameter file pt_mode will be gone:
# cat /sys/module/kvm_intel/parameters/pt_mode cat: /sys/module/kvm_intel/parameters/pt_mode: No such file or directory
Nevertheless:
Quoted 3 lines
arch/x86/kvm/vmx/vmx.c +0
-0struct x86_pmu_lbr __ro_after_init vmx_lbr_caps;
Sean Christopherson
maintainerRe: [PATCH 1/2] KVM: VMX: Bury Intel PT virtualization (guest/host mode) behind CONFIG_BROKEN
Quoted 39 lines · Adrian Hunter
Quoted 38 lines · Sean Christopherson
Hide KVM's
pt_modemodule param behindCONFIG_BROKEN, i.e. disable support for virtualizing Intel PT via guest/host mode unless BROKEN=y. There are myriad bugs in the implementation, some of which are fatal to the guest, and others which put the stability and health of the host at risk.For guest fatalities, the most glaring issue is that KVM fails to ensure tracing is disabled, and *stays* disabled prior to VM-Enter, which is necessary as hardware disallows loading (the guest's)
RTIT_CTLif tracing is enabled (enforced via a VMX consistency check). Per the SDM:If the logical processor is operating with Intel PT enabled (if IA32_RTIT_CTL.TraceEn = 1) at the time of VM entry, the "load
IA32_RTIT_CTL" VM-entry control must be 0.On the host side, KVM doesn't validate the guest CPUID configuration provided by userspace, and even worse, uses the guest configuration to decide what MSRs to save/load at VM-Enter and VM-Exit. E.g. configuring guest CPUID to enumerate more address ranges than are supported in hardware will result in KVM trying to passthrough, save, and load non-existent MSRs, which generates a variety of WARNs, ToPA ERRORs in the host, a potential deadlock, etc.
Fixes f99e3daf94ff ("KVM: x86: Add Intel PT virtualization work mode")Cc Adrian Hunter <[email protected]>Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/vmx.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)arch/x86/kvm/vmx/vmx.c +0
-0@@ -217,9 +217,11 @@ module_param(ple_window_shrink, uint, 0444);static unsigned int ple_window_max =KVM_VMX_DEFAULT_PLE_WINDOW_MAX ;module_param (ple_window_max, uint,0444 );
Quoted 10 lines
arch/x86/kvm/vmx/vmx.c module_param(ple_window_shrink, uint, 0444); · line 224
Quoted 6 lines
arch/x86/kvm/vmx/vmx.c +3
-1-/* Default is SYSTEM mode, 1 for host-guest mode */ +/* Default is SYSTEM mode, 1 for host-guest mode (which is BROKEN) */ int __read_mostly pt_mode =PT_MODE_SYSTEM ; +#ifdef CONFIG_BROKEN module_param (pt_mode, int,S_IRUGO ); +#endif Side effects are:
1. Ifpt_modeis passed via modprobe, there will be a warning in kernel messages:
kvm_intel: unknown parameter 'pt_mode' ignored
This is more of a feature in this case, as it's a non-fatal way of alerting the user that trying to enable PT virtualization won't work.
Quoted 3 lines
2. The sysfs module parameter file
pt_modewill be gone:# cat /sys/module/kvm_intel/parameters/pt_mode cat: /sys/module/kvm_intel/parameters/pt_mode: No such file or directory
Hrm, this could be slightly more problematic, e.g. if userspace were asserting on the state of the parameter. But AFAIK, module params aren't considered ABI.
Paolo, any thoughts on how best to handle this?
Re: [PATCH 1/2] KVM: VMX: Bury Intel PT virtualization (guest/host mode) behind CONFIG_BROKEN
In reply to Sean ChristophersonQuoted 37 lines · Sean Christopherson
Hide KVM's
pt_modemodule param behindCONFIG_BROKEN, i.e. disable support for virtualizing Intel PT via guest/host mode unless BROKEN=y. There are myriad bugs in the implementation, some of which are fatal to the guest, and others which put the stability and health of the host at risk.For guest fatalities, the most glaring issue is that KVM fails to ensure tracing is disabled, and *stays* disabled prior to VM-Enter, which is necessary as hardware disallows loading (the guest's)
RTIT_CTLif tracing is enabled (enforced via a VMX consistency check). Per the SDM:If the logical processor is operating with Intel PT enabled (if IA32_RTIT_CTL.TraceEn = 1) at the time of VM entry, the "load
IA32_RTIT_CTL" VM-entry control must be 0.On the host side, KVM doesn't validate the guest CPUID configuration provided by userspace, and even worse, uses the guest configuration to decide what MSRs to save/load at VM-Enter and VM-Exit. E.g. configuring guest CPUID to enumerate more address ranges than are supported in hardware will result in KVM trying to passthrough, save, and load non-existent MSRs, which generates a variety of WARNs, ToPA ERRORs in the host, a potential deadlock, etc.
Fixes f99e3daf94ff ("KVM: x86: Add Intel PT virtualization work mode")Cc Adrian Hunter <[email protected]>Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/vmx.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)arch/x86/kvm/vmx/vmx.c +0
-0@@ -217,9 +217,11 @@ module_param(ple_window_shrink, uint, 0444);static unsigned int ple_window_max =KVM_VMX_DEFAULT_PLE_WINDOW_MAX ;module_param (ple_window_max, uint,0444 );
arch/x86/kvm/vmx/vmx.c module_param(ple_window_shrink, uint, 0444); · line 227
Quoted 10 lines
arch/x86/kvm/vmx/vmx.c +3
-1-/* Default is SYSTEM mode, 1 for host-guest mode */ +/* Default is SYSTEM mode, 1 for host-guest mode (which is BROKEN) */ int __read_mostly pt_mode =PT_MODE_SYSTEM ; +#ifdef CONFIG_BROKEN module_param (pt_mode, int,S_IRUGO ); +#endif struct x86_pmu_lbr __ro_after_init vmx_lbr_caps;
Applied, thanks.
Paolo
Sean Christopherson
maintainerPATCH 2/2 KVM: VMX: Allow toggling bits in MSR_IA32_RTIT_CTL when enable bit is cleared
In reply to Sean ChristophersonFrom: Adrian Hunter <[email protected]>
Allow toggling other bits in MSR_IA32_RTIT_CTL if the enable bit is being cleared, the existing logic simply ignores the enable bit. E.g. KVM will incorrectly reject a write of '0' to stop tracing.
--- arch/x86/kvm/vmx/vmx.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/vmx.c +2 -1
@@ -1636,7 +1636,8 @@ static int vmx_rtit_ctl_check(struct kvm_vcpu *vcpu, u64 data)* result in a #GP unless the same write also clears TraceEn. */ if ((vmx->pt_desc.guest.ctl &RTIT_CTL_TRACEEN ) && - ((vmx->pt_desc.guest.ctl ^ data) & ~RTIT_CTL_TRACEEN )) + (data &RTIT_CTL_TRACEEN ) && + data != vmx->pt_desc.guest.ctl)return 1 ;/*
Signature · 1 line
2.47.0.163.g1226f6d8fa-goog
Re: [PATCH 2/2] KVM: VMX: Allow toggling bits in MSR_IA32_RTIT_CTL when enable bit is cleared
Quoted 5 lines · Sean Christopherson
From: Adrian Hunter <[email protected]>
Allow toggling other bits in
MSR_IA32_RTIT_CTLif the enable bit is being cleared, the existing logic simply ignores the enable bit. E.g. KVM will incorrectly reject a write of '0' to stop tracing.
Quoted 22 lines
Fixes bf8c55d8dc09 ("KVM: x86: Implement Intel PT MSRs read/write emulation")Signed-off-by Adrian Hunter <[email protected]>[sean: rework changelog, drop stable@]Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/vmx.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)arch/x86/kvm/vmx/vmx.c +2
-1@@ -1636,7 +1636,8 @@ static int vmx_rtit_ctl_check(struct kvm_vcpu *vcpu, u64 data)* result in a #GP unless the same write also clears TraceEn. */ if ((vmx->pt_desc.guest.ctl &RTIT_CTL_TRACEEN ) && - ((vmx->pt_desc.guest.ctl ^ data) & ~RTIT_CTL_TRACEEN )) + (data &RTIT_CTL_TRACEEN ) && + data != vmx->pt_desc.guest.ctl)return 1 ;/*
Quoted 8 lines · Sean Christopherson
Hide Intel PT virtualization behind BROKEN, as it has multiple fatal flaws, several which put the host at risk, and several of which are far too invasive to backport to stable trees.
I included one of the easier fixes from Adrian to help show just how broken PT virtualization is, e.g. to illustrate the apparent lack of usage.
[...]
Applied patch 2 to kvm-x86 vmx (patch 1 already went in).
[1/2] KVM: VMX: Bury Intel PT virtualization (guest/host mode) behind CONFIG_BROKEN
(no commit info)
[2/2] KVM: VMX: Allow toggling bits in MSR_IA32_RTIT_CTL when enable bit is cleared
https://github.com/kvm-x86/linux/commit/591ff4670c7b