x86/cpu: KVM: Make SGX and VMX depend on FEAT_CTL
Subsystems (4)
Sean Christopherson
maintainercover x86/cpu: KVM: Make SGX and VMX depend on FEAT_CTL
"Officially" make SGX and VMX depend on X86_FEATURE_MSR_IA32_FEAT_CTL, and drop manual checks on X86_FEATURE_MSR_IA32_FEAT_CTL when querying VMX support.
To make dependencies on MSR_IA32_FEAT_CTL work as expected, process all CPUID dependencies at the end of CPU indentification. Because MSR_IA32_FEAT_CTL is a synthetic flag, it is effectively off-by-default, and thus may never be unset via clear_cpu_cap(), i.e. never triggers processing of its dependents.
The obvious alternative would be to explicitly clear MSR_IA32_FEAT_CTL if the MSR is unsupported, but that ends up being rather ugly as it would require clearing the flag in default_init() to handle the scenario where hardware supports the MSR, but the kernel was built without support for the CPU vendor. E.g. running on an Intel CPU with CPU_SUP_INTEL=n. This edge case is also why the existing manual checks in KVM are necessary; KVM_INTEL effectively depends on any of CPU_SUP_{INTEL,CENATUR,ZHAOXIN}.
Processing all dependencies also seems like the correct thing to do across the board, e.g. if the kernel ends up with more synthetic features with dependents.
The placement of the call to apply_cpuid_deps() isn't super scientific. I placed it after, AFAICT, the overwhelming majority of cpu cap updates had already been done, but before anything was likely to want the dependencies to be processed. Specifically, I couldn't find any set_cpu_caps() in the machine check code, but there are definitely cpu_has() calls under mcheck_cpu_init().
Last thought, patch 3 will conflict with at least one in-flight KVM series[*]. The conflict should be straightfoward to resolve, but at the same time this is far from urgent, i.e. kicking this series down the road until KVM settles down is totally ok.
[*] https://lore.kernel.org/all/[email protected]
Sean Christopherson (3):
x86/cpu: Process all CPUID dependencies after identifying CPU info x86/cpu: Mark SGX and VMX as being dependent on MSR_IA32_FEAT_CTL
KVM: VMX: Drop manual checks on X86_FEATURE_MSR_IA32_FEAT_CTL
arch/x86/include/asm/cpufeature.h | 1 + arch/x86/kernel/cpu/bugs.c | 3 +-- arch/x86/kernel/cpu/common.c | 6 ++++++ arch/x86/kernel/cpu/cpuid-deps.c | 12 ++++++++++++ arch/x86/kernel/cpu/feat_ctl.c | 3 +-- arch/x86/kvm/vmx/vmx.c | 6 ++---- 6 files changed, 23 insertions(+), 8 deletions(-)
base-commit: d800169041c0e035160c8b81f30d4b7e8f8ef777
Signature · 1 line
2.39.0.rc0.267.gcb52ba06e7-goog
Sean Christopherson
maintainerPATCH 1/3 x86/cpu: Process all CPUID dependencies after identifying CPU info
Process all CPUID dependencies to ensure that a dependent is disabled if one or more of its parent features is unsupported. As is, cpuid_deps is processed if an only if a feature is explicitly disabled via clear_cpu_cap(), which makes it annoying/dangerous to use cpuid_deps for features whose parent(s) do not always have explicit processing.
E.g. VMX and SGX depend on the synthetic X86_FEATURE_MSR_IA32_FEAT_CTL, but there is no common location to clear MSR_IA32_FEAT_CTL, and so consumers of VMX and SGX are forced to check MSR_IA32_FEAT_CTL on top of the dependent feature.
Manually clearing X86_FEATURE_MSR_IA32_FEAT_CTL is the obvious alternative, but it's subtly more difficult that updating init_ia32_feat_ctl(). CONFIG_IA32_FEAT_CTL depends on any of CONFIG_CPU_SUP_{INTEL,CENATUR,ZHAOXIN}, but init_ia32_feat_ctl() is invoked if and only if the actual CPU type matches one of the aforementioned CPU_SUP_* types. E.g. running a kernel built with
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
# CONFIG_CPU_SUP_HYGON is not set
# CONFIG_CPU_SUP_CENTAUR is not set
# CONFIG_CPU_SUP_ZHAOXIN is not set
on a Cenatur or Zhaoxin CPU will leave X86_FEATURE_VMX set but not set X86_FEATURE_MSR_IA32_FEAT_CTL, and will never call init_ia32_feat_ctl() to give the kernel a convenient opportunity to clear X86_FEATURE_MSR_IA32_FEAT_CTL.
--- arch/x86/include/asm/cpufeature.h | 1 + arch/x86/kernel/cpu/common.c | 6 ++++++ arch/x86/kernel/cpu/cpuid-deps.c | 10 ++++++++++ 3 files changed, 17 insertions(+)
arch/x86/include/asm/cpufeature.h +1 -0
@@ -147,6 +147,7 @@ extern const char * const x86_bug_flags[NBUGINTS*32];extern void setup_clear_cpu_cap (unsigned int bit);extern void clear_cpu_cap (struct cpuinfo_x86 *c,unsigned int bit); +extern void apply_cpuid_deps (struct cpuinfo_x86 *c);#define setup_force_cpu_cap (bit) do { \ set_cpu_cap(&boot_cpu_data, bit); \
arch/x86/kernel/cpu/common.c +6 -0
@@ -1887,6 +1887,12 @@ static void identify_cpu(struct cpuinfo_x86 *c)ppin_init (c); +/* +* Apply CPUID dependencies to ensure dependent features are disabled +* if a parent feature is unsupported but wasn't explicitly disabled. +*/ +apply_cpuid_deps (c); +/* Init Machine Check Exception if available. */ mcheck_cpu_init (c);
arch/x86/kernel/cpu/cpuid-deps.c +10 -0
@@ -138,3 +138,13 @@ void setup_clear_cpu_cap(unsigned int feature) {do_clear_cpu_cap (NULL , feature); } + +void apply_cpuid_deps (struct cpuinfo_x86 *c) +{ +const struct cpuid_dep *d; + +for (d = cpuid_deps; d->feature; d++) { +if (!cpu_has (c, d->depends)) +clear_cpu_cap (c, d->feature); + } +}
Signature · 1 line
2.39.0.rc0.267.gcb52ba06e7-goog
Borislav Petkov
maintainerRe: [PATCH 1/3] x86/cpu: Process all CPUID dependencies after identifying CPU info
Quoted 2 lines · Sean Christopherson
Process all CPUID dependencies to ensure that a dependent is disabled if one or more of its parent features is unsupported.
Just out of curiosity: this is some weird guest configuration, right? Not addressing a real hw issue...
Quoted 17 lines
As is,
cpuid_depsis
processed if an only if a feature is explicitly disabled viaclear_cpu_cap(), which makes it annoying/dangerous to usecpuid_depsfor features whose parent(s) do not always have explicit processing.E.g. VMX and SGX depend on the synthetic
X86_FEATURE_MSR_IA32_FEAT_CTL, but there is no common location to clearMSR_IA32_FEAT_CTL, and so consumers of VMX and SGX are forced to checkMSR_IA32_FEAT_CTLon top of the dependent feature.Manually clearing
X86_FEATURE_MSR_IA32_FEAT_CTLis the obvious alternative, but it's subtly more difficult that updatinginit_ia32_feat_ctl().CONFIG_IA32_FEAT_CTLdepends on any of CONFIG_CPU_SUP_{INTEL,CENATUR,ZHAOXIN}, butinit_ia32_feat_ctl()is invoked if and only if the actual CPU type matches one of the aforementioned CPU_SUP_* types. E.g. running a kernel built with
Quoted 7 lines
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
#CONFIG_CPU_SUP_HYGONis not set
#CONFIG_CPU_SUP_CENTAURis not set
#CONFIG_CPU_SUP_ZHAOXINis not seton a Cenatur or Zhaoxin CPU will leave
X86_FEATURE_VMXset but not set
Typo fix for the committer: Centaur
Quoted 10 lines
X86_FEATURE_MSR_IA32_FEAT_CTL, and will never callinit_ia32_feat_ctl()to give the kernel a convenient opportunity to clearX86_FEATURE_MSR_IA32_FEAT_CTL.Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/include/asm/cpufeature.h | 1 + arch/x86/kernel/cpu/common.c | 6 ++++++ arch/x86/kernel/cpu/cpuid-deps.c | 10 ++++++++++ 3 files changed, 17 insertions(+)
...
arch/x86/kernel/cpu/common.c static void identify_cpu(struct cpuinfo_x86 *c) · line 1894
Quoted 13 lines
arch/x86/kernel/cpu/common.c +5
-0@@ -1887,6 +1887,12 @@ static void identify_cpu(struct cpuinfo_x86 *c)ppin_init (c); +/* +* Apply CPUID dependencies to ensure dependent features are disabled +* if a parent feature is unsupported but wasn't explicitly disabled. +*/ +apply_cpuid_deps (c);
I'd probably call that resolve_cpuid_deps()...
But yeah, that init path would need cleaning up at some point - all kinds of init detection functions have been hastily slapped there over the years...
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteSean Christopherson
maintainerRe: [PATCH 1/3] x86/cpu: Process all CPUID dependencies after identifying CPU info
Quoted 5 lines · Borislav Petkov
Quoted 2 lines · Sean Christopherson
Process all CPUID dependencies to ensure that a dependent is disabled if one or more of its parent features is unsupported.
Just out of curiosity: this is some weird guest configuration, right?
No, it's also relevant for bare metal.
Quoted 1 line
Not addressing a real hw issue...
But it's not really a hardware issue either. More like an admin/user issue.
The problem is that if a kernel is built for subset of CPU types, e.g. just Intel or just Centaur, and then booted on an "unsupported" CPU type, init_ia32_feat_ctl() will never be invoked because ->c_init() will point a default_init(), and so the kernel never checks MSR_IA32_FEAT_CTL to see if VMX and/or SGX are fully enabled.
E.g. if someone booted an "unsupported" kernel and also disabled VMX in BIOS, then the CPU will enumerate support for VMX in CPUID, but attempting to actually enable VMX will fail due to VMX being disabled in MSR_IA32_FEAT_CTL.
Quoted 8 lines
Quoted 8 lines
arch/x86/kernel/cpu/common.c +0
-0@@ -1887,6 +1887,12 @@ static void identify_cpu(struct cpuinfo_x86 *c)ppin_init (c);
Quoted 7 lines
arch/x86/kernel/cpu/common.c static void identify_cpu(struct cpuinfo_x86 *c) · line 1894
Quoted 5 lines
arch/x86/kernel/cpu/common.c +5
-0+/* +* Apply CPUID dependencies to ensure dependent features are disabled +* if a parent feature is unsupported but wasn't explicitly disabled. +*/ +apply_cpuid_deps (c);I'd probably call that
resolve_cpuid_deps()...
"resolve" works for me.
Borislav Petkov
maintainerRe: [PATCH 1/3] x86/cpu: Process all CPUID dependencies after identifying CPU info
Quoted 6 lines · Sean Christopherson
But it's not really a hardware issue either. More like an admin/user issue.
The problem is that if a kernel is built for subset of CPU types, e.g. just Intel or just Centaur, and then booted on an "unsupported" CPU type,
init_ia32_feat_ctl()will never be invoked because ->c_init() will point adefault_init(), and so the kernel never checksMSR_IA32_FEAT_CTLto see if VMX and/or SGX are fully enabled.
Yeah, you called it an "edge case". I'm wondering whether we should even worry about that case...
I mean, the majority of Linuxes out there are allmodconfig-like kernels and booting on unsupported CPU type doesn't happen.
Hell, I'd even say that if you attempt booting on unsupported CPU type, we should simply fail that boot attempt.
I.e., what validate_cpu() does in some cases.
IOW, I don't mind what you're doing but I wonder whether we should even go the trouble to do so or simply deny that by saying "Well, don't do that then".
Hmmm.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteSean Christopherson
maintainerRe: [PATCH 1/3] x86/cpu: Process all CPUID dependencies after identifying CPU info
Quoted 14 lines · Borislav Petkov
Quoted 6 lines · Sean Christopherson
But it's not really a hardware issue either. More like an admin/user issue.
The problem is that if a kernel is built for subset of CPU types, e.g. just Intel or just Centaur, and then booted on an "unsupported" CPU type,
init_ia32_feat_ctl()will never be invoked because ->c_init() will point adefault_init(), and so the kernel never checksMSR_IA32_FEAT_CTLto see if VMX and/or SGX are fully enabled.Yeah, you called it an "edge case". I'm wondering whether we should even worry about that case...
I mean, the majority of Linuxes out there are allmodconfig-like kernels and booting on unsupported CPU type doesn't happen.
Quoted 8 lines
Hell, I'd even say that if you attempt booting on unsupported CPU type, we should simply fail that boot attempt.
I.e., what
validate_cpu()does in some cases.IOW, I don't mind what you're doing but I wonder whether we should even go the trouble to do so or simply deny that by saying "Well, don't do that then".
I agree with the "don't do that" sentiment, but IMO refusing to boot is too much. Unlike the validate_cpu() cases, the kernel can likely boot and run just fine, albeit with limited feature enabling.
And there's a non-zero chance we'd end up with a kernel param to allow booting unknown CPUs, e.g. for people doing weird things with VMs or running old, esoteric hardware. At that point we'd end up with a more complex implementation than processing dependencies on synthetic flags, especially if there's ever a more legitimate need to process such dependencies.
Borislav Petkov
maintainerRe: [PATCH 1/3] x86/cpu: Process all CPUID dependencies after identifying CPU info
Quoted 5 lines · Sean Christopherson
And there's a non-zero chance we'd end up with a kernel param to allow booting unknown CPUs, e.g. for people doing weird things with VMs or running old, esoteric hardware. At that point we'd end up with a more complex implementation than processing dependencies on synthetic flags, especially if there's ever a more legitimate need to process such dependencies.
I'm sorry but I'm still unclear on what actual use care are we even fixing here?
If it is about people who'd like to tinker with old hw or doing weird VM things, they can just as well adjust their kernel .configs and rebuild.
Peeking around your patchset, if all this is about dropping the X86_FEATURE_MSR_IA32_FEAT_CTL check and checking only X86_FEATURE_VMX and in order to do that, you want to cover those obscure cases where init_ia32_feat_ctl() won't get run, then sure, I guess - changes look simple enough. :)
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteSean Christopherson
maintainerRe: [PATCH 1/3] x86/cpu: Process all CPUID dependencies after identifying CPU info
Quoted 8 lines · Borislav Petkov
Quoted 5 lines · Sean Christopherson
And there's a non-zero chance we'd end up with a kernel param to allow booting unknown CPUs, e.g. for people doing weird things with VMs or running old, esoteric hardware. At that point we'd end up with a more complex implementation than processing dependencies on synthetic flags, especially if there's ever a more legitimate need to process such dependencies.
I'm sorry but I'm still unclear on what actual use care are we even fixing here?
There's no fix. What I was trying to say is that modifying the kernel to refuse to boot on unknown CPUs is opening a can of worms for very little benefit.
Quoted 8 lines
If it is about people who'd like to tinker with old hw or doing weird VM things, they can just as well adjust their kernel .configs and rebuild.
Peeking around your patchset, if all this is about dropping the
X86_FEATURE_MSR_IA32_FEAT_CTLcheck and checking onlyX86_FEATURE_VMXand in order to do that, you want to cover those obscure cases whereinit_ia32_feat_ctl()won't get run, then sure, I guess - changes look simple enough. :)
Yes, this is purely to drop the explicit X86_FEATURE_MSR_IA32_FEAT_CTL checks.
Alternatively, we could just drop the checks without processing the dependency, i.e. take the stance that running KVM with a funky .config is a user error, but that feels unnecessarily hostile since it's quite easy to play nice.
Or I guess do nothing and carry the explicit checks.
Borislav Petkov
maintainerRe: [PATCH 1/3] x86/cpu: Process all CPUID dependencies after identifying CPU info
Quoted 1 line · Sean Christopherson
Yes, this is purely to drop the explicit
X86_FEATURE_MSR_IA32_FEAT_CTLchecks.
Yeah, we can do that as it is simple enough.
Btw, we already resolve deps - or forced features but whatever, it is similar - in apply_forced_caps(). And we call it right before we "sync" the feature bit arrays with boot_cpu_data's in identify_cpu().
So I'm thinking this all, including your change, should be carved out in a separate function and all the CPU flags massaging should be concentrated there.
And that should happen last in identify_cpu() - that ppin_init() thing sets and clears cpu caps too. ;-\
But I can do that ontop, so how do you wanna merge this?
I take it or I review it and you take it or... ?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteSean Christopherson
maintainerPATCH 2/3 x86/cpu: Mark SGX and VMX as being dependent on MSR_IA32_FEAT_CTL
In reply to Sean ChristophersonMark SGX and VMX as being dependent on MSR_IA32_FEAT_CTL now that CPUID dependencies are processed unconditionally, i.e. now that SGX and VMX will be disabled if MSR_IA32_FEAT_CTL is unsupported even if the kernel never explicitly disables MSR_IA32_FEAT_CTL. Since init_ia32_feat_ctl() is invoked if and only if the CPU might possibly support the MSR and the kernel was built with the necessary CPU_SUP_*=y, it's possible for a CPU that supports VMX and/or SGX to run on a kernel that never sets the feature flag.
Explicitly clear MSR_IA32_FEAT_CTL if reading the MSR faults to handle the extremely unlikely edge case where the RDMSR fails when restoring CPU state after suspend, hibernate, kexec, etc.
Capturing the SGX and VMX dependencies will allow dropping manual checks on X86_FEATURE_MSR_IA32_FEAT_CTL for code that just wants to detect if SGX or VMX is fully supported.
--- arch/x86/kernel/cpu/cpuid-deps.c | 2 ++ arch/x86/kernel/cpu/feat_ctl.c | 3 +-- 2 files changed, 3 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/cpuid-deps.c +2 -0
@@ -72,6 +72,8 @@ static const struct cpuid_dep cpuid_deps[] = { { X86_FEATURE_AVX512_FP16, X86_FEATURE_AVX512BW }, { X86_FEATURE_ENQCMD, X86_FEATURE_XSAVES }, { X86_FEATURE_PER_THREAD_MBA, X86_FEATURE_MBA }, + { X86_FEATURE_VMX, X86_FEATURE_MSR_IA32_FEAT_CTL }, + { X86_FEATURE_SGX, X86_FEATURE_MSR_IA32_FEAT_CTL }, { X86_FEATURE_SGX_LC, X86_FEATURE_SGX }, { X86_FEATURE_SGX1, X86_FEATURE_SGX }, { X86_FEATURE_SGX2, X86_FEATURE_SGX1 },
arch/x86/kernel/cpu/feat_ctl.c +1 -2
@@ -117,8 +117,7 @@ void init_ia32_feat_ctl(struct cpuinfo_x86 *c)u64 msr;if (rdmsrl_safe (MSR_IA32_FEAT_CTL , &msr)) { -clear_cpu_cap (c,X86_FEATURE_VMX ); -clear_cpu_cap (c,X86_FEATURE_SGX ); +clear_cpu_cap (c,X86_FEATURE_MSR_IA32_FEAT_CTL );return ; }
Signature · 1 line
2.39.0.rc0.267.gcb52ba06e7-goog
Sean Christopherson
maintainerPATCH 3/3 KVM: VMX: Drop manual checks on X86_FEATURE_MSR_IA32_FEAT_CTL
In reply to Sean ChristophersonDrop manual checks on X86_FEATURE_MSR_IA32_FEAT_CTL and instead rely solely on the X86_FEATURE_VMX check to detect VMX support now that VMX is disabled via apply_cpuid_deps() if X86_FEATURE_MSR_IA32_FEAT_CTL isn't supported.
--- arch/x86/kernel/cpu/bugs.c | 3 +-- arch/x86/kvm/vmx/vmx.c | 6 ++---- 2 files changed, 3 insertions(+), 6 deletions(-)
arch/x86/kernel/cpu/bugs.c +1 -2
@@ -2222,8 +2222,7 @@ static ssize_t l1tf_show_state(char *buf)static ssize_t itlb_multihit_show_state (char *buf) { -if (!boot_cpu_has (X86_FEATURE_MSR_IA32_FEAT_CTL ) || - !boot_cpu_has (X86_FEATURE_VMX )) +if (!boot_cpu_has (X86_FEATURE_VMX ))return sysfs_emit (buf,"KVM: Mitigation: VMX unsupported\n" );else if (!(cr4_read_shadow () &X86_CR4_VMXE ))return sysfs_emit (buf,"KVM: Mitigation: VMX disabled\n" );
arch/x86/kvm/vmx/vmx.c +2 -4
@@ -2423,8 +2423,7 @@ static __init int cpu_has_kvm_support(void)static __init int vmx_disabled_by_bios (void ) { -return !boot_cpu_has (X86_FEATURE_MSR_IA32_FEAT_CTL ) || - !boot_cpu_has (X86_FEATURE_VMX ); +return !boot_cpu_has (X86_FEATURE_VMX ); }static int kvm_cpu_vmxon (u64 vmxon_pointer)
@@ -7413,8 +7412,7 @@ static int __init vmx_check_processor_compat(void)struct vmcs_config vmcs_conf;struct vmx_capability vmx_cap; -if (!this_cpu_has (X86_FEATURE_MSR_IA32_FEAT_CTL ) || - !this_cpu_has (X86_FEATURE_VMX )) { +if (!this_cpu_has (X86_FEATURE_VMX )) {pr_err ("kvm: VMX is disabled on CPU %d\n" ,smp_processor_id ());return -EIO ; }
Signature · 1 line
2.39.0.rc0.267.gcb52ba06e7-goog