x86/cpu, kvm: Support AMD Automatic IBRS
Subsystems (5)
v5 cover x86/cpu, kvm: Support AMD Automatic IBRS
The AMD Zen4 core supports a new feature called Automatic IBRS (Indirect Branch Restricted Speculation).
Enable Automatic IBRS by default if the CPU feature is present. It typically provides greater performance over the incumbent generic retpolines mitigation.
Patches 1-3 take the existing CPUID 0x80000021 EAX feature bits that are being propagated to the guest and define scattered versions for patch 4.
Patch 4 moves CPUID 0x80000021 EAX feature bits propagation code to kvm_set_cpu_caps().
Patch 5 Defines the AutoIBRS feature bit.
Patch 6 Adds support for AutoIBRS by turning its EFER enablement bit on at startup if the feature is available.
Patch 7 Adds support for propagating AutoIBRS to the guest.
Thanks to Babu Moger for helping debug guest propagation (patch 7).
Also thanks to Boris and Paolo for their early version diffs.
v5: Address v4 comments from Dave Hansen, Pawan Gupta, and Boris:
- Don't add new user-visible 'autoibrs' command line options that we have to document: reuse 'eibrs'
- Update Documentation/admin-guide/hw-vuln/spectre.rst
- Add NO_EIBRS_PBRSB to Hygon as well
- Re-word commit texts to not use words like 'us'
v4: https://lore.kernel.org/lkml/[email protected]/
Moved some kvm bits that had crept into patch 6/7 back into 7/7, and addressed v3 comments:
- Don't put ", kvm" in titles of patches that don't touch kvm. [SeanC]
- () after function names, i.e. kvm_set_cpu_caps(). [SeanC]
- follow the established kvm_cpu_cap_init_scattered() style [SeanC]
- Add using cpu_feature_enabled() instead of static_cpu_has() to commit text [SeanC]
- Pawan Gupta mentioned that the ordering of enabling the Intel feature bit past Intel EIBRS bug detection could be avoided by setting NO_EIBRS_PBRSB to cpu_vuln_whitelist, so did that which allowed regrouping all EIBRS related code to one place in cpu_set_bug_bits().
v3: https://lore.kernel.org/lkml/[email protected]/
- Remove Co-developed-bys. They require signed-off-bys,
so co-developers need to add them themselves.
- update check_null_seg_clears_base() [Boris]
- Made the feature bit additions separate patches
because v2 patch was clearly doing too many things at once.
v2: https://lore.kernel.org/lkml/[email protected]/
https://lkml.org/lkml/2022/11/23/1690
- Use synthetic/scattered bits instead of introducing new leaf [Boris]
- Combine the rest of the leaf's bits being used [Paolo]
Note: Bits not used by the host can be moved to kvm/cpuid.c if maintainers do not want them in cpufeatures.h.
- Hoist bitsetting code to kvm_set_cpu_caps(), and use cpuid_entry_override() in __do_cpuid_func() [Paolo]
- Reuse SPECTRE_V2_EIBRS spectre_v2_mitigation enum [Boris, PeterZ, D.Hansen]
- Change from Boris' diff:
Moved setting X86_FEATURE_IBRS_ENHANCED to after BUG_EIBRS_PBRSB so PBRSB mitigations wouldn't be enabled.
- Allow for users to specify "autoibrs,lfence/retpoline" instead of actively preventing the extra protections. AutoIBRS doesn't require the extra protection, but we allow it anyway.
v1: https://lore.kernel.org/lkml/[email protected]/
Kim Phillips (7):
x86/cpu: Define a scattered No Nested Data Breakpoints feature bit x86/cpu: Define a scattered Null Selector Clears Base feature bit x86/cpu: Make X86_FEATURE_LFENCE_RDTSC a scattered feature bit x86/cpu, kvm: Move CPUID 0x80000021 EAX feature bits propagation to
kvm_set_cpu_caps()
x86/cpu: Define a scattered AMD Automatic IBRS feature bit x86/cpu: Support AMD Automatic IBRS
x86/cpu, kvm: Propagate the AMD Automatic IBRS feature to the guest
Documentation/admin-guide/hw-vuln/spectre.rst | 6 ++--- .../admin-guide/kernel-parameters.txt | 6 ++--- arch/x86/include/asm/cpufeatures.h | 3 +++ arch/x86/include/asm/msr-index.h | 2 ++ arch/x86/kernel/cpu/bugs.c | 20 +++++++++------ arch/x86/kernel/cpu/common.c | 22 ++++++++-------- arch/x86/kernel/cpu/scattered.c | 4 +++ arch/x86/kvm/cpuid.c | 25 ++++++++----------- arch/x86/kvm/reverse_cpuid.h | 24 +++++++++++++----- arch/x86/kvm/svm/svm.c | 3 +++ arch/x86/kvm/x86.c | 3 +++ 11 files changed, 73 insertions(+), 45 deletions(-)
Signature · 1 line
2.34.1
PATCH v5 1/7 x86/cpu: Define a scattered No Nested Data Breakpoints feature bit
It's a part of the CPUID 0x80000021 leaf, and will be grouped with other feature bits to being propagated via kvm_set_cpu_caps() instead of open-coding them in __do_cpuid_func().
--- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/kernel/cpu/scattered.c | 1 + 2 files changed, 2 insertions(+)
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -307,6 +307,7 @@#define X86_FEATURE_SGX_EDECCSSA (11*32+18)/* "" SGX EDECCSSA user leaf function */ #define X86_FEATURE_CALL_DEPTH (11*32+19)/* "" Call depth tracking for RSB stuffing */ #define X86_FEATURE_MSR_TSX_CTRL (11*32+20)/* "" MSR IA32_TSX_CTRL (Intel) implemented */ +#define X86_FEATURE_NO_NESTED_DATA_BP (11*32+21)/* "" AMD No Nested Data Breakpoints */ /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX_VNNI (12*32+ 4)/* AVX VNNI instructions */
arch/x86/kernel/cpu/scattered.c +1 -0
@@ -45,6 +45,7 @@ static const struct cpuid_bit cpuid_bits[] = { { X86_FEATURE_CPB, CPUID_EDX, 9, 0x80000007, 0 }, { X86_FEATURE_PROC_FEEDBACK, CPUID_EDX, 11, 0x80000007, 0 }, { X86_FEATURE_MBA, CPUID_EBX, 6, 0x80000008, 0 }, + { X86_FEATURE_NO_NESTED_DATA_BP,CPUID_EAX, 0, 0x80000021, 0 }, { X86_FEATURE_PERFMON_V2, CPUID_EAX, 0, 0x80000022, 0 }, { X86_FEATURE_AMD_LBR_V2, CPUID_EAX, 1, 0x80000022, 0 }, { 0, 0, 0, 0, 0 }
Signature · 1 line
2.34.1
PATCH v5 2/7 x86/cpu: Define a scattered Null Selector Clears Base feature bit
In reply to Kim PhillipsIt's a part of the CPUID 0x80000021 leaf, and will be grouped with other feature bits to being propagated via kvm_set_cpu_caps() instead of open-coding them in __do_cpuid_func().
Also use the feature bit definition in check_null_seg_clears_base() instead of open-coding it.
--- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/kernel/cpu/common.c | 3 +-- arch/x86/kernel/cpu/scattered.c | 1 + 3 files changed, 3 insertions(+), 2 deletions(-)
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -308,6 +308,7 @@#define X86_FEATURE_CALL_DEPTH (11*32+19)/* "" Call depth tracking for RSB stuffing */ #define X86_FEATURE_MSR_TSX_CTRL (11*32+20)/* "" MSR IA32_TSX_CTRL (Intel) implemented */ #define X86_FEATURE_NO_NESTED_DATA_BP (11*32+21)/* "" AMD No Nested Data Breakpoints */ +#define X86_FEATURE_NULL_SEL_CLR_BASE (11*32+22)/* "" AMD Null Selector Clears Base */ /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX_VNNI (12*32+ 4)/* AVX VNNI instructions */
arch/x86/kernel/cpu/common.c +1 -2
@@ -1683,8 +1683,7 @@ void check_null_seg_clears_base(struct cpuinfo_x86 *c)return ;/* Zen3 CPUs advertise Null Selector Clears Base in CPUID. */ -if (c->extended_cpuid_level >=0x80000021 && -cpuid_eax (0x80000021 ) &BIT (6 )) +if (cpu_has (c,X86_FEATURE_NULL_SEL_CLR_BASE ))return ;/*
arch/x86/kernel/cpu/scattered.c +1 -0
@@ -46,6 +46,7 @@ static const struct cpuid_bit cpuid_bits[] = { { X86_FEATURE_PROC_FEEDBACK, CPUID_EDX, 11, 0x80000007, 0 }, { X86_FEATURE_MBA, CPUID_EBX, 6, 0x80000008, 0 }, { X86_FEATURE_NO_NESTED_DATA_BP,CPUID_EAX, 0, 0x80000021, 0 }, + { X86_FEATURE_NULL_SEL_CLR_BASE,CPUID_EAX, 6, 0x80000021, 0 }, { X86_FEATURE_PERFMON_V2, CPUID_EAX, 0, 0x80000022, 0 }, { X86_FEATURE_AMD_LBR_V2, CPUID_EAX, 1, 0x80000022, 0 }, { 0, 0, 0, 0, 0 }
Signature · 1 line
2.34.1
PATCH v5 3/7 x86/cpu: Make X86_FEATURE_LFENCE_RDTSC a scattered feature bit
In reply to Kim PhillipsIt's a part of the CPUID 0x80000021 leaf, and will be grouped with other feature bits to being propagated via kvm_set_cpu_caps() instead of open-coding them in __do_cpuid_func().
Unlike the other CPUID 0x80000021 EAX feature bits, X86_FEATURE_LFENCE_RDTSC already had an entry in cpufeatures.h.
--- arch/x86/kernel/cpu/scattered.c | 1 + 1 file changed, 1 insertion(+)
arch/x86/kernel/cpu/scattered.c +1 -0
@@ -46,6 +46,7 @@ static const struct cpuid_bit cpuid_bits[] = { { X86_FEATURE_PROC_FEEDBACK, CPUID_EDX, 11, 0x80000007, 0 }, { X86_FEATURE_MBA, CPUID_EBX, 6, 0x80000008, 0 }, { X86_FEATURE_NO_NESTED_DATA_BP,CPUID_EAX, 0, 0x80000021, 0 }, + { X86_FEATURE_LFENCE_RDTSC, CPUID_EAX, 2, 0x80000021, 0 }, { X86_FEATURE_NULL_SEL_CLR_BASE,CPUID_EAX, 6, 0x80000021, 0 }, { X86_FEATURE_PERFMON_V2, CPUID_EAX, 0, 0x80000022, 0 }, { X86_FEATURE_AMD_LBR_V2, CPUID_EAX, 1, 0x80000022, 0 },
Signature · 1 line
2.34.1
Borislav Petkov
maintainerRe: [PATCH v5 3/7] x86/cpu: Make X86_FEATURE_LFENCE_RDTSC a scattered feature bit
Quoted 17 lines · Kim Phillips
It's a part of the CPUID
0x80000021leaf, and will be grouped with other feature bits to being propagated viakvm_set_cpu_caps()instead of open-coding them in__do_cpuid_func().Unlike the other CPUID
0x80000021EAX feature bits,X86_FEATURE_LFENCE_RDTSCalready had an entry incpufeatures.h.Signed-off-by Kim Phillips <[email protected]>--- arch/x86/kernel/cpu/scattered.c | 1 + 1 file changed, 1 insertion(+)arch/x86/kernel/cpu/scattered.c +0
-0
arch/x86/kernel/cpu/scattered.c static const struct cpuid_bit cpuid_bits[] = { · line 49
Quoted 4 lines
arch/x86/kernel/cpu/scattered.c +1
-0@@ -46,6 +46,7 @@ static const struct cpuid_bit cpuid_bits[] = { { X86_FEATURE_PROC_FEEDBACK, CPUID_EDX, 11, 0x80000007, 0 }, { X86_FEATURE_MBA, CPUID_EBX, 6, 0x80000008, 0 }, { X86_FEATURE_NO_NESTED_DATA_BP,CPUID_EAX, 0, 0x80000021, 0 }, + { X86_FEATURE_LFENCE_RDTSC, CPUID_EAX, 2, 0x80000021, 0 },
Hmm, so this patchset keeps growing and growing with new bits.
Perhaps my initial suggestion to make it a scattered one doesn't make a whole lot of sense anymore.
/me goes and looks at CPUID_Fn80000021_EAX [Extended Feature 2 EAX] (Core::X86::Cpuid::FeatureExt2Eax)
Yah, judging by what's there in that leaf, we are likely to use a lot more bits in the future I think you should go back to
https://lore.kernel.org/lkml/[email protected]/
But please state in that commit message that the majority of the feature bits in CPUID_Fn80000021_EAX will be used in the kernel and thus a separate leaf makes sense.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v5 4/7 x86/cpu, kvm: Move CPUID 0x80000021 EAX feature bits propagation to kvm_set_cpu_caps()
In reply to Kim PhillipsMove CPUID 0x80000021 EAX feature bits guest propagation to kvm_set_cpu_caps() instead of open-coding them in __do_cpuid_func(). Drop the bit description comments now the code is more self-describing.
Whilst there, switch to using the more efficient cpu_feature_enabled() instead of static_cpu_has().
--- arch/x86/kvm/cpuid.c | 25 ++++++++++--------------- arch/x86/kvm/reverse_cpuid.h | 22 ++++++++++++++++------ 2 files changed, 26 insertions(+), 21 deletions(-)
arch/x86/kvm/cpuid.c +10 -15
@@ -730,6 +730,15 @@ void kvm_set_cpu_caps(void)0 /* SME */ |F (SEV ) |0 /* VM_PAGE_FLUSH */ |F (SEV_ES ) | F(SME_COHERENT)); +kvm_cpu_cap_init_scattered (CPUID_8000_0021_EAX , +SF (NO_NESTED_DATA_BP ) |SF (LFENCE_RDTSC ) |0 /* SmmPgCfgLock */ | +SF (NULL_SEL_CLR_BASE ) |0 /* PrefetchCtlMsr */ + ); +if (cpu_feature_enabled (X86_FEATURE_LFENCE_RDTSC )) +kvm_cpu_cap_set (X86_FEATURE_LFENCE_RDTSC ); +if (!static_cpu_has_bug (X86_BUG_NULL_SEG )) +kvm_cpu_cap_set (X86_FEATURE_NULL_SEL_CLR_BASE ); + kvm_cpu_cap_mask(CPUID_C000_0001_EDX, F(XSTORE) | F(XSTORE_EN) | F(XCRYPT) | F(XCRYPT_EN) | F(ACE2) | F(ACE2_EN) | F(PHE) | F(PHE_EN) |
@@ -1211,21 +1220,7 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)break ;case 0x80000021 : entry->ebx = entry->ecx = entry->edx =0 ; -/* -* Pass down these bits: -* EAX 0 NNDBP, Processor ignores nested data breakpoints -* EAX 2 LAS, LFENCE always serializing -* EAX 6 NSCB, Null selector clear base -* -* Other defined bits are for MSRs that KVM does not expose: -* EAX 3 SPCL, SMM page configuration lock -* EAX 13 PCMSR, Prefetch control MSR -*/ - entry->eax &=BIT (0 ) |BIT (2 ) |BIT (6 ); -if (static_cpu_has (X86_FEATURE_LFENCE_RDTSC )) - entry->eax |=BIT (2 ); -if (!static_cpu_has_bug (X86_BUG_NULL_SEG )) - entry->eax |=BIT (6 ); +cpuid_entry_override (entry,CPUID_8000_0021_EAX );break ;/*Add support for Centaur's CPUID instruction*/ case 0xC0000000 :
arch/x86/kvm/reverse_cpuid.h +16 -6
@@ -13,6 +13,7 @@*/ enum kvm_only_cpuid_leafs {CPUID_12_EAX =NCAPINTS , +CPUID_8000_0021_EAX ,NR_KVM_CPU_CAPS ,NKVMCAPINTS =NR_KVM_CPU_CAPS -NCAPINTS ,
@@ -25,6 +26,11 @@ enum kvm_only_cpuid_leafs {#define KVM_X86_FEATURE_SGX2 KVM_X86_FEATURE(CPUID_12_EAX, 1)#define KVM_X86_FEATURE_SGX_EDECCSSA KVM_X86_FEATURE(CPUID_12_EAX, 11) +/* AMD-defined Extended Feature 2 EAX, CPUID level 0x80000021 (EAX) */ +#define KVM_X86_FEATURE_NO_NESTED_DATA_BP KVM_X86_FEATURE(CPUID_8000_0021_EAX, 0) +#define KVM_X86_FEATURE_LFENCE_RDTSC KVM_X86_FEATURE(CPUID_8000_0021_EAX, 2) +#define KVM_X86_FEATURE_NULL_SEL_CLR_BASE KVM_X86_FEATURE(CPUID_8000_0021_EAX, 6) +struct cpuid_reg {u32 function;u32 index;
@@ -49,6 +55,7 @@ static const struct cpuid_reg reverse_cpuid[] = { [CPUID_7_1_EAX] = { 7, 1, CPUID_EAX}, [CPUID_12_EAX] = {0x00000012, 0, CPUID_EAX}, [CPUID_8000_001F_EAX] = {0x8000001f, 0, CPUID_EAX}, + [CPUID_8000_0021_EAX] = {0x80000021, 0, CPUID_EAX}, };/*
@@ -75,12 +82,15 @@ static __always_inline void reverse_cpuid_check(unsigned int x86_leaf)*/ static __always_inline u32 __feature_translate(int x86_feature) { -if (x86_feature ==X86_FEATURE_SGX1 ) -return KVM_X86_FEATURE_SGX1 ; -else if (x86_feature ==X86_FEATURE_SGX2 ) -return KVM_X86_FEATURE_SGX2 ; -else if (x86_feature ==X86_FEATURE_SGX_EDECCSSA ) -return KVM_X86_FEATURE_SGX_EDECCSSA ; +switch (x86_feature) { +case X86_FEATURE_SGX1 :return KVM_X86_FEATURE_SGX1 ; +case X86_FEATURE_SGX2 :return KVM_X86_FEATURE_SGX2 ; +case X86_FEATURE_SGX_EDECCSSA :return KVM_X86_FEATURE_SGX_EDECCSSA ; +case X86_FEATURE_NO_NESTED_DATA_BP :return KVM_X86_FEATURE_NO_NESTED_DATA_BP ; +case X86_FEATURE_LFENCE_RDTSC :return KVM_X86_FEATURE_LFENCE_RDTSC ; +case X86_FEATURE_NULL_SEL_CLR_BASE :return KVM_X86_FEATURE_NULL_SEL_CLR_BASE ; +default :break ; + }return x86_feature; }
Signature · 1 line
2.34.1
PATCH v5 5/7 x86/cpu: Define a scattered AMD Automatic IBRS feature bit
In reply to Kim PhillipsIt's bit 8 of the hardware CPUID 0x80000021 EAX leaf.
--- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/kernel/cpu/scattered.c | 1 + 2 files changed, 2 insertions(+)
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -309,6 +309,7 @@#define X86_FEATURE_MSR_TSX_CTRL (11*32+20)/* "" MSR IA32_TSX_CTRL (Intel) implemented */ #define X86_FEATURE_NO_NESTED_DATA_BP (11*32+21)/* "" AMD No Nested Data Breakpoints */ #define X86_FEATURE_NULL_SEL_CLR_BASE (11*32+22)/* "" AMD Null Selector Clears Base */ +#define X86_FEATURE_AUTOIBRS (11*32+23)/* AMD Automatic IBRS */ /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX_VNNI (12*32+ 4)/* AVX VNNI instructions */
arch/x86/kernel/cpu/scattered.c +1 -0
@@ -48,6 +48,7 @@ static const struct cpuid_bit cpuid_bits[] = { { X86_FEATURE_NO_NESTED_DATA_BP,CPUID_EAX, 0, 0x80000021, 0 }, { X86_FEATURE_LFENCE_RDTSC, CPUID_EAX, 2, 0x80000021, 0 }, { X86_FEATURE_NULL_SEL_CLR_BASE,CPUID_EAX, 6, 0x80000021, 0 }, + { X86_FEATURE_AUTOIBRS, CPUID_EAX, 8, 0x80000021, 0 }, { X86_FEATURE_PERFMON_V2, CPUID_EAX, 0, 0x80000022, 0 }, { X86_FEATURE_AMD_LBR_V2, CPUID_EAX, 1, 0x80000022, 0 }, { 0, 0, 0, 0, 0 }
Signature · 1 line
2.34.1
PATCH v5 6/7 x86/cpu: Support AMD Automatic IBRS
In reply to Kim PhillipsThe AMD Zen4 core supports a new feature called Automatic IBRS.
It is a "set-and-forget" feature that means that, like Intel's Enhanced IBRS, h/w manages its IBRS mitigation resources automatically across CPL transitions.
The feature is advertised by CPUID_Fn80000021_EAX bit 8 and is enabled by setting MSR C000_0080 (EFER) bit 21.
Enable Automatic IBRS by default if the CPU feature is present. It typically provides greater performance over the incumbent generic retpolines mitigation.
Reuse the SPECTRE_V2_EIBRS spectre_v2_mitigation enum. AMD Automatic IBRS and Intel Enhanced IBRS have similar bugs.c enablement. Add NO_EIBRS_PBRSB to cpu_vuln_whitelist, since AMD Automatic IBRS isn't affected by PBRSB-eIBRS.
The kernel command line option spectre_v2=eibrs is used to select AMD Automatic IBRS, if available.
--- Documentation/admin-guide/hw-vuln/spectre.rst | 6 +++--- .../admin-guide/kernel-parameters.txt | 6 +++--- arch/x86/include/asm/msr-index.h | 2 ++ arch/x86/kernel/cpu/bugs.c | 20 +++++++++++-------- arch/x86/kernel/cpu/common.c | 19 ++++++++++-------- 5 files changed, 31 insertions(+), 22 deletions(-)
Documentation/admin-guide/hw-vuln/spectre.rst +3 -3
@@ -610,9 +610,9 @@ kernel command line. retpoline,generic Retpolines retpoline,lfence LFENCE; indirect branch retpoline,amd alias for retpoline,lfence - eibrs enhanced IBRS - eibrs,retpoline enhanced IBRS + Retpolines - eibrs,lfence enhanced IBRS + LFENCE + eibrs Enhanced/Auto IBRS + eibrs,retpoline Enhanced/Auto IBRS + Retpolines + eibrs,lfence Enhanced/Auto IBRS + LFENCE ibrs use IBRS to protect kernel Not specifying this option is equivalent to
Documentation/admin-guide/kernel-parameters.txt +3 -3
@@ -5698,9 +5698,9 @@ retpoline,generic - Retpolines retpoline,lfence - LFENCE; indirect branch retpoline,amd - alias for retpoline,lfence - eibrs - enhanced IBRS - eibrs,retpoline - enhanced IBRS + Retpolines - eibrs,lfence - enhanced IBRS + LFENCE + eibrs - Enhanced/Auto IBRS + eibrs,retpoline - Enhanced/Auto IBRS + Retpolines + eibrs,lfence - Enhanced/Auto IBRS + LFENCE ibrs - use IBRS to protect kernel Not specifying this option is equivalent to
arch/x86/include/asm/msr-index.h +2 -0
@@ -25,6 +25,7 @@#define _EFER_SVME 12/* Enable virtualization */ #define _EFER_LMSLE 13/* Long Mode Segment Limit Enable */ #define _EFER_FFXSR 14/* Enable Fast FXSAVE/FXRSTOR */ +#define _EFER_AUTOIBRS 21/* Enable Automatic IBRS */ #define EFER_SCE (1<<_EFER_SCE)#define EFER_LME (1<<_EFER_LME)
@@ -33,6 +34,7 @@#define EFER_SVME (1<<_EFER_SVME)#define EFER_LMSLE (1<<_EFER_LMSLE)#define EFER_FFXSR (1<<_EFER_FFXSR) +#define EFER_AUTOIBRS (1<<_EFER_AUTOIBRS)/* Intel MSRs. Some also available on other CPUs */
arch/x86/kernel/cpu/bugs.c +12 -8
@@ -1222,9 +1222,9 @@ static const char * const spectre_v2_strings[] = { [SPECTRE_V2_NONE] = "Vulnerable", [SPECTRE_V2_RETPOLINE] = "Mitigation: Retpolines", [SPECTRE_V2_LFENCE] = "Mitigation: LFENCE", - [SPECTRE_V2_EIBRS] = "Mitigation: Enhanced IBRS", - [SPECTRE_V2_EIBRS_LFENCE] = "Mitigation: Enhanced IBRS + LFENCE", - [SPECTRE_V2_EIBRS_RETPOLINE] = "Mitigation: Enhanced IBRS + Retpolines", + [SPECTRE_V2_EIBRS] = "Mitigation: Enhanced / Automatic IBRS", + [SPECTRE_V2_EIBRS_LFENCE] = "Mitigation: Enhanced / Automatic IBRS + LFENCE", + [SPECTRE_V2_EIBRS_RETPOLINE] = "Mitigation: Enhanced / Automatic IBRS + Retpolines", [SPECTRE_V2_IBRS] = "Mitigation: IBRS", };
@@ -1293,7 +1293,7 @@ static enum spectre_v2_mitigation_cmd __init spectre_v2_parse_cmdline(void) cmd == SPECTRE_V2_CMD_EIBRS_LFENCE || cmd == SPECTRE_V2_CMD_EIBRS_RETPOLINE) && !boot_cpu_has(X86_FEATURE_IBRS_ENHANCED)) { -pr_err ("%s selected but CPU doesn't have eIBRS. Switching to AUTO select\n" , +pr_err ("%s selected but CPU doesn't have Enhanced or Automatic IBRS. Switching to AUTO select\n" , mitigation_options[i].option);return SPECTRE_V2_CMD_AUTO ; }
@@ -1479,8 +1479,12 @@ static void __init spectre_v2_select_mitigation(void)pr_err (SPECTRE_V2_EIBRS_EBPF_MSG );if (spectre_v2_in_ibrs_mode (mode)) { - x86_spec_ctrl_base |=SPEC_CTRL_IBRS ; -write_spec_ctrl_current (x86_spec_ctrl_base, true); +if (boot_cpu_has (X86_FEATURE_AUTOIBRS )) { +msr_set_bit (MSR_EFER , _EFER_AUTOIBRS); + }else { + x86_spec_ctrl_base |=SPEC_CTRL_IBRS ; +write_spec_ctrl_current (x86_spec_ctrl_base, true); + } }switch (mode) {
@@ -1564,8 +1568,8 @@ static void __init spectre_v2_select_mitigation(void)/* * Retpoline protects the kernel, but doesn't protect firmware. IBRS * and Enhanced IBRS protect firmware too, so enable IBRS around -* firmware calls only when IBRS / Enhanced IBRS aren't otherwise -* enabled. +* firmware calls only when IBRS / Enhanced / Automatic IBRS aren't +* otherwise enabled. * * Use "mode" to check Enhanced IBRS instead of boot_cpu_has(), because * the user might select retpoline on the kernel command line and if
arch/x86/kernel/cpu/common.c +11 -8
@@ -1226,8 +1226,8 @@ static const __initconst struct x86_cpu_id cpu_vuln_whitelist[] = {VULNWL_AMD (0x12 ,NO_MELTDOWN |NO_SSB |NO_L1TF |NO_MDS |NO_SWAPGS |NO_ITLB_MULTIHIT |NO_MMIO ),/* FAMILY_ANY must be last, otherwise 0x0f - 0x12 matches won't work */ -VULNWL_AMD (X86_FAMILY_ANY ,NO_MELTDOWN |NO_L1TF |NO_MDS |NO_SWAPGS |NO_ITLB_MULTIHIT |NO_MMIO ), -VULNWL_HYGON (X86_FAMILY_ANY ,NO_MELTDOWN |NO_L1TF |NO_MDS |NO_SWAPGS |NO_ITLB_MULTIHIT |NO_MMIO ), +VULNWL_AMD (X86_FAMILY_ANY ,NO_MELTDOWN |NO_L1TF |NO_MDS |NO_SWAPGS |NO_ITLB_MULTIHIT |NO_MMIO |NO_EIBRS_PBRSB ), +VULNWL_HYGON (X86_FAMILY_ANY ,NO_MELTDOWN |NO_L1TF |NO_MDS |NO_SWAPGS |NO_ITLB_MULTIHIT |NO_MMIO |NO_EIBRS_PBRSB ),/* Zhaoxin Family 7 */ VULNWL(CENTAUR, 7, X86_MODEL_ANY, NO_SPECTRE_V2 | NO_SWAPGS | NO_MMIO),
@@ -1338,8 +1338,16 @@ static void __init cpu_set_bug_bits(struct cpuinfo_x86 *c) !cpu_has(c, X86_FEATURE_AMD_SSB_NO))setup_force_cpu_bug (X86_BUG_SPEC_STORE_BYPASS ); -if (ia32_cap &ARCH_CAP_IBRS_ALL ) +/* +* AMD's AutoIBRS is equivalent to Intel's eIBRS - use the Intel feature +* flag and protect from vendor-specific bugs via the whitelist. +*/ +if ((ia32_cap &ARCH_CAP_IBRS_ALL ) ||cpu_has (c,X86_FEATURE_AUTOIBRS )) {setup_force_cpu_cap (X86_FEATURE_IBRS_ENHANCED ); +if (!cpu_matches (cpu_vuln_whitelist,NO_EIBRS_PBRSB ) && + !(ia32_cap &ARCH_CAP_PBRSB_NO )) +setup_force_cpu_bug (X86_BUG_EIBRS_PBRSB ); + }if (!cpu_matches (cpu_vuln_whitelist,NO_MDS ) && !(ia32_cap &ARCH_CAP_MDS_NO )) {
@@ -1401,11 +1409,6 @@ static void __init cpu_set_bug_bits(struct cpuinfo_x86 *c)setup_force_cpu_bug (X86_BUG_RETBLEED ); } -if (cpu_has (c,X86_FEATURE_IBRS_ENHANCED ) && - !cpu_matches (cpu_vuln_whitelist,NO_EIBRS_PBRSB ) && - !(ia32_cap &ARCH_CAP_PBRSB_NO )) -setup_force_cpu_bug (X86_BUG_EIBRS_PBRSB ); -if (cpu_matches (cpu_vuln_whitelist,NO_MELTDOWN ))return ;
Signature · 1 line
2.34.1
PATCH v5 7/7 x86/cpu, kvm: Propagate the AMD Automatic IBRS feature to the guest
In reply to Kim PhillipsAdd the AMD Automatic IBRS feature bit to those being propagated to the guest, and enable the guest EFER bit.
--- arch/x86/kvm/cpuid.c | 2 +- arch/x86/kvm/reverse_cpuid.h | 2 ++ arch/x86/kvm/svm/svm.c | 3 +++ arch/x86/kvm/x86.c | 3 +++ 4 files changed, 9 insertions(+), 1 deletion(-)
arch/x86/kvm/cpuid.c +1 -1
@@ -732,7 +732,7 @@ void kvm_set_cpu_caps(void)kvm_cpu_cap_init_scattered (CPUID_8000_0021_EAX ,SF (NO_NESTED_DATA_BP ) |SF (LFENCE_RDTSC ) |0 /* SmmPgCfgLock */ | -SF (NULL_SEL_CLR_BASE ) |0 /* PrefetchCtlMsr */ +SF (NULL_SEL_CLR_BASE ) |SF (AUTOIBRS ) |0 /* PrefetchCtlMsr */ );if (cpu_feature_enabled (X86_FEATURE_LFENCE_RDTSC ))kvm_cpu_cap_set (X86_FEATURE_LFENCE_RDTSC );
arch/x86/kvm/reverse_cpuid.h +2 -0
@@ -30,6 +30,7 @@ enum kvm_only_cpuid_leafs {#define KVM_X86_FEATURE_NO_NESTED_DATA_BP KVM_X86_FEATURE(CPUID_8000_0021_EAX, 0)#define KVM_X86_FEATURE_LFENCE_RDTSC KVM_X86_FEATURE(CPUID_8000_0021_EAX, 2)#define KVM_X86_FEATURE_NULL_SEL_CLR_BASE KVM_X86_FEATURE(CPUID_8000_0021_EAX, 6) +#define KVM_X86_FEATURE_AUTOIBRS KVM_X86_FEATURE(CPUID_8000_0021_EAX, 8)struct cpuid_reg {u32 function;
@@ -89,6 +90,7 @@ static __always_inline u32 __feature_translate(int x86_feature)case X86_FEATURE_NO_NESTED_DATA_BP :return KVM_X86_FEATURE_NO_NESTED_DATA_BP ;case X86_FEATURE_LFENCE_RDTSC :return KVM_X86_FEATURE_LFENCE_RDTSC ;case X86_FEATURE_NULL_SEL_CLR_BASE :return KVM_X86_FEATURE_NULL_SEL_CLR_BASE ; +case X86_FEATURE_AUTOIBRS :return KVM_X86_FEATURE_AUTOIBRS ;default :break ; }
arch/x86/kvm/svm/svm.c +3 -0
@@ -4946,6 +4946,9 @@ static __init int svm_hardware_setup(void) tsc_aux_uret_slot =kvm_add_user_return_msr (MSR_TSC_AUX ); +if (boot_cpu_has (X86_FEATURE_AUTOIBRS )) +kvm_enable_efer_bits (EFER_AUTOIBRS ); +/* Check for pause filtering support */ if (!boot_cpu_has (X86_FEATURE_PAUSEFILTER )) { pause_filter_count =0 ;
arch/x86/kvm/x86.c +3 -0
@@ -1688,6 +1688,9 @@ static int do_get_msr_feature(struct kvm_vcpu *vcpu, unsigned index, u64 *data)static bool __kvm_valid_efer (struct kvm_vcpu *vcpu,u64 efer) { +if (efer &EFER_AUTOIBRS && !guest_cpuid_has (vcpu,X86_FEATURE_AUTOIBRS )) +return false; +if (efer &EFER_FFXSR && !guest_cpuid_has (vcpu,X86_FEATURE_FXSR_OPT ))return false;
Signature · 1 line
2.34.1