x86/tsx: Cope with TSX deprecation on SKL/KBL/CFL/WHL
PATCH 1/1 x86/tsx: Cope with TSX deprecation on SKL/KBL/CFL/WHL
The June 2021 microcode is formally de-featuring TSX on the older Skylake client CPUs. The workaround from the March 2019 microcode is being dropped, and replaced with additions to MSR_TSX_FORCE_ABORT to hide the HLE/RTM CPUID bits.
With this microcode in place, TSX is disabled by default on these CPUs. Backwards compatibility is provided in the same way as for TAA - RTM force aborts, rather than suffering #UD, and the CPUID bits can be hidden to recover performance.
---
CC: Jan Beulich <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Wei Liu <[email protected]>
--- docs/misc/xen-command-line.pandoc | 13 +++++ tools/misc/xen-cpuid.c | 2 +- xen/arch/x86/tsx.c | 76 +++++++++++++++++++++++++++-- xen/include/asm-x86/cpufeature.h | 1 + xen/include/asm-x86/msr-index.h | 2 + xen/include/public/arch-x86/cpufeatureset.h | 1 + 6 files changed, 89 insertions(+), 6 deletions(-)
docs/misc/xen-command-line.pandoc +13 -0
@@ -2309,6 +2309,12 @@ Several microcode updates are relevant: and formally retiring HLE from the architecture. The user can disable TSX to mitigate TAA, and elect to hide the HLE/RTM CPUID bits. + * June 2021, removing the workaround for March 2019 on client CPUs and + formally de-featured TSX on SKL/KBL/WHL/CFL (Note: SKX still retains the + March 2019 fix). Introduced the ability to hide the HLE/RTM CPUID bits. + PCR3 works fine, and TSX is disabled by default, but the user can re-enable + TSX at their own risk, accepting that the memory order erratum is unfixed. + On systems with the ability to configure TSX, this boolean offers system wide control of whether TSX is enabled or disabled.
@@ -2326,6 +2332,13 @@ control of whether TSX is enabled or disabled. ordering errata default to `true` to enable working TSX. Alternatively, selecting `tsx=0` will disable TSX and restore PCR3 to a working state. + SKX and SKL/KBL/WHL/CFL on pre-June 2021 microcode default to `true`. + Alternatively, selecting `tsx=0` will disable TSX and restore PCR3 to a + working state. + + SKL/KBL/WHL/CFL on the June 2021 microcode or later default to `false`. + Alternatively, selecting `tsx=1` will re-enable TSX at the users own risk. + ### ucode > `= List of [ <integer> | scan=<bool>, nmi=<bool>, allow-same=<bool> ]`
tools/misc/xen-cpuid.c +1 -1
@@ -162,7 +162,7 @@ static const char *const str_7d0[32] = [ 4] = "fsrm", [ 8] = "avx512-vp2intersect", [ 9] = "srbds-ctrl", - [10] = "md-clear", + [10] = "md-clear", [11] = "rtm-always-abort", /* 12 */ [13] = "tsx-force-abort", [14] = "serialize",
xen/arch/x86/tsx.c +71 -5
102 lines not on this page · raw message
xen/include/asm-x86/cpufeature.h +1 -0
@@ -131,6 +131,7 @@#define cpu_has_avx512_4vnniw boot_cpu_has(X86_FEATURE_AVX512_4VNNIW)#define cpu_has_avx512_4fmaps boot_cpu_has(X86_FEATURE_AVX512_4FMAPS)#define cpu_has_avx512_vp2intersect boot_cpu_has(X86_FEATURE_AVX512_VP2INTERSECT) +#define cpu_has_rtm_always_abort boot_cpu_has(X86_FEATURE_RTM_ALWAYS_ABORT)#define cpu_has_tsx_force_abort boot_cpu_has(X86_FEATURE_TSX_FORCE_ABORT)#define cpu_has_serialize boot_cpu_has(X86_FEATURE_SERIALIZE)#define cpu_has_arch_caps boot_cpu_has(X86_FEATURE_ARCH_CAPS)
xen/include/asm-x86/msr-index.h +2 -0
@@ -61,6 +61,8 @@#define MSR_TSX_FORCE_ABORT 0x0000010f#define TSX_FORCE_ABORT_RTM (_AC(1, ULL) << 0) +#define TSX_CPUID_CLEAR (_AC(1, ULL) << 1) +#define TSX_ENABLE_RTM (_AC(1, ULL) << 2)#define MSR_TSX_CTRL 0x00000122#define TSX_CTRL_RTM_DISABLE (_AC(1, ULL) << 0)
xen/include/public/arch-x86/cpufeatureset.h +1 -0
@@ -265,6 +265,7 @@ XEN_CPUFEATURE(FSRM, 9*32+ 4) /*A Fast Short REP MOVS */ XEN_CPUFEATURE(AVX512_VP2INTERSECT, 9*32+8) /*a VP2INTERSECT{D,Q} insns */ XEN_CPUFEATURE(SRBDS_CTRL, 9*32+ 9) /* MSR_MCU_OPT_CTRL and RNGDS_MITG_DIS. */ XEN_CPUFEATURE(MD_CLEAR, 9*32+10) /*A VERW clears microarchitectural buffers */ +XEN_CPUFEATURE(RTM_ALWAYS_ABORT, 9*32+11) /*! June 2021 TSX defeaturing in microcode. */ XEN_CPUFEATURE(TSX_FORCE_ABORT, 9*32+13) /* MSR_TSX_FORCE_ABORT.RTM_ABORT */ XEN_CPUFEATURE(SERIALIZE, 9*32+14) /*a SERIALIZE insn */XEN_CPUFEATURE (CET_IBT ,9 *32 +20 )/* CET - Indirect Branch Tracking */
Signature · 1 line
2.11.0
Quoted 26 lines · Andrew Cooper
xen/arch/x86/tsx.c +20
-0@@ -60,6 +60,38 @@ void tsx_init(void)*/ /* +* Probe for the June 2021 microcode which de-features TSX on +* client parts. (Note - this is a subset of parts impacted by +* the memory ordering errata.) +* +* RTM_ALWAYS_ABORT enumerates the new functionality, but is also +* read as zero if TSX_FORCE_ABORT.ENABLE_RTM has been set before +* we run. +* +* Undo this behaviour in Xen's view of the world. +*/ +bool has_rtm_always_abort = cpu_has_rtm_always_abort; + +if ( !has_rtm_always_abort ) + { +uint64_t val; + +rdmsrl (MSR_TSX_FORCE_ABORT , val); + +if ( val &TSX_ENABLE_RTM ) + has_rtm_always_abort = true;
xen/arch/x86/tsx.c void tsx_init(void) · line 92
Quoted 10 lines
xen/arch/x86/tsx.c +10
-0+ } + +/* +* Always force RTM_ALWAYS_ABORT to be visibile, even if it +* currently is. If the user explicitly opts to enable TSX, we'll +* set TSX_FORCE_ABORT.ENABLE_RTM and hide RTM_ALWAYS_ABORT from +* the general CPUID scan later. +*/ +if ( has_rtm_always_abort ) +setup_force_cpu_cap (X86_FEATURE_RTM_ALWAYS_ABORT );
I understand the "we'll set" part, but I don't think "we'll hide" anything explicitly. Aiui it is ...
Quoted 13 lines
xen/arch/x86/tsx.c +6
-3@@ -131,9 +170,36 @@ void tsx_init(void)/* Check bottom bit only. Higher bits are various sentinels. */ rtm_disabled = !(opt_tsx &1 ); - lo &= ~TSX_FORCE_ABORT_RTM ; -if ( rtm_disabled ) - lo |=TSX_FORCE_ABORT_RTM ; + lo &= ~(TSX_FORCE_ABORT_RTM |TSX_CPUID_CLEAR |TSX_ENABLE_RTM ); + +if ( cpu_has_rtm_always_abort ) + { +/* +* June 2021 microcode, on a client part with TSX de-featured:
xen/arch/x86/tsx.c void tsx_init(void) · line 188
Quoted 10 lines
xen/arch/x86/tsx.c +10
-0+* - There are no mitigations for the TSX memory ordering errata. +* - Performance counter 3 works. (I.e. it isn't being used by +* microcode to work around the memory ordering errata.) +* - TSX_FORCE_ABORT.FORCE_ABORT_RTM is fixed read1/write-discard. +* - TSX_FORCE_ABORT.TSX_CPUID_CLEAR can be used to hide the +* HLE/RTM CPUID bits. +* - TSX_FORCE_ABORT.ENABLE_RTM may be used to opt in to +* re-enabling RTM, at the users own risk. +*/ + lo |= rtm_disabled ?TSX_CPUID_CLEAR :TSX_ENABLE_RTM ;
... the setting of TSX_ENABLE_RTM here which, as a result, causes RTM_ALWAYS_ABORT to be clear. If that's correct, perhaps the wording in that earlier comment would better be something like "we'll set TSX_FORCE_ABORT.ENABLE_RTM and hence cause RTM_ALWAYS_ABORT to be hidden from the general CPUID scan later"?
If this understanding of mine is correct, then preferably with some suitable adjustment to the comment wording
Also Intel recommends this for SDVs only - can we consider such a setup supported (not to speak of security supported) at all? I guess you mean to express this by saying "at their own risk" in the cmdline doc? If so, perhaps mentioning this in SUPPORT.md would be a good thing nevertheless, notwithstanding the fact that we're not really good at expressing there how command line option use affects support status.
Jan
Quoted 60 lines · Jan Beulich
Quoted 26 lines · Andrew Cooper
xen/arch/x86/tsx.c +20
-0@@ -60,6 +60,38 @@ void tsx_init(void)*/ /* +* Probe for the June 2021 microcode which de-features TSX on +* client parts. (Note - this is a subset of parts impacted by +* the memory ordering errata.) +* +* RTM_ALWAYS_ABORT enumerates the new functionality, but is also +* read as zero if TSX_FORCE_ABORT.ENABLE_RTM has been set before +* we run. +* +* Undo this behaviour in Xen's view of the world. +*/ +bool has_rtm_always_abort = cpu_has_rtm_always_abort; + +if ( !has_rtm_always_abort ) + { +uint64_t val; + +rdmsrl (MSR_TSX_FORCE_ABORT , val); + +if ( val &TSX_ENABLE_RTM ) + has_rtm_always_abort = true;xen/arch/x86/tsx.c void tsx_init(void) · line 92
Quoted 10 lines
xen/arch/x86/tsx.c +10
-0+ } + +/* +* Always force RTM_ALWAYS_ABORT to be visibile, even if it +* currently is. If the user explicitly opts to enable TSX, we'll +* set TSX_FORCE_ABORT.ENABLE_RTM and hide RTM_ALWAYS_ABORT from +* the general CPUID scan later. +*/ +if ( has_rtm_always_abort ) +setup_force_cpu_cap (X86_FEATURE_RTM_ALWAYS_ABORT );I understand the "we'll set" part, but I don't think "we'll hide" anything explicitly. Aiui it is ...
Quoted 20 lines
xen/arch/x86/tsx.c +13
-3@@ -131,9 +170,36 @@ void tsx_init(void)/* Check bottom bit only. Higher bits are various sentinels. */ rtm_disabled = !(opt_tsx &1 ); - lo &= ~TSX_FORCE_ABORT_RTM ; -if ( rtm_disabled ) - lo |=TSX_FORCE_ABORT_RTM ; + lo &= ~(TSX_FORCE_ABORT_RTM |TSX_CPUID_CLEAR |TSX_ENABLE_RTM ); + +if ( cpu_has_rtm_always_abort ) + { +/* +* June 2021 microcode, on a client part with TSX de-featured: +* - There are no mitigations for the TSX memory ordering errata. +* - Performance counter 3 works. (I.e. it isn't being used by +* microcode to work around the memory ordering errata.) +* - TSX_FORCE_ABORT.FORCE_ABORT_RTM is fixed read1/write-discard. +* - TSX_FORCE_ABORT.TSX_CPUID_CLEAR can be used to hide the +* HLE/RTM CPUID bits. +* - TSX_FORCE_ABORT.ENABLE_RTM may be used to opt in to
Quoted 8 lines
xen/arch/x86/tsx.c void tsx_init(void) · line 188
Quoted 3 lines
xen/arch/x86/tsx.c +3
-0+* re-enabling RTM, at the users own risk. +*/ + lo |= rtm_disabled ?TSX_CPUID_CLEAR :TSX_ENABLE_RTM ;... the setting of
TSX_ENABLE_RTMhere which, as a result, causesRTM_ALWAYS_ABORTto be clear. If that's correct, perhaps the wording in that earlier comment would better be something like "we'll set TSX_FORCE_ABORT.ENABLE_RTM and hence causeRTM_ALWAYS_ABORTto be hidden from the general CPUID scan later"?
Yes - that is the intended meaning. I'll adjust.
Quoted 3 lines
If this understanding of mine is correct, then preferably with some suitable adjustment to the comment wording
Reviewed-by Jan Beulich <[email protected]>
Thanks.
Quoted 7 lines
Also Intel recommends this for SDVs only - can we consider such a setup supported (not to speak of security supported) at all? I guess you mean to express this by saying "at their own risk" in the cmdline doc? If so, perhaps mentioning this in SUPPORT.md would be a good thing nevertheless, notwithstanding the fact that we're not really good at expressing there how command line option use affects support status.
I think this is too fine grained to be expressed in SUPPORT.md, but given that there is a very clear specific issue, I wouldn't consider this an unsupported configuration overall.
I don't expect people to want to use TSX on these CPUs in the first place (which was a factor in choosing off-by-default), but if they do, there is one specific memory ordering issue of read/writes with a committed transaction.
None of our code uses RTM, so issues in Xen which manifest with tsx=1 won't be related to TSX being enabled.
Obviously, if someone does report an issue, we can tell them to re-confirm the issue without tsx=1 just to rule out interactions, but I don't think it is likely for it to be relevant to reported issues.
~Andrew