TAA 0

From: Pawan Gupta <[email protected]>
Subject: [PATCH 1/9] x86/msr: Add the IA32_TSX_CTRL MSR

Transactional Synchronization Extensions (TSX) may be used on certain processors as part of a speculative side channel attack. A microcode update for existing processors that are vulnerable to this attack will add a new MSR - IA32_TSX_CTRL to allow the system administrator the option to disable TSX as one of the possible mitigations.

[ Note that future processors that are not vulnerable will also
support the IA32_TSX_CTRL MSR. ]

Add defines for the new IA32_TSX_CTRL MSR and its bits.

TSX has two sub-features:

1. Restricted Transactional Memory (RTM) is an explicitly-used feature where new instructions begin and end TSX transactions.
2. Hardware Lock Elision (HLE) is implicitly used when certain kinds of "old" style locks are used by software.

Bit 7 of the IA32_ARCH_CAPABILITIES indicates the presence of the IA32_TSX_CTRL MSR.

There are two control bits in IA32_TSX_CTRL MSR:

Bit 0: When set, it disables the Restricted Transactional Memory (RTM)
sub-feature of TSX (will force all transactions to abort on the XBEGIN instruction).

Bit 1: When set, it disables the enumeration of the RTM and HLE feature
(i.e. it will make CPUID(EAX=7).EBX{bit4} and
CPUID(EAX=7).EBX{bit11} read as 0).

The other TSX sub-feature, Hardware Lock Elision (HLE), is unconditionally disabled

but still enumerated as present by
CPUID(EAX=7).EBX{bit4}.

This should clarify that not *all* TAA-vulnerable CPUs will get IA32_TSX_CTRL, instead only the ones which aren't also vulnerable to MDS.

Quoted 2 lines

The other TSX sub-feature, Hardware Lock Elision (HLE), is unconditionally disabled

... by the new microcode ...

Quoted 2 lines

but still enumerated as present by
CPUID(EAX=7).EBX{bit4}.

... unless disabled by bit 1 of IA32_TSX_CTRL_MSR.

Quoted 3 lines · speck for Josh Poimboeuf

This should clarify that not *all* TAA-vulnerable CPUs will get IA32_TSX_CTRL, instead only the ones which aren't also vulnerable to MDS.

Added:

"The CPUs which get this new MSR after a microcode upgrade are the ones which do not set MSR_IA32_ARCH_CAPABILITIES.MDS_NO (bit 5) because those CPUs have CPUID.MD_CLEAR, i.e., the VERW implementation which clears all CPU buffers takes care of the TAA case as well."

Hopefully that makes it more clear.

Quoted 8 lines

Quoted 2 lines

The other TSX sub-feature, Hardware Lock Elision (HLE), is unconditionally disabled

... by the new microcode ...

Quoted 2 lines

but still enumerated as present by
CPUID(EAX=7).EBX{bit4}.

Quoted 1 line

... unless disabled by bit 1 of IA32_TSX_CTRL_MSR.

Changed it to:

"The other TSX sub-feature, Hardware Lock Elision (HLE), is unconditionally disabled by the new microcode but still enumerated as present by CPUID(EAX=7).EBX{bit4}, unless disabled by IA32_TSX_CTRL_MSR[1] - TSX_CTRL_CPUID_CLEAR."

Documentation/admin-guide/kernel-parameters.txt +0 -0

@@ -4848,6 +4848,17 @@
 			interruptions from clocksource watchdog are not
 			acceptable).
 

This still needs details about when 'tsx=off' does and doesn't work.

The above makes it sound like it's off for all CPUs, when in fact it's only off for newer MDS_NO CPUs.

It should also perhaps describe the risks associated with tsx=on. While there are mitigations for all known issues (i.e., the tsx_async_abort= option), TSX has been known to be an accelerator for several previous speculation-related CVEs, and so there may be unknown security risks associated with leaving it enabled.

Quoted 4 lines · speck for Josh Poimboeuf

This still needs details about when 'tsx=off' does and doesn't work.

The above makes it sound like it's off for all CPUs, when in fact it's only off for newer MDS_NO CPUs.

How does that sound (and that is being mentioned somewhere in all the text but here it is important to have):

off - Disable TSX on the system. (Note that this
option takes effect only on newer CPUs which are not vulnerable to MDS, i.e., have MSR_IA32_ARCH_CAPABILITIES.MDS_NO=1 and which get the new IA32_TSX_CTRL MSR through a microcode update. This new MSR allows for the reliable deactivation of the TSX functionality.)

Quoted 5 lines

It should also perhaps describe the risks associated with tsx=on. While there are mitigations for all known issues (i.e., the tsx_async_abort= option), TSX has been known to be an accelerator for several previous speculation-related CVEs, and so there may be unknown security risks associated with leaving it enabled.

You've basically said it nicely already:

"Although there are mitigations for all known security vulnerabilities, TSX has been known to be an accelerator for several previous speculation-related CVEs, and so there may be unknown security risks associated with leaving it enabled."

ACK?

arch/x86/kernel/cpu/tsx.c line 101

Quoted 9 lines · speck for Pawan Gupta

arch/x86/kernel/cpu/tsx.c new file +9 -0

+	if (tsx_ctrl_state == TSX_CTRL_DISABLE) {
+		tsx_disable();
+
+		/*
+		 * tsx_disable() will change the state of the
+		 * RTM CPUID bit.  Clear it here since it is now
+		 * expected to be not set.
+		 */
+		setup_clear_cpu_cap(X86_FEATURE_RTM);

This same argument applies to HLE, and it would be weird for pre-TSX_CTRL CPUs with tsx=off to report HLE but not RTM in /proc/cpuid

Furthermore, while grepping through the tree, I found

events/intel/lbr.c-267-static inline bool lbr_from_signext_quirk_needed(void)
events/intel/lbr.c-268-{
events/intel/lbr.c-269- int lbr_format = x86_pmu.intel_cap.lbr_format; events/intel/lbr.c:270: bool tsx_support = boot_cpu_has(X86_FEATURE_HLE) || events/intel/lbr.c-271-                    boot_cpu_has(X86_FEATURE_RTM); events/intel/lbr.c-272-
events/intel/lbr.c-273- return !tsx_support && (lbr_desc[lbr_format] & LBR_TSX);

which is going to need an adjustment to avoid applying the quirks on non-broken hardware.

Quoted 13 lines · speck for Andrew Cooper

arch/x86/kernel/cpu/tsx.c line 101

Quoted 9 lines · speck for Pawan Gupta

arch/x86/kernel/cpu/tsx.c new file +9 -0

+	if (tsx_ctrl_state == TSX_CTRL_DISABLE) {
+		tsx_disable();
+
+		/*
+		 * tsx_disable() will change the state of the
+		 * RTM CPUID bit.  Clear it here since it is now
+		 * expected to be not set.
+		 */
+		setup_clear_cpu_cap(X86_FEATURE_RTM);

This same argument applies to HLE, and it would be weird for pre-TSX_CTRL CPUs with tsx=off to report HLE but not RTM in /proc/cpuid

Right, the correct fix for that would be to run tsx_init() before we read CPUID leafs in get_cpu_cap() because then it'll load the proper bits already and we won't have to clear anything - the MSR write would've cleared both CPUID bits already.

I just checked that it happens globally even:

$ ./cpuid -r | grep -E "^\s+0x00000007" | awk '{ print $6 }' | uniq
edx=0xbc000400

Bits 4 (HLE) and 11 (RTM) are cleared on all CPUs.

I'll try this tomorrow to check whether it would even work that early.

If there's issues with it, then we'll have to do the above thing and clear HLE too, by hand.

But pre-TSX_CTRL CPUs won't reach the above code, because of the tsx_ctrl_is_supported() check at the beginning of tsx_init().

And post-TSX_CTRL CPUs will have the HLE bit already cleared by microcode.

Quoted 2 lines · speck for Josh Poimboeuf

But pre-TSX_CTRL CPUs won't reach the above code, because of the tsx_ctrl_is_supported() check at the beginning of tsx_init().

Yes, pre-TSX_CTRL won't have those CPUID bits cleared because they cannot anyway - MSR is not there.

And even if we clear the corresponding X86_FEATURE_ bits, nothing is stopping people from using TSX - they won't query /proc/cpuinfo but CPUID directly. Hell, they can even try the instructions and see if they fault or not, without querying any feature bits.

Quoted 2 lines

And post-TSX_CTRL CPUs will have the HLE bit already cleared by microcode.

Yes.

In both cases, the X86_FEATURE_* bits mirror what's in CPUID. At least for the two TSX bits: HLE and RTM.

Actually, according to the patches, with TSX_CTRL CPUs, HLE is unconditionally disabled, but still enumerated as present in CPUID (why???)

Quoted 3 lines · speck for Josh Poimboeuf

Actually, according to the patches, with TSX_CTRL CPUs, HLE is unconditionally disabled, but still enumerated as present in CPUID (why???)

I found it: in an older email thread Andrew talks about migration compatibility and HLE. Dunno if we care but maybe someone else has a better idea.

The first TSX problem in Haswell/Broadwell lead to chaos when a microcode update took feature bits out.

When the second problem came around, I understand from IRL conversations that several vendors firmly expressed their opinion about whether feature bits should disappear in microcode.

As HLE is a hint bit at best, it was left visible despite being disabled behind the scenes.

Quoted 3 lines · speck for Josh Poimboeuf

Actually, according to the patches, with TSX_CTRL CPUs, HLE is unconditionally disabled, but still enumerated as present in CPUID (why???)

I found it: in an older email thread Andrew talks about migration compatibility and HLE. Dunno if we care but maybe someone else has a better idea.

Quoted 12 lines · speck for Andrew Cooper

arch/x86/kernel/cpu/tsx.c line 101

Quoted 9 lines · speck for Pawan Gupta

arch/x86/kernel/cpu/tsx.c new file +9 -0

+	if (tsx_ctrl_state == TSX_CTRL_DISABLE) {
+		tsx_disable();
+
+		/*
+		 * tsx_disable() will change the state of the
+		 * RTM CPUID bit.  Clear it here since it is now
+		 * expected to be not set.
+		 */
+		setup_clear_cpu_cap(X86_FEATURE_RTM);

This same argument applies to HLE, and it would be weird for pre-TSX_CTRL CPUs with tsx=off to report HLE but not RTM in /proc/cpuid

Right, the correct fix for that would be to run tsx_init() before we read CPUID leafs in get_cpu_cap() because then it'll load the proper bits already and we won't have to clear anything - the MSR write would've cleared both CPUID bits already.

I just checked that it happens globally even:

$ ./cpuid -r | grep -E "^\s+0x00000007" | awk '{ print $6 }' | uniq
edx=0xbc000400

Bits 4 (HLE) and 11 (RTM) are cleared on all CPUs.

I'll try this tomorrow to check whether it would even work that early.

If there's issues with it, then we'll have to do the above thing and clear HLE too, by hand.

Quoted 12 lines

Furthermore, while grepping through the tree, I found

events/intel/lbr.c-267-static inline bool lbr_from_signext_quirk_needed(void)
events/intel/lbr.c-268-{
events/intel/lbr.c-269- int lbr_format = x86_pmu.intel_cap.lbr_format; events/intel/lbr.c:270: bool tsx_support = boot_cpu_has(X86_FEATURE_HLE) || events/intel/lbr.c-271-                    boot_cpu_has(X86_FEATURE_RTM); events/intel/lbr.c-272-
events/intel/lbr.c-273- return !tsx_support && (lbr_desc[lbr_format] & LBR_TSX);

Quoted 2 lines

which is going to need an adjustment to avoid applying the quirks on non-broken hardware.

When both HLE and RTM bits are cleared, that should still be correct, no? Or am I missing something?

On Xen, I've juggled things such that we load microcode, then interpret tsx= if the user has provided it (taking care to always write MSR_TSX_CTRL if it is available, to discard whatever settings firmware or kexec left), before querying CPUID.

Later, the spec-ctrl= interpretation happens, which might choose to turn off TSX due to TAA, which then has to modify MSR_TSX_CTRL and force clear the bits in the policy.

Given that speculative mitigations rely heavily on CPUID, I can't reason about a clean way to disentangle this, but the above seems to be the least complicated algorithm.

On Haswell and Broadwell, the microcode which turned HLE/RTM off in the pipeline left the LBR MSRs in a state where you can't context switch the value, because they would yield a value via RDMSR which WRMSR faulted on, because the two operations had an asymmetric view of how the top bits of metadata should be interpreted, given some TSX-related metadata and a sign extended linear address.

On Skylake where you can't actually turn RTM off, but we may hide FEATURE_RTM/HLE, the above quirk is probably not true.

On Cascadelake, who knows?  RTM is being turned off in the pipeline, but maybe the HSX/BWX bug has been fixed, or maybe it is being turned off in a different way, or ...

Quoted 4 lines · speck for Andrew Cooper

On Xen, I've juggled things such that we load microcode, then interpret tsx= if the user has provided it (taking care to always write MSR_TSX_CTRL if it is available, to discard whatever settings firmware or kexec left), before querying CPUID.

Yap, wanna try the same thing, in that exact order.

Quoted 3 lines

Later, the spec-ctrl= interpretation happens, which might choose to turn off TSX due to TAA, which then has to modify MSR_TSX_CTRL and force clear the bits in the policy.

Well, the kernel doesn't reeval CPUID feature bits in that case because it has gone on booting and enabled all kinds of feature supporting code. This is the reason why the whole late microcode loading is such a bad thing to do.

Quoted 8 lines

On Haswell and Broadwell, the microcode which turned HLE/RTM off in the pipeline left the LBR MSRs in a state where you can't context switch the value, because they would yield a value via RDMSR which WRMSR faulted on, because the two operations had an asymmetric view of how the top bits of metadata should be interpreted, given some TSX-related metadata and a sign extended linear address.

Quoted 2 lines

On Skylake where you can't actually turn RTM off, but we may hide FEATURE_RTM/HLE, the above quirk is probably not true.

Huh? How is that possible? TSX_CTRL has defined only bit 1 there, the CPUID enumeration bit, and bit 0 doesn't do any RTM disabling? Srsly?!

Quoted 4 lines

On Cascadelake, who knows?  RTM is being turned off in the pipeline, but maybe the HSX/BWX bug has been fixed, or maybe it is being turned off in a different way, or ...

Right, I guess we'll deal with any perfcounters fallout in public, when the stuff releases...

Quoted 7 lines

Quoted 7 lines

On Haswell and Broadwell, the microcode which turned HLE/RTM off in the pipeline left the LBR MSRs in a state where you can't context switch the value, because they would yield a value via RDMSR which WRMSR faulted on, because the two operations had an asymmetric view of how the top bits of metadata should be interpreted, given some TSX-related metadata and a sign extended linear address.

Quoted 4 lines

Quoted 2 lines

On Skylake where you can't actually turn RTM off, but we may hide FEATURE_RTM/HLE, the above quirk is probably not true.

Huh? How is that possible? TSX_CTRL has defined only bit 1 there, the CPUID enumeration bit, and bit 0 doesn't do any RTM disabling? Srsly?!

Skylake CPUs aren't getting TSX_CTRL, but force setting/clearing bits at boot will affect later logic.  (Unless I'm being blind while reading the patches, which is a distinct possibility).

tsx_init() will not force bits at boot on Skylake because of tsx_ctrl_is_supported() check.

Would adding this comment help?

I don't necessarily disagree, but the customers (who ultimately pay my salary) want late microcode loading and livepatching, so we've delivered.

Skylake CPUs aren't getting TSX_CTRL, but force setting/clearing bits at boot will affect later logic.  (Unless I'm being blind while reading the patches, which is a distinct possibility).

Quoted 2 lines · speck for Andrew Cooper

I don't necessarily disagree, but the customers (who ultimately pay my salary) want late microcode loading and livepatching, so we've delivered.

Yeah, you guys promised too much. How do you deal with userspace using a feature and you wanna upgrade microcode which disables it? TSX might not be a good example here

because feature bits disappearing is still ok,

it doesn't fault but it would simply start aborting transactions unconditionally but what if it is a CPU feature which userspace is actively using and it disappears underneath its feet all of a sudden?

Just upgrade the microcode and forget about it is not enough. I'm pretty sure you'll have to "dance". But hey, you can buy almost everything with money nowadays so... :-)

Quoted 4 lines

Skylake CPUs aren't getting TSX_CTRL, but force setting/clearing bits at boot will affect later logic.  (Unless I'm being blind while reading the patches, which is a distinct possibility).

Yes, that's why I'm saying we should not blindly force set and clear bits but mirror what CPUID is telling us. At least wrt TSX.

Its the perfect example here.  The answer is by requesting that Intel change bit 0's behaviour from causing #UD's to causing aborts.

The first version of this microcode was definitely not safe to late load.

Some userspace apparently gets confused when CPUID changes behind its back, which is why the CPUID control in bit 1 was split out from an otherwise monolithic bit 0.

At late load, choose (or not) to use bit 0 only. At boot, choose (or not) both bits 0 and 1 in unison.

Noone has guaranteed that all microcode ever in the future is going to be safe to use on a running system.  If it really can't be made to be safe, then customers are really going to have to reboot.

However, there is a lot of effort going into trying to make sure that fixes such as this one are made safe for late loading.

To give a concrete example, we have customers who's elapsed time for a reboot, conforming to SLAs, is in excess of 9 months, and new microcode with critical fixes is coming out faster than that.  I bet that I'm not the only person on this list with this type of customer.

arch/x86/kernel/cpu/tsx.c line 101

Quoted 9 lines · speck for Pawan Gupta

arch/x86/kernel/cpu/tsx.c new file +9 -0

+	if (tsx_ctrl_state == TSX_CTRL_DISABLE) {
+		tsx_disable();
+
+		/*
+		 * tsx_disable() will change the state of the
+		 * RTM CPUID bit.  Clear it here since it is now
+		 * expected to be not set.
+		 */
+		setup_clear_cpu_cap(X86_FEATURE_RTM);

This same argument applies to HLE, and it would be weird for pre-TSX_CTRL CPUs with tsx=off to report HLE but not RTM in /proc/cpuid

Furthermore, while grepping through the tree, I found

events/intel/lbr.c-267-static inline bool lbr_from_signext_quirk_needed(void)
events/intel/lbr.c-268-{
events/intel/lbr.c-269- int lbr_format = x86_pmu.intel_cap.lbr_format; events/intel/lbr.c:270: bool tsx_support = boot_cpu_has(X86_FEATURE_HLE) || events/intel/lbr.c-271-                    boot_cpu_has(X86_FEATURE_RTM); events/intel/lbr.c-272-
events/intel/lbr.c-273- return !tsx_support && (lbr_desc[lbr_format] & LBR_TSX);

which is going to need an adjustment to avoid applying the quirks on non-broken hardware.

From: Pawan Gupta <[email protected]>
Subject: [PATCH 3/9] x86/cpu: Add a "tsx=" cmdline option with TSX disabled by
default

Quoted 4 lines · speck for Pawan Gupta

From: Pawan Gupta <[email protected]>
Subject: [PATCH 3/9] x86/cpu: Add a "tsx=" cmdline option with TSX disabled by
default

As late at this is,

I really would like to change the name here. We
already have a powerpc command line option to disable this for those chips, and we should use the same name here as well.

I talked a bit
about this with some powerpc people and they really want to disable this option as well, much like is being done here, for lots of other, non-security reasons.

So can we please go back to the generic name so that we have some unification, making userspace stuff more uniform?

Quoted 4 lines · speck for Pawan Gupta

From: Pawan Gupta <[email protected]>
Subject: [PATCH 3/9] x86/cpu: Add a "tsx=" cmdline option with TSX disabled by
default

As late at this is, I really would like to change the name here. We already have a powerpc command line option to disable this for those chips, and we should use the same name here as well.

I talked a bit
about this with some powerpc people and they really want to disable this option as well, much like is being done here, for lots of other, non-security reasons.

So can we please go back to the generic name so that we have some unification, making userspace stuff more uniform?

Quoted 9 lines · speck for Greg KH

Quoted 4 lines · speck for Pawan Gupta

From: Pawan Gupta <[email protected]>
Subject: [PATCH 3/9] x86/cpu: Add a "tsx=" cmdline option with TSX disabled by
default

As late at this is, I really would like to change the name here. We already have a powerpc command line option to disable this for those chips, and we should use the same name here as well.

I think powerpc calls it "ppc_tm=". We can't obviously change to this.

	ppc_tm=		[PPC]
			Format: {"off"}
			Disable Hardware Transactional Memory

Quoted 13 lines · speck for Pawan Gupta

Quoted 9 lines · speck for Greg KH

Quoted 4 lines · speck for Pawan Gupta

From: Pawan Gupta <[email protected]>
Subject: [PATCH 3/9] x86/cpu: Add a "tsx=" cmdline option with TSX disabled by
default

As late at this is, I really would like to change the name here. We already have a powerpc command line option to disable this for those chips, and we should use the same name here as well.

I think powerpc calls it "ppc_tm=". We can't obviously change to this.

Quoted 3 lines

	ppc_tm=		[PPC]
			Format: {"off"}
			Disable Hardware Transactional Memory

No, but we can use "tm", and PPC said they would drop the "ppc_" prefix and use the generic flag as well. "Transactional Memory" is the generic term for this from what I can tell.

Quoted 18 lines · speck for Greg KH

Quoted 16 lines · speck for Pawan Gupta

Quoted 9 lines · speck for Greg KH

Quoted 4 lines · speck for Pawan Gupta

From: Pawan Gupta <[email protected]>
Subject: [PATCH 3/9] x86/cpu: Add a "tsx=" cmdline option with TSX disabled by
default

As late at this is, I really would like to change the name here. We already have a powerpc command line option to disable this for those chips, and we should use the same name here as well.

I think powerpc calls it "ppc_tm=". We can't obviously change to this.

	ppc_tm=		[PPC]
			Format: {"off"}
			Disable Hardware Transactional Memory

Quoted 3 lines

No, but we can use "tm", and PPC said they would drop the "ppc_" prefix and use the generic flag as well. "Transactional Memory" is the generic term for this from what I can tell.

Are those two (tsx and tm) going to have the same semantic? I have little doubts TBH considering how tsx itself is tricky.

Anyway, yes, TM shoudl have the same semantic. But, if it's too late at the moment, we can always unify this later on if PPC really wants it.

Quoted 2 lines · speck for Greg KH

Anyway, yes, TM shoudl have the same semantic. But, if it's too late at the moment, we can always unify this later on if PPC really wants it.

We can s/unify/add additional/ to minimize confusion. What I mean is, we can add "tm=" as an additional option which coexists with the "tsx=" option on x86 because the patches would have released already and we would have communicated "tsx=" to people by then. So "tsx=" and "tm=" would do the same thing on x86.

Quoted 23 lines · speck for Michal Hocko

Quoted 21 lines · speck for Greg KH

Quoted 11 lines · speck for Pawan Gupta

Quoted 9 lines · speck for Greg KH

Quoted 4 lines · speck for Pawan Gupta

From: Pawan Gupta <[email protected]>
Subject: [PATCH 3/9] x86/cpu: Add a "tsx=" cmdline option with TSX disabled by
default

As late at this is, I really would like to change the name here. We already have a powerpc command line option to disable this for those chips, and we should use the same name here as well.

Quoted 5 lines

I think powerpc calls it "ppc_tm=". We can't obviously change to this.

	ppc_tm=		[PPC]
			Format: {"off"}
			Disable Hardware Transactional Memory

No, but we can use "tm", and PPC said they would drop the "ppc_" prefix and use the generic flag as well. "Transactional Memory" is the generic term for this from what I can tell.

Quoted 2 lines

Are those two (tsx and tm) going to have the same semantic? I have little doubts TBH considering how tsx itself is tricky.

Ugh, turns out gmail was marking all of these threads as spam, that answers my "why is this list so quiet?" question :(

Anyway, yes, TM shoudl have the same semantic.

arch/x86/kernel/cpu/common.c +5 -0

index 885d4ac2111a..f8b8afc8f5b5 100644
@@ -1128,6 +1128,21 @@ static void __init cpu_set_bug_bits(struct cpuinfo_x86 *c)
 	if (!cpu_matches(NO_SWAPGS))
 		setup_force_cpu_bug(X86_BUG_SWAPGS);
 
+	/*
+	 * When the CPU is not mitigated for TAA (TAA_NO=0) set TAA bug when:
+	 *	- TSX is supported or
+	 *	- TSX_CTRL is present
+	 *

As I said before this would be a lot nicer if we could just add NO_TAA to the cpu_vuln_whitelist.

Quoted 2 lines · speck for Josh Poimboeuf

As I said before this would be a lot nicer if we could just add NO_TAA to the cpu_vuln_whitelist.

We're waiting for a list of CPUs from Intel here, right?

Quoted 5 lines · speck for Borislav Petkov

Quoted 2 lines · speck for Josh Poimboeuf

As I said before this would be a lot nicer if we could just add NO_TAA to the cpu_vuln_whitelist.

We're waiting for a list of CPUs from Intel here, right?

I guess so, unless somebody else can deduce that from existing information...

Quoted 9 lines · speck for Josh Poimboeuf

Quoted 5 lines · speck for Borislav Petkov

Quoted 2 lines · speck for Josh Poimboeuf

As I said before this would be a lot nicer if we could just add NO_TAA to the cpu_vuln_whitelist.

We're waiting for a list of CPUs from Intel here, right?

I guess so, unless somebody else can deduce that from existing information...

This is the list to the best of my knowledge.

+----------------------------+------------------+------------+
|            Name            |  Family / model  |  Stepping  |
+============================+==================+============+
| Whiskey Lake (ULT refresh) |      06_8E       |    0xC     |
+----------------------------+------------------+------------+
|    2nd gen Cascade Lake    |      06_55       |    6, 7    |
+----------------------------+------------------+------------+
|       Coffee Lake R        |      06_9E       |    0xD     |
+----------------------------+------------------+------------+

Quoted 9 lines · speck for Josh Poimboeuf

Quoted 5 lines · speck for Borislav Petkov

Quoted 2 lines · speck for Josh Poimboeuf

As I said before this would be a lot nicer if we could just add NO_TAA to the cpu_vuln_whitelist.

We're waiting for a list of CPUs from Intel here, right?

I guess so, unless somebody else can deduce that from existing information...

This is the list to the best of my knowledge.

+----------------------------+------------------+------------+
|            Name            |  Family / model  |  Stepping  |
+============================+==================+============+
| Whiskey Lake (ULT refresh) |      06_8E       |    0xC     |
+----------------------------+------------------+------------+
|    2nd gen Cascade Lake    |      06_55       |    6, 7    |
+----------------------------+------------------+------------+
|       Coffee Lake R        |      06_9E       |    0xD     |
+----------------------------+------------------+------------+

Quoted 2 lines · speck for Josh Poimboeuf

As I said before this would be a lot nicer if we could just add NO_TAA to the cpu_vuln_whitelist.

We're waiting for a list of CPUs from Intel here, right?

Quoted 5 lines · speck for Borislav Petkov

Quoted 2 lines · speck for Josh Poimboeuf

As I said before this would be a lot nicer if we could just add NO_TAA to the cpu_vuln_whitelist.

We're waiting for a list of CPUs from Intel here, right?

There is no model list required.  Vulnerability to TAA is calculable directly from existing architectural sources.

Quoted 7 lines · speck for Andrew Cooper

Quoted 5 lines · speck for Borislav Petkov

Quoted 2 lines · speck for Josh Poimboeuf

As I said before this would be a lot nicer if we could just add NO_TAA to the cpu_vuln_whitelist.

We're waiting for a list of CPUs from Intel here, right?

Quoted 2 lines

There is no model list required.  Vulnerability to TAA is calculable directly from existing architectural sources.

Can you elaborate? Earlier I suggested relying on NO_MDS in cpu_vuln_whitelist, but I believe you said that's not sufficient, because some of the non-MDS models don't have TSX, in which case we

shouldn't set TAA_BUG.

Which models are those?

Here's the current struct:

static const __initconst struct x86_cpu_id cpu_vuln_whitelist[] = {
VULNWL(ANY, 4, X86_MODEL_ANY, NO_SPECULATION),
VULNWL(CENTAUR, 5, X86_MODEL_ANY, NO_SPECULATION),
VULNWL(INTEL, 5, X86_MODEL_ANY, NO_SPECULATION),
VULNWL(NSC, 5, X86_MODEL_ANY, NO_SPECULATION),

/* Intel Family 6 */
VULNWL_INTEL(ATOM_SALTWELL, NO_SPECULATION),
VULNWL_INTEL(ATOM_SALTWELL_TABLET, NO_SPECULATION),
VULNWL_INTEL(ATOM_SALTWELL_MID, NO_SPECULATION),
VULNWL_INTEL(ATOM_BONNELL, NO_SPECULATION),
VULNWL_INTEL(ATOM_BONNELL_MID, NO_SPECULATION),

VULNWL_INTEL(ATOM_SILVERMONT, NO_SSB | NO_L1TF | MSBDS_ONLY | NO_SWAPGS),
VULNWL_INTEL(ATOM_SILVERMONT_D, NO_SSB | NO_L1TF | MSBDS_ONLY | NO_SWAPGS),
VULNWL_INTEL(ATOM_SILVERMONT_MID, NO_SSB | NO_L1TF | MSBDS_ONLY | NO_SWAPGS),
VULNWL_INTEL(ATOM_AIRMONT, NO_SSB | NO_L1TF | MSBDS_ONLY | NO_SWAPGS),
VULNWL_INTEL(XEON_PHI_KNL, NO_SSB | NO_L1TF | MSBDS_ONLY | NO_SWAPGS),
VULNWL_INTEL(XEON_PHI_KNM, NO_SSB | NO_L1TF | MSBDS_ONLY | NO_SWAPGS),

VULNWL_INTEL(CORE_YONAH, NO_SSB),

VULNWL_INTEL(ATOM_AIRMONT_MID, NO_L1TF | MSBDS_ONLY | NO_SWAPGS),
VULNWL_INTEL(ATOM_AIRMONT_NP, NO_L1TF | NO_SWAPGS),

VULNWL_INTEL(ATOM_GOLDMONT, NO_TAA | NO_MDS | NO_L1TF | NO_SWAPGS),
VULNWL_INTEL(ATOM_GOLDMONT_D, NO_TAA | NO_MDS | NO_L1TF | NO_SWAPGS),
VULNWL_INTEL(ATOM_GOLDMONT_PLUS, NO_TAA | NO_MDS | NO_L1TF | NO_SWAPGS),

/*
* Technically, swapgs isn't serializing on AMD (despite it previously
* being documented as such in the APM). But according to AMD, %gs is
* updated non-speculatively, and the issuing of %gs-relative memory
* operands will be blocked until the %gs update completes, which is
* good enough for our purposes.
*/

/* AMD Family 0xf - 0x12 */
VULNWL_AMD(0x0f, NO_MELTDOWN | NO_SSB | NO_L1TF | NO_MDS | NO_SWAPGS),
VULNWL_AMD(0x10, NO_MELTDOWN | NO_SSB | NO_L1TF | NO_MDS | NO_SWAPGS),
VULNWL_AMD(0x11, NO_MELTDOWN | NO_SSB | NO_L1TF | NO_MDS | NO_SWAPGS),
VULNWL_AMD(0x12, NO_MELTDOWN | NO_SSB | NO_L1TF | NO_MDS | NO_SWAPGS),

/* FAMILY_ANY must be last, otherwise 0x0f - 0x12 matches won't work */
VULNWL_AMD(X86_FAMILY_ANY, NO_MELTDOWN | NO_L1TF | NO_MDS | NO_SWAPGS),
VULNWL_HYGON(X86_FAMILY_ANY, NO_MELTDOWN | NO_L1TF | NO_MDS | NO_SWAPGS),
{}
};

Signature · 1 line

Josh

Quoted 6 lines · speck for Andrew Cooper

Quoted 5 lines · speck for Borislav Petkov

Quoted 2 lines · speck for Josh Poimboeuf

As I said before this would be a lot nicer if we could just add NO_TAA to the cpu_vuln_whitelist.

We're waiting for a list of CPUs from Intel here, right?

Quoted 2 lines

There is no model list required.  Vulnerability to TAA is calculable directly from existing architectural sources.

Can you elaborate? Earlier I suggested relying on NO_MDS in cpu_vuln_whitelist, but I believe you said that's not sufficient, because some of the non-MDS models don't have TSX, in which case we shouldn't set TAA_BUG.

Which models are those?

Quoted 7 lines · speck for Josh Poimboeuf

Quoted 6 lines · speck for Andrew Cooper

Quoted 5 lines · speck for Borislav Petkov

Quoted 2 lines · speck for Josh Poimboeuf

As I said before this would be a lot nicer if we could just add NO_TAA to the cpu_vuln_whitelist.

We're waiting for a list of CPUs from Intel here, right?

Quoted 8 lines

Quoted 2 lines

There is no model list required.  Vulnerability to TAA is calculable directly from existing architectural sources.

Can you elaborate? Earlier I suggested relying on NO_MDS in cpu_vuln_whitelist, but I believe you said that's not sufficient, because some of the non-MDS models don't have TSX, in which case we shouldn't set TAA_BUG.

Which models are those?

Ok.  First things first.  Do you (and by this, I really mean Linux) want to consider TAA an overlapping set with MDS, or a disjoint set?

After considering this for ages, and particularly, how to explain it clearly to non-experts in Xen's security advisory, I chose to go with this:

---8<---
Vulnerability to TAA is a little complicated to quantify.

In the pipeline, it is just another way to get speculative access to stale load port, store buffer or fill buffer data, and therefore can be considered a superset of MDS.  On parts which predate MDS_NO, the existing VERW flushing will mitigate this sidechannel as well.

On parts which contain MDS_NO, the lack of VERW flushing means that an attacker can still target microarchitectural buffers to leak secrets. Therefore, we consider TAA to be the set of parts which have MDS_NO but lack TAA_NO.
---8<---

The simplifying fact is that vulnerability to TAA doesn't matter on CPUs which don't advertise MDS_NO, because you're already doing VERW and disabling hyperthreading, *and* can't turn TSX off if it actually available.

People who were not taking MDS mitigations in the first place won't change their minds because of TAA, either.

Quoted 40 lines · speck for Andrew Cooper

Quoted 7 lines · speck for Josh Poimboeuf

Quoted 6 lines · speck for Andrew Cooper

Quoted 5 lines · speck for Borislav Petkov

Quoted 2 lines · speck for Josh Poimboeuf

As I said before this would be a lot nicer if we could just add NO_TAA to the cpu_vuln_whitelist.

We're waiting for a list of CPUs from Intel here, right?

Quoted 8 lines

Quoted 2 lines

There is no model list required.  Vulnerability to TAA is calculable directly from existing architectural sources.

Can you elaborate? Earlier I suggested relying on NO_MDS in cpu_vuln_whitelist, but I believe you said that's not sufficient, because some of the non-MDS models don't have TSX, in which case we shouldn't set TAA_BUG.

Which models are those?

Ok.  First things first.  Do you (and by this, I really mean Linux) want to consider TAA an overlapping set with MDS, or a disjoint set?

After considering this for ages, and particularly, how to explain it clearly to non-experts in Xen's security advisory, I chose to go with this:

---8<---
Vulnerability to TAA is a little complicated to quantify.

In the pipeline, it is just another way to get speculative access to stale load port, store buffer or fill buffer data, and therefore can be considered a superset of MDS.  On parts which predate MDS_NO, the existing VERW flushing will mitigate this sidechannel as well.

On parts which contain MDS_NO, the lack of VERW flushing means that an attacker can still target microarchitectural buffers to leak secrets. Therefore, we consider TAA to be the set of parts which have MDS_NO but lack TAA_NO.
---8<---

The simplifying fact is that vulnerability to TAA doesn't matter on CPUs which don't advertise MDS_NO, because you're already doing VERW and disabling hyperthreading, *and* can't turn TSX off if it actually available.

Quoted 2 lines

People who were not taking MDS mitigations in the first place won't change their minds because of TAA, either.

Good question.

The current Linux patches consider them overlapping. But it _might_ possibly be easier to communicate if we considered them disjoint. I don't know if there's a good answer, but at this point it might be easiest to stick with our current overlapping approach.

Quoted 45 lines · speck for Josh Poimboeuf

Quoted 36 lines · speck for Andrew Cooper

Quoted 7 lines · speck for Josh Poimboeuf

Quoted 6 lines · speck for Andrew Cooper

Quoted 5 lines · speck for Borislav Petkov

Quoted 2 lines · speck for Josh Poimboeuf

As I said before this would be a lot nicer if we could just add NO_TAA to the cpu_vuln_whitelist.

We're waiting for a list of CPUs from Intel here, right?

Quoted 8 lines

Quoted 2 lines

There is no model list required.  Vulnerability to TAA is calculable directly from existing architectural sources.

Can you elaborate? Earlier I suggested relying on NO_MDS in cpu_vuln_whitelist, but I believe you said that's not sufficient, because some of the non-MDS models don't have TSX, in which case we shouldn't set TAA_BUG.

Which models are those?

Ok.  First things first.  Do you (and by this, I really mean Linux) want to consider TAA an overlapping set with MDS, or a disjoint set?

After considering this for ages, and particularly, how to explain it clearly to non-experts in Xen's security advisory, I chose to go with this:

---8<---
Vulnerability to TAA is a little complicated to quantify.

In the pipeline, it is just another way to get speculative access to stale load port, store buffer or fill buffer data, and therefore can be considered a superset of MDS.  On parts which predate MDS_NO, the existing VERW flushing will mitigate this sidechannel as well.

On parts which contain MDS_NO, the lack of VERW flushing means that an attacker can still target microarchitectural buffers to leak secrets. Therefore, we consider TAA to be the set of parts which have MDS_NO but lack TAA_NO.
---8<---

Quoted 6 lines

The simplifying fact is that vulnerability to TAA doesn't matter on CPUs which don't advertise MDS_NO, because you're already doing VERW and disabling hyperthreading, *and* can't turn TSX off if it actually available.

People who were not taking MDS mitigations in the first place won't change their minds because of TAA, either.

Good question.

Quoted 5 lines

The current Linux patches consider them overlapping. But it _might_ possibly be easier to communicate if we considered them disjoint. I don't know if there's a good answer, but at this point it might be easiest to stick with our current overlapping approach.

I'll bring this up with the group.  I bet we are not the only people wondering the same, and it won't do any downstream users any good if they see conflicting descriptions from software vendors.

Quoted 6 lines · speck for Pawan Gupta

void cpu_bugs_smt_update(void)
{

(quoted diff) +0 -0

@@ -819,6 +916,19 @@ void cpu_bugs_smt_update(void)
 		break;
 	}
 

arch/x86/kernel/cpu/bugs.c void cpu_bugs_smt_update(void) · line 925

Quoted 7 lines

arch/x86/kernel/cpu/bugs.c +7 -0

+	switch (taa_mitigation) {
+	case TAA_MITIGATION_VERW:
+	case TAA_MITIGATION_UCODE_NEEDED:
+		if (sched_smt_active())
+			pr_warn_once(TAA_MSG_SMT);
+		/* TSX is enabled, apply MDS idle buffer clearing. */
+		update_mds_branch_idle();

With partial clearing removed, this call can go away.

From: Pawan Gupta <[email protected]>
Subject: [PATCH 5/9] x86/speculation/taa: Add sysfs reporting for TSX Async
Abort

Add the sysfs reporting file for TSX Async Abort. It exposes the vulnerability and the mitigation state similar to the existing files for the other hardware vulnerabilities.

Sysfs file path is:
/sys/devices/system/cpu/vulnerabilities/tsx_async_abort

Signed-off-by Pawan Gupta <[email protected]>

Signed-off-by Borislav Petkov <[email protected]>

Reviewed-by Mark Gross <[email protected]>

Reviewed-by Tony Luck <[email protected]>

Tested-by Neelima Krishnan <[email protected]>

Cc Greg Kroah-Hartman <[email protected]>

Cc "H. Peter Anvin" <[email protected]>

Cc Ingo Molnar <[email protected]>

Cc Jiri Kosina <[email protected]>

Cc Josh Poimboeuf <[email protected]>

Cc Peter Zijlstra <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc x86-ml <[email protected]>

---
 arch/x86/kernel/cpu/bugs.c | 23 +++++++++++++++++++++++
 drivers/base/cpu.c         |  9 +++++++++
 include/linux/cpu.h        |  3 +++
 3 files changed, 35 insertions(+)

From: Michal Hocko <[email protected]>
Subject: [PATCH 9/9] x86/tsx: Add config options to set tsx=on|off|auto

There is a general consensus that TSX usage is not largely spread while the history shows there is a non trivial space for side channel attacks possible. Therefore the tsx is disabled by default even on platforms that might have a safe implementation of TSX according to the current knowledge. This is a fair trade off to make.

There are, however, workloads that really do benefit from using TSX and updating to a newer kernel with TSX disabled might introduce a noticeable regressions. This would be especially a problem for Linux distributions which will provide TAA mitigations.

Introduce config options X86_INTEL_TSX_MODE_OFF, X86_INTEL_TSX_MODE_ON and X86_INTEL_TSX_MODE_AUTO to control the TSX feature. The config setting can be overridden by the tsx cmdline options.

Suggested-by Borislav Petkov <[email protected]>

Signed-off-by Michal Hocko <[email protected]>

Signed-off-by Pawan Gupta <[email protected]>

Signed-off-by Borislav Petkov <[email protected]>

Cc "H. Peter Anvin" <[email protected]>

Cc Ingo Molnar <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Tony Luck <[email protected]>

Cc x86-ml <[email protected]>

---
 arch/x86/Kconfig          | 45 +++++++++++++++++++++++++++++++++++++++
 arch/x86/kernel/cpu/tsx.c | 22 +++++++++++++------
 2 files changed, 61 insertions(+), 6 deletions(-)

arch/x86/Kconfig +8 -0

index d6e1faa28c58..eebae89726c4 100644
@@ -1940,6 +1940,51 @@ config X86_INTEL_MEMORY_PROTECTION_KEYS
 
 	  If unsure, say y.
 
+choice
+	prompt "TSX enable mode"
+	depends on CPU_SUP_INTEL
+	default X86_INTEL_TSX_MODE_OFF
+	help
+	  Intel's TSX (Transactional Synchronization Extensions) feature
+	  allows to optimize locking protocols through lock elision which
+	  can lead to a noticeable performance boost.

I think this is misleading. tsx=on doesn't make you vulnerable to TAA, because we still the TAA mitigation.

tsx=on vs tsx=auto is not a security consideration, but rather a performance one. With tsx=auto you disable TSX on some TAA-affected CPUs so you don't have to pay the performance penalty of the MDS mitigations.

arch/x86/Kconfig config X86_INTEL_MEMORY_PROTECTION_KEYS · line 1973

Quoted 6 lines

arch/x86/Kconfig +5 -0

+
+config X86_INTEL_TSX_MODE_OFF
+	bool "off"
+	help
+	  TSX is always disabled - equals tsx=off command line parameter.

Define "always" :-)

Not exactly :-) This also leaves TSX enabled on MDS vulnerable parts.

Quoted 2 lines · speck for Josh Poimboeuf

I think this is misleading. tsx=on doesn't make you vulnerable to TAA, because we still the TAA mitigation.

Changed to:

Therefore TSX is not enabled by default (aka tsx=off). An admin might override this decision by tsx=on the command line parameter. Even with TSX enabled, the kernel will attempt to enable the best possible TAA mitigation setting depending on the microcode available for the particular machine.

Quoted 4 lines

tsx=on vs tsx=auto is not a security consideration, but rather a performance one. With tsx=auto you disable TSX on some TAA-affected CPUs so you don't have to pay the performance penalty of the MDS mitigations.

By performance penalty you mean, when you have TSX disabled on those parts, you'll save yourself the VERW which should be taking care of TAA too?

Quoted 8 lines

arch/x86/Kconfig config X86_INTEL_MEMORY_PROTECTION_KEYS · line 1973

Quoted 6 lines

arch/x86/Kconfig +5 -0

+
+config X86_INTEL_TSX_MODE_OFF
+	bool "off"
+	help
+	  TSX is always disabled - equals tsx=off command line parameter.

Define "always" :-)

Changed to:

"TSX is disabled if possible - equals to tsx=off command line parameter."

Quoted 2 lines · speck for Josh Poimboeuf

I think this is misleading. tsx=on doesn't make you vulnerable to TAA, because we still the TAA mitigation.

Changed to:

Therefore TSX is not enabled by default (aka tsx=off). An admin might override this decision by tsx=on the command line parameter. Even with TSX enabled, the kernel will attempt to enable the best possible TAA mitigation setting depending on the microcode available for the particular machine.

Quoted 4 lines

tsx=on vs tsx=auto is not a security consideration, but rather a performance one. With tsx=auto you disable TSX on some TAA-affected CPUs so you don't have to pay the performance penalty of the MDS mitigations.

By performance penalty you mean, when you have TSX disabled on those parts, you'll save yourself the VERW which should be taking care of TAA too?

Quoted 6 lines

arch/x86/Kconfig config X86_INTEL_MEMORY_PROTECTION_KEYS · line 1973

Quoted 4 lines

arch/x86/Kconfig +4 -0

+config X86_INTEL_TSX_MODE_OFF
+	bool "off"
+	help
+	  TSX is always disabled - equals tsx=off command line parameter.

Define "always" :-)

Changed to:

"TSX is disabled if possible - equals to tsx=off command line parameter."

Quoted 1 line

Not exactly :-) This also leaves TSX enabled on MDS vulnerable parts.

Your point being, the MD_CLEAR which takes care of TAA too?

Documentation/admin-guide/kernel-parameters.txt +4 -0

+				  otherwise enable TSX on the system.
+
+			More details on X86_BUG_TAA here:
+			Documentation/admin-guide/hw-vuln/tsx_async_abort.rst
 
 			Not specifying this option is equivalent to tsx=off.

Quoted 6 lines · speck for Pawan Gupta

Documentation/admin-guide/kernel-parameters.txt +0 -0

index ad6b69057bb0..e1aca10f2a7f 100644
@@ -4856,6 +4856,11 @@
 

Documentation/admin-guide/kernel-parameters.txt line 4859

Quoted 3 lines

Documentation/admin-guide/kernel-parameters.txt +1 -0

 			on	- Enable TSX on the system.
 			off	- Disable TSX on the system.
+			auto	- Disable TSX if X86_BUG_TAA is present,

Not quite true, since MDS-affected parts don't have the new microcode bits to disable TSX.

"Disable TSX if the CPU is affected by the TSX Async Abort (TAA) vulnerability and microcode provides a special MSR - TSX_CTRL_MSR - which provides the required TSX control knobs. On MDS-affected parts where VERW takes care of the TAA vulnerability, that controlling MSR is not present and thus TSX cannot be disabled there."

Quoted 5 lines · speck for Borislav Petkov

"Disable TSX if the CPU is affected by the TSX Async Abort (TAA) vulnerability and microcode provides a special MSR - TSX_CTRL_MSR - which provides the required TSX control knobs. On MDS-affected parts where VERW takes care of the TAA vulnerability, that controlling MSR is not present and thus TSX cannot be disabled there."

This is true if you ignore hyperthreading.

On SMT systems, TSX disable is 100% complete mitigation, while VERW clearing is not.

Quoted 8 lines · speck for Jiri Kosina

Quoted 5 lines · speck for Borislav Petkov

"Disable TSX if the CPU is affected by the TSX Async Abort (TAA) vulnerability and microcode provides a special MSR - TSX_CTRL_MSR - which provides the required TSX control knobs. On MDS-affected parts where VERW takes care of the TAA vulnerability, that controlling MSR is not present and thus TSX cannot be disabled there."

Quoted 4 lines

This is true if you ignore hyperthreading.

On SMT systems, TSX disable is 100% complete mitigation, while VERW clearing is not.

So why is our default this then?

static enum taa_mitigations taa_mitigation __ro_after_init = TAA_MITIGATION_VERW;

and we only do the TAA_MITIGATION_TSX_DISABLED thing only if TSX has been disabled earlier?

Because of those MDS_NO=0 machines which don't get the TSX_CTRL MSR so that TSX cannot be disabled there?

Are some of those machines SMT?

Because if so, we *must* disable SMT unconditionally to mitigate TAA completely there... methinks.

For the same reasons we kept SMT enabled for L1TF and MDS... We don't want to break existing SMT workloads.

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst new file +32 -0

+
+In all other cases, if the host either does not have the TAA microcode or the
+kernel is not mitigated, the system might be vulnerable.
+
+
+.. _taa_mitigation_control_command_line:
+
+Mitigation control on the kernel command line
+---------------------------------------------
+
+The kernel command line allows to control the TAA mitigations at boot time with
+the option "tsx_async_abort=". The valid arguments for this option are:
+
+  ============  =============================================================
+  off		This option disables the TAA mitigation on affected platforms.
+                If the system has TSX enabled (see next parameter) and the CPU
+                is affected, the system is vulnerable.
+
+  full	        TAA mitigation is enabled. If TSX is enabled, on an affected
+                system it will clear CPU buffers on ring transitions. On
+                systems which are MDS-affected and deploy MDS mitigation,
+                TAA is also mitigated. Specifying this option on those
+                systems will have no effect.
+
+  full,nosmt    The same as tsx_async_abort=full, with SMT disabled on
+                vulnerable CPUs that have TSX enabled. This is the complete
+                mitigation. When TSX is disabled, SMT is not disabled because
+                CPU is not vulnerable to cross-thread TAA attacks.
+  ============  =============================================================
+
+Not specifying this option is equivalent to "tsx_async_abort=full".
+

+ on Enables TSX.

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst new file +11 -0

+    1          X            1        MDS fixed, TAA can be mitigated by
+                                     VERW or TSX_CTRL_MSR
+  =======  =========  =============  ========================================
+
+Mitigation selection guide
+--------------------------
+
+1. Trusted userspace and guests
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If all user space applications are from a trusted source and do not execute

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst new file +10 -0

+
+
+.. _taa_default_mitigations:
+
+Default mitigations
+-------------------
+
+The kernel's default action for vulnerable processors is:
+
+  - Deploy TSX disable mitigation (tsx_async_abort=full tsx=off).

Documentation/admin-guide/kernel-parameters.txt +11 -0

index e1aca10f2a7f..6eb1c0c8018c 100644
@@ -2636,6 +2636,7 @@
 					       ssbd=force-off [ARM64]
 					       l1tf=off [X86]
 					       mds=off [X86]
+					       tsx_async_abort=off [X86]
 
 			auto (default)
 				Mitigate all CPU vulnerabilities, but leave SMT

@@ -2651,6 +2652,7 @@
 				be fully mitigated, even if it means losing SMT.
 				Equivalent to: l1tf=flush,nosmt [X86]
 					       mds=full,nosmt [X86]
+					       tsx_async_abort=full,nosmt [X86]
 
 	mminit_loglevel=
 			[KNL] When CONFIG_DEBUG_MEMORY_INIT is set, this

@@ -4864,6 +4866,40 @@
 
 			Not specifying this option is equivalent to tsx=off.
 
+	tsx_async_abort= [X86,INTEL] Control mitigation for the TSX Async
+			Abort (TAA) vulnerability.
+
+			Similar to Micro-architectural Data Sampling (MDS)
+			certain CPUs that support Transactional
+			Synchronization Extensions (TSX) are vulnerable to an
+			exploit against CPU internal buffers which can forward
+			information to a disclosure gadget under certain
+			conditions.

Documentation/admin-guide/kernel-parameters.txt +15 -0

+			full,nosmt - Enable TAA mitigation and disable SMT on
+				     vulnerable CPUs. If TSX is disabled, SMT
+				     is not disabled because CPU is not
+				     vulnerable to cross-thread TAA attacks.
+			off        - Unconditionally disable TAA mitigation
+
+			Not specifying this option is equivalent to
+			tsx_async_abort=full.  On CPUs which are MDS affected
+			and deploy MDS mitigation, TAA mitigation is not
+			required and doesn't provide any additional
+			mitigation.
+
+			For details see:
+			Documentation/admin-guide/hw-vuln/tsx_async_abort.rst
+
 	turbografx.map[2|3]=	[HW,JOY]
 			TurboGraFX parallel port interface
 			Format:

Documentation/x86/index.rst +1 -0

index af64c4bb4447..a8de2fbc1caa 100644
@@ -27,6 +27,7 @@ x86-specific Documentation
    mds
    microcode
    resctrl_ui
+   tsx_async_abort
    usb-legacy-support
    i386/index
    x86_64/index

Documentation/x86/tsx_async_abort.rst new file +15 -0

new file mode 100644
index 000000000000..583ddc185ba2
@@ -0,0 +1,117 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+TSX Async Abort (TAA) mitigation
+================================
+
+.. _tsx_async_abort:
+
+Overview
+--------
+
+TSX Async Abort (TAA) is a side channel attack on internal buffers in some
+Intel processors similar to Microachitectural Data Sampling (MDS).  In this
+case certain loads may speculatively pass invalid data to dependent operations
+when an asynchronous abort condition is pending in a Transactional
+Synchronization Extensions (TSX) transaction.  This includes loads with no

Documentation/x86/tsx_async_abort.rst new file +28 -0

+
+Kernel internal mitigation modes
+--------------------------------
+
+ =============    ============================================================
+ off              Mitigation is disabled. Either the CPU is not affected or
+                  tsx_async_abort=off is supplied on the kernel command line.
+
+ tsx disabled     Mitigation is enabled. TSX feature is disabled by default at
+                  bootup on processors that support TSX control.
+
+ verw             Mitigation is enabled. CPU is affected and MD_CLEAR is
+                  advertised in CPUID.
+
+ ucode needed     Mitigation is enabled. CPU is affected and MD_CLEAR is not
+                  advertised in CPUID. That is mainly for virtualization
+                  scenarios where the host has the updated microcode but the
+                  hypervisor does not expose MD_CLEAR in CPUID. It's a best
+                  effort approach without guarantee.
+ =============    ============================================================
+
+If the CPU is affected and the "tsx_async_abort" kernel command line parameter is
+not provided then the kernel selects an appropriate mitigation depending on the
+status of RTM and MD_CLEAR CPUID bits.
+
+Below tables indicate the impact of tsx=on|off|auto cmdline options on state of
+TAA mitigation, VERW behavior and TSX feature for various combinations of
+MSR_IA32_ARCH_CAPABILITIES bits.

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst line 145

Quoted 5 lines · speck for Pawan Gupta

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst new file +5 -0

+Virtualization mitigation
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Affected systems where the host has the TAA microcode and the TAA mitigation is
+ON (with TSX disabled) are not vulnerable regardless of the status of the VMs.

This is is confusing: "the TAA mitigation is ON (with TSX disabled)".

Which is it? Is the TAA mitigation on, or is TSX disabled?

This is not universally true.

Quoted 2 lines

+ on Enables TSX.

This probably needs the same "TSX is fundamentally insecure" caveat I proposed for kernel-parameters.txt.

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst line 190

Quoted 3 lines

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst new file +2 -0

+
+  auto		Disables TSX on affected platform, otherwise enables TSX.

This is not universally true.

Also, it would be relevant to refer to that table Pawan posted which shows exactly which CPUs are vulnerable to TAA but not MDS.

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst line 200

Quoted 11 lines

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst new file +10 -0

+  ============  =============================================================
+
+Not specifying this option is equivalent to "tsx=off".
+
+The following combinations of the "tsx_async_abort" and "tsx" are possible. For
+affected platforms tsx=auto is equivalent to tsx=off and the result will be:
+
+  =========  ====================   =========================================
+  tsx=on     tsx_async_abort=full   The system will use VERW to clear CPU
+                                    buffers.

The system may still be vulnerable to SMT-based attacks.

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst line 204

Quoted 4 lines

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst new file +4 -0

+  tsx=on     tsx_async_abort=off    The system is vulnerable.
+  tsx=off    tsx_async_abort=full   TSX is disabled. System is not vulnerable.
+  tsx=off    tsx_async_abort=off    TSX is disabled. System is not vulnerable.
+  =========  ====================   =========================================

Combinations with tsx_async_abort=full,nosmt should also be described.

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst line 208

Quoted 3 lines

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst new file +3 -0

+For unaffected platforms "tsx=on" and "tsx_async_abort=full" does not clear CPU
+buffers.  For platforms without TSX control "tsx" command line argument has no
+effect.

Which platforms are those?

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst line 221

Quoted 13 lines

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst new file +12 -0

+For the affected platforms below table indicates the mitigation status for the
+combinations of CPUID bit MD_CLEAR and IA32_ARCH_CAPABILITIES MSR bits MDS_NO
+and TSX_CTRL_MSR.
+
+  =======  =========  =============  ========================================
+  MDS_NO   MD_CLEAR   TSX_CTRL_MSR   Status
+  =======  =========  =============  ========================================
+    0          0            0        Vulnerable (needs ucode)
+    0          1            0        MDS and TAA mitigated via VERW
+    1          1            0        MDS fixed, TAA vulnerable if TSX enabled
+                                     because MD_CLEAR has no meaning and
+                                     VERW is not guaranteed to clear buffers

(needs ucode) ?

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst line 242

Quoted 6 lines

+2. Untrusted userspace and guests
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst new file +4 -0

+
+If there are untrusted applications or guests on the system, enabling TSX
+might allow a malicious actor to leak data from the host or from other
+processes running on the same physical core.

Unless the mitigation is enabled (which is on by default, BTW...)

This makes it sounds like the only mitigation is to disable TSX.

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst line 246

Quoted 3 lines

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst new file +3 -0

+If the microcode is available and the TSX is disabled on the host, attacks
+are prevented in a virtualized environment as well, even if the VMs do not
+explicitly enable the mitigation.

What's the effect on VM security if TSX is enabled and the host TAA mitigation is also enabled?

Documentation/admin-guide/kernel-parameters.txt line 4887

Quoted 4 lines

Documentation/admin-guide/kernel-parameters.txt +4 -0

+			This parameter controls the TAA mitigation.  The
+			options are:
+
+			full       - Enable TAA mitigation on vulnerable CPUs

if TSX is disabled

Documentation/x86/tsx_async_abort.rst line 25

Quoted 5 lines

Documentation/x86/tsx_async_abort.rst new file +5 -0

+Mitigation strategy
+-------------------
+
+a) TSX disable - one of the mitigations is to disable TSX. A new MSR
+IA32_TSX_CTRL will be available in future and current processors after

which processors?

Documentation/x86/tsx_async_abort.rst line 31

Quoted 7 lines

Documentation/x86/tsx_async_abort.rst new file +6 -0

+microcode update which can be used to disable TSX. In addition, it
+controls the enumeration of the TSX feature bits (RTM and HLE) in CPUID.
+
+b) Clear CPU buffers - similar to MDS, clearing the CPU buffers mitigates this
+vulnerability. More details on this approach can be found in
+:ref:`Documentation/admin-guide/hw-vuln/mds.rst <mds>`.

It should be clarified the mitigation is a) OR b), not both.

Documentation/x86/tsx_async_abort.rst line 69

Quoted 9 lines

+1. "tsx=off"

Documentation/x86/tsx_async_abort.rst new file +8 -0

+
+=========  =========  ============  ============  ==============  ===================  ======================
+MSR_IA32_ARCH_CAPABILITIES bits     Result with cmdline tsx=off
+----------------------------------  -------------------------------------------------------------------------
+TAA_NO     MDS_NO     TSX_CTRL_MSR  TSX state     VERW can clear  TAA mitigation       TAA mitigation
+                                    after bootup  CPU buffers     tsx_async_abort=off  tsx_async_abort=full
+=========  =========  ============  ============  ==============  ===================  ======================
+    0          0           0         HW default         Yes           Same as MDS           Same as MDS

Does "HW default" mean "Enabled"?

Quoted 10 lines · speck for Josh Poimboeuf

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst line 145

Quoted 5 lines · speck for Pawan Gupta

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst new file +5 -0

+Virtualization mitigation
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Affected systems where the host has the TAA microcode and the TAA mitigation is
+ON (with TSX disabled) are not vulnerable regardless of the status of the VMs.

This is is confusing: "the TAA mitigation is ON (with TSX disabled)".

Which is it? Is the TAA mitigation on, or is TSX disabled?

I think it wants to say:

"Affected systems where the host has TAA microcode and TAA is mitigated by having disabled TSX, are not vulnerable..."

Quoted 1 line

This is not universally true.

"Disables TSX on systems which have the TSX-disable MSR."

Quoted 4 lines

Quoted 1 line

+ on Enables TSX.

This probably needs the same "TSX is fundamentally insecure" caveat I proposed for kernel-parameters.txt.

I'm great at copy'n'paste:

"Enables TSX.

Although there are mitigations for all known security vulnerabilities, TSX has been known to be an accelerator for several previous speculation-related CVEs, and so there may be unknown security risks associated with leaving it enabled."

Quoted 5 lines

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst line 190

Quoted 3 lines

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst new file +2 -0

+
+  auto		Disables TSX on affected platform, otherwise enables TSX.

This is not universally true.

Quoted 2 lines

Also, it would be relevant to refer to that table Pawan posted which shows exactly which CPUs are vulnerable to TAA but not MDS.

This one?

I don't know whether this one will get more models later or what the TAA_NO plan is...

+----------------------------+------------------+------------+
|            Name            |  Family / model  |  Stepping  |
+============================+==================+============+
| Whiskey Lake (ULT refresh) |      06_8E       |    0xC     |
+----------------------------+------------------+------------+
|    2nd gen Cascade Lake    |      06_55       |    6, 7    |
+----------------------------+------------------+------------+
|       Coffee Lake R        |      06_9E       |    0xD     |
+----------------------------+------------------+------------+

Quoted 13 lines

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst line 200

Quoted 10 lines

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst new file +10 -0

+  ============  =============================================================
+
+Not specifying this option is equivalent to "tsx=off".
+
+The following combinations of the "tsx_async_abort" and "tsx" are possible. For
+affected platforms tsx=auto is equivalent to tsx=off and the result will be:
+
+  =========  ====================   =========================================
+  tsx=on     tsx_async_abort=full   The system will use VERW to clear CPU
+                                    buffers.

The system may still be vulnerable to SMT-based attacks.

Quoted 6 lines

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst line 204

Quoted 4 lines

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst new file +4 -0

+  tsx=on     tsx_async_abort=off    The system is vulnerable.
+  tsx=off    tsx_async_abort=full   TSX is disabled. System is not vulnerable.
+  tsx=off    tsx_async_abort=off    TSX is disabled. System is not vulnerable.
+  =========  ====================   =========================================

Combinations with tsx_async_abort=full,nosmt should also be described.

  =========  ====================   =========================================
  tsx=on     tsx_async_abort=full   The system will use VERW to clear CPU
                                    buffers. Cross-thread attacks still possible
                                    on SMT machines.
  tsx=on     tsx_async_abort=full,nosmt As above, cross-thread attacks on SMT
                                    mitigated.
  tsx=on     tsx_async_abort=off    The system is vulnerable.
  tsx=off    tsx_async_abort=full   TSX is disabled. System is not vulnerable.

  tsx=off    tsx_async_abort=full,nosmt ditto
  tsx=off    tsx_async_abort=off    ditto
  =========  ====================   =========================================

Quoted 5 lines

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst line 208

Quoted 3 lines

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst new file +3 -0

+For unaffected platforms "tsx=on" and "tsx_async_abort=full" does not clear CPU
+buffers.  For platforms without TSX control "tsx" command line argument has no
+effect.

Which platforms are those?

The MDS_NO=0 ones.

Quoted 16 lines

Quoted 4 lines

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst new file +2 -0

+For the affected platforms below table indicates the mitigation status for the
+combinations of CPUID bit MD_CLEAR and IA32_ARCH_CAPABILITIES MSR bits MDS_NO

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst line 221

Quoted 10 lines

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst new file +10 -0

+and TSX_CTRL_MSR.
+
+  =======  =========  =============  ========================================
+  MDS_NO   MD_CLEAR   TSX_CTRL_MSR   Status
+  =======  =========  =============  ========================================
+    0          0            0        Vulnerable (needs ucode)
+    0          1            0        MDS and TAA mitigated via VERW
+    1          1            0        MDS fixed, TAA vulnerable if TSX enabled
+                                     because MD_CLEAR has no meaning and
+                                     VERW is not guaranteed to clear buffers

(needs ucode) ?

Will there even be microcode for those to beef up VERW?

Quoted 11 lines

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst line 242

Quoted 7 lines

+2. Untrusted userspace and guests
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst new file +4 -0

+
+If there are untrusted applications or guests on the system, enabling TSX
+might allow a malicious actor to leak data from the host or from other
+processes running on the same physical core.

Unless the mitigation is enabled (which is on by default, BTW...)

This makes it sounds like the only mitigation is to disable TSX.

"If there are untrusted applications or guests on the system, enabling TSX and the inability to enable the TAA mitigation (missing microcode for a particular model, etc.) might allow a malicious actor to leak data from the host or from other processes running on the same physical core."

Quoted 6 lines

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst line 246

Quoted 3 lines

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst new file +3 -0

+If the microcode is available and the TSX is disabled on the host, attacks
+are prevented in a virtualized environment as well, even if the VMs do not
+explicitly enable the mitigation.

What's the effect on VM security if TSX is enabled and the host TAA mitigation is also enabled?

Same as in the !VM case, I'd assume. tsx_async_abort=full,nosmt should give you full mitigation.

Quoted 6 lines

Documentation/admin-guide/kernel-parameters.txt line 4887

Quoted 4 lines

Documentation/admin-guide/kernel-parameters.txt +4 -0

+			This parameter controls the TAA mitigation.  The
+			options are:
+
+			full       - Enable TAA mitigation on vulnerable CPUs

if TSX is disabled

No, that's TAA_MITIGATION_VERW which does the buffer clearing.

Quoted 7 lines

Documentation/x86/tsx_async_abort.rst line 25

Quoted 5 lines

Documentation/x86/tsx_async_abort.rst new file +5 -0

+Mitigation strategy
+-------------------
+
+a) TSX disable - one of the mitigations is to disable TSX. A new MSR
+IA32_TSX_CTRL will be available in future and current processors after

which processors?

The MDS_NO=1 and future parts, I guess.

Quoted 15 lines

Quoted 14 lines

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst line 221

Quoted 12 lines

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst new file +12 -0

+For the affected platforms below table indicates the mitigation status for the
+combinations of CPUID bit MD_CLEAR and IA32_ARCH_CAPABILITIES MSR bits MDS_NO
+and TSX_CTRL_MSR.
+
+  =======  =========  =============  ========================================
+  MDS_NO   MD_CLEAR   TSX_CTRL_MSR   Status
+  =======  =========  =============  ========================================
+    0          0            0        Vulnerable (needs ucode)
+    0          1            0        MDS and TAA mitigated via VERW
+    1          1            0        MDS fixed, TAA vulnerable if TSX enabled
+                                     because MD_CLEAR has no meaning and
+                                     VERW is not guaranteed to clear buffers

(needs ucode) ?

Quoted 1 line

Will there even be microcode for those to beef up VERW?

This might be a question for Intel, but I assumed this is the case where the new microcode on the MDS_NO parts would enable the VERW buffer clearing.

Quoted 7 lines

Quoted 6 lines

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst line 246

Quoted 3 lines

Documentation/admin-guide/hw-vuln/tsx_async_abort.rst new file +3 -0

+If the microcode is available and the TSX is disabled on the host, attacks
+are prevented in a virtualized environment as well, even if the VMs do not
+explicitly enable the mitigation.

What's the effect on VM security if TSX is enabled and the host TAA mitigation is also enabled?

Quoted 2 lines

Same as in the !VM case, I'd assume. tsx_async_abort=full,nosmt should give you full mitigation.

Right, the effects of the host mitigation options on the guest would be useful here.

Quoted 9 lines

Quoted 7 lines

Documentation/x86/tsx_async_abort.rst line 25

Quoted 5 lines

Documentation/x86/tsx_async_abort.rst new file +5 -0

+Mitigation strategy
+-------------------
+
+a) TSX disable - one of the mitigations is to disable TSX. A new MSR
+IA32_TSX_CTRL will be available in future and current processors after

which processors?

The MDS_NO=1 and future parts, I guess.

Right, that should be clarified.

Ditto, I guess whatever wording is used in kernel-parameters.txt can be cribbed for here.

That's the one, but yeah... it might not be future-complete, and even if it is, Intel probably doesn't want to disclose that anyway.

TSX _might_ be disabled, depending...

Right, update the wording?

Right, the effects of the host mitigation options on the guest would be useful here.

I meant to say "if TSX is enabled". Otherwise if TSX is disabled this option doesn't do anything

Quoted 1 line

The MDS_NO=1 and future parts, I guess.

Right, that should be clarified.