Skip to contentSkip to conversation

x86/tsx: fix KVM guest live migration for tsx=on

PATCH Superseded Jon Kohler 3 participants 14 messages Apr 11, 2022 – Apr 13, 2022

Subsystems (2)
lore
Share & tools
lore

Jon Kohler

author
#0 lore raw

PATCH 1/1 x86/tsx: fix KVM guest live migration for tsx=on

Move automatic disablement for TSX microcode deprecation from tsx_init() to x86_get_tsx_auto_mode(), such that systems with tsx=on will continue to see the TSX CPU features (HLE, RTM) even on updated microcode.

KVM live migration could be possibly be broken in 5.14+ commit 293649307ef9 ("x86/tsx: Clear CPUID bits when TSX always force aborts"). Consider the following scenario:

1. KVM hosts clustered in a live migration capable setup.
2. KVM guests have TSX CPU features HLE and/or RTM presented.
3. One of the three maintenance events occur:
3a. An existing host running kernel >= 5.14 in the pool updated with the
new microcode.
3b. A new host running kernel >= 5.14 is commissioned that already has the
microcode update preloaded.
3c. All hosts are running kernel < 5.14 with microcode update already
loaded and one existing host gets updated to kernel >= 5.14.
4. After maintenance event, the impacted host will not have HLE and RTM exposed, and live migrations with guests with TSX features might not migrate.

Users using tsx=on or CONFIG_X86_INTEL_TSX_MODE_ON should always see HLE and RTM on capable Intel SKUs, even if microcode has been clubbed to prevent functionality.

Users using tsx=auto get or CONFIG_X86_INTEL_TSX_MODE_AUTO get to roll the dice with whatever the kernel believes the appropriate default is, which includes the feature disappearing after a kernel and/or microcode update. These users should consider masking HLE and RTM at a higher control plane level, e.g. qemu or libvirt, such that guests on TSX enabled systems do not see HLE/RTM and therefore do not enable TAA mitigation.

Fixes 293649307ef9 ("x86/tsx: Clear CPUID bits when TSX always force aborts")
Cc Pawan Gupta <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Tony Luck <[email protected]>
Cc Neelima Krishnan <[email protected]>
---
 arch/x86/kernel/cpu/tsx.c | 29 ++++++++++++++---------------
 1 file changed, 14 insertions(+), 15 deletions(-)
arch/x86/kernel/cpu/tsx.c +14 -15
index 9c7a5f049292..a24e5e471e3f 100644
@@ -78,6 +78,20 @@ static bool __init tsx_ctrl_is_supported(void)
 
 static enum tsx_ctrl_states x86_get_tsx_auto_mode(void)
 {
+	/*
+	 * Hardware will always abort a TSX transaction if both CPUID bits
+	 * RTM_ALWAYS_ABORT and TSX_FORCE_ABORT are set. In this case, it is
+	 * better not to enumerate CPUID.RTM and CPUID.HLE bits. Clear them
+	 * here.
+	 */
+	if (boot_cpu_has(X86_FEATURE_RTM_ALWAYS_ABORT) &&
+	    boot_cpu_has(X86_FEATURE_TSX_FORCE_ABORT)) {
+		tsx_clear_cpuid();
+		setup_clear_cpu_cap(X86_FEATURE_RTM);
+		setup_clear_cpu_cap(X86_FEATURE_HLE);
+		return TSX_CTRL_RTM_ALWAYS_ABORT;
+	}
+
 	if (boot_cpu_has_bug(X86_BUG_TAA))
 		return TSX_CTRL_DISABLE;
 
@@ -105,21 +119,6 @@ void __init tsx_init(void)
 	char arg[5] = {};
 	int ret;
 
-	/*
-	 * Hardware will always abort a TSX transaction if both CPUID bits
-	 * RTM_ALWAYS_ABORT and TSX_FORCE_ABORT are set. In this case, it is
-	 * better not to enumerate CPUID.RTM and CPUID.HLE bits. Clear them
-	 * here.
-	 */
-	if (boot_cpu_has(X86_FEATURE_RTM_ALWAYS_ABORT) &&
-	    boot_cpu_has(X86_FEATURE_TSX_FORCE_ABORT)) {
-		tsx_ctrl_state = TSX_CTRL_RTM_ALWAYS_ABORT;
-		tsx_clear_cpuid();
-		setup_clear_cpu_cap(X86_FEATURE_RTM);
-		setup_clear_cpu_cap(X86_FEATURE_HLE);
-		return;
-	}
-
 	if (!tsx_ctrl_is_supported()) {
 		tsx_ctrl_state = TSX_CTRL_NOT_SUPPORTED;
 		return;
Signature · 1 line
2.30.1 (Apple Git-130)
#1 lore raw
Quoted 5 lines · Jon Kohler

static enum tsx_ctrl_states x86_get_tsx_auto_mode(void) {

arch/x86/kernel/cpu/tsx.c +3 -0
+	/*
+	 * Hardware will always abort a TSX transaction if both CPUID bits
+	 * RTM_ALWAYS_ABORT and TSX_FORCE_ABORT are set. In this case, it is

arch/x86/kernel/cpu/tsx.c static bool __init tsx_ctrl_is_supported(void) · line 93

Quoted 10 lines
arch/x86/kernel/cpu/tsx.c +10 -0
+	 * better not to enumerate CPUID.RTM and CPUID.HLE bits. Clear them
+	 * here.
+	 */
+	if (boot_cpu_has(X86_FEATURE_RTM_ALWAYS_ABORT) &&
+	    boot_cpu_has(X86_FEATURE_TSX_FORCE_ABORT)) {
+		tsx_clear_cpuid();
+		setup_clear_cpu_cap(X86_FEATURE_RTM);
+		setup_clear_cpu_cap(X86_FEATURE_HLE);
+		return TSX_CTRL_RTM_ALWAYS_ABORT;
+	}

I don't really like hiding the setup_clear_cpu_cap() like this. Right now, all of the setup_clear_cpu_cap()'s are in a single function and they are pretty easy to figure out.

This seems like logic that deserves to be appended down to the last if() block of code in tsx_init() instead of squirreled away in a "get mode" function. Does this work?

        if (tsx_ctrl_state == TSX_CTRL_DISABLE) {
		...
        } else if (tsx_ctrl_state == TSX_CTRL_ENABLE) {
		...	
        } else if (tsx_ctrl_state == TSX_CTRL_RTM_ALWAYS_ABORT) {
		tsx_clear_cpuid();

		setup_clear_cpu_cap(X86_FEATURE_RTM);
		setup_clear_cpu_cap(X86_FEATURE_HLE);
	}
author
#2 lore raw
Quoted 25 lines · Dave Hansen
Quoted 5 lines · Jon Kohler

static enum tsx_ctrl_states x86_get_tsx_auto_mode(void) {

arch/x86/kernel/cpu/tsx.c +3 -0
+	/*
+	 * Hardware will always abort a TSX transaction if both CPUID bits
+	 * RTM_ALWAYS_ABORT and TSX_FORCE_ABORT are set. In this case, it is

arch/x86/kernel/cpu/tsx.c static bool __init tsx_ctrl_is_supported(void) · line 93

Quoted 10 lines
arch/x86/kernel/cpu/tsx.c +10 -0
+	 * better not to enumerate CPUID.RTM and CPUID.HLE bits. Clear them
+	 * here.
+	 */
+	if (boot_cpu_has(X86_FEATURE_RTM_ALWAYS_ABORT) &&
+	    boot_cpu_has(X86_FEATURE_TSX_FORCE_ABORT)) {
+		tsx_clear_cpuid();
+		setup_clear_cpu_cap(X86_FEATURE_RTM);
+		setup_clear_cpu_cap(X86_FEATURE_HLE);
+		return TSX_CTRL_RTM_ALWAYS_ABORT;
+	}

I don't really like hiding the setup_clear_cpu_cap() like this. Right now, all of the setup_clear_cpu_cap()'s are in a single function and they are pretty easy to figure out.

Quoted 3 lines

This seems like logic that deserves to be appended down to the last if() block of code in tsx_init() instead of squirreled away in a "get mode" function. Does this work?

Thanks for the review, Dave. Was trying to make the change simple with just a cut-n-paste of existing code from one place to the other, but I see what you’re saying. Yea, I can rework the logic as you suggested, I’ll send out a v2 patch.

Also, while I’ve got you, I’d also like to send out a patch to simply force abort all transactions even when tsx=on, and just be done with TSX. Now that we’ve had the patch that introduced this functionality I’m patching for roughly a year, combined with the microcode going out, it seems like TSX’s numbered days have come to an end.

That could greatly simplify the kernels handling of TAA on systems that have ARCH_CAP_TSX_CTRL_MSR.

Thoughts?

Quoted 11 lines
       if (tsx_ctrl_state == TSX_CTRL_DISABLE) {
		...
       } else if (tsx_ctrl_state == TSX_CTRL_ENABLE) {
		...	
       } else if (tsx_ctrl_state == TSX_CTRL_RTM_ALWAYS_ABORT) {
		tsx_clear_cpuid();

		setup_clear_cpu_cap(X86_FEATURE_RTM);
		setup_clear_cpu_cap(X86_FEATURE_HLE);
	}
#3 lore raw
Quoted 5 lines · Jon Kohler

Also, while I’ve got you, I’d also like to send out a patch to simply force abort all transactions even when tsx=on, and just be done with TSX. Now that we’ve had the patch that introduced this functionality I’m patching for roughly a year, combined with the microcode going out, it seems like TSX’s numbered days have come to an end.

Could you elaborate a little more here? Why would we ever want to force abort transactions that don't need to be aborted for some reason?

author
#4 lore raw
Quoted 13 lines · Dave Hansen
Quoted 5 lines · Jon Kohler

Also, while I’ve got you, I’d also like to send out a patch to simply force abort all transactions even when tsx=on, and just be done with TSX. Now that we’ve had the patch that introduced this functionality I’m patching for roughly a year, combined with the microcode going out, it seems like TSX’s numbered days have come to an end.

Could you elaborate a little more here? Why would we ever want to force abort transactions that don't need to be aborted for some reason?

Sure, I'm talking specifically about when users of tsx=on (or CONFIG_X86_INTEL_TSX_MODE_ON) on X86_BUG_TAA CPU SKUs. In this situation, TSX features are enabled, as are TAA mitigations. Using our own use case as an example, we only do this because of legacy live migration reasons.

This is fine on Skylake (because we're signed up for MDS mitigation anyhow) and fine on Ice Lake because TAA_NO=1; however this is wicked painful on Cascade Lake, because MDS_NO=1 and TAA_NO=0, so we're still signed up for TAA mitigation by default. On CLX, this hits us on host syscalls as well as vmexits with the mds clear on every one :(

So tsx=on is this oddball for us, because if we switch to auto, we'll break live migration for some of our customers (but TAA overhead is gone), but if we leave tsx=on, we keep the feature enabled (but no one likely uses it) and still have to pay the TAA tax even if a customer doesn't use it.

So my theory here is to extend the logical effort of the microcode driven automatic disablement as well as the tsx=auto automatic disablement and have tsx=on force abort all transactions on X86_BUG_TAA SKUs, but leave the CPU features enumerated to maintain live migration.

This would still leave TSX totally good on Ice Lake / non-buggy systems.

If it would help, I'm working up an RFC patch, and we could discuss there?

In the mean time, I did send out a v2 patch for this series addressing your comments.

Thanks again,
Jon

#5 lore raw
Quoted 8 lines · Jon Kohler

So my theory here is to extend the logical effort of the microcode driven automatic disablement as well as the tsx=auto automatic disablement and have tsx=on force abort all transactions on X86_BUG_TAA SKUs, but leave the CPU features enumerated to maintain live migration.

This would still leave TSX totally good on Ice Lake / non-buggy systems.

If it would help, I'm working up an RFC patch, and we could discuss there?

Sure. But, it sounds like you really want a new tdx=something rather than to muck with tsx=on behavior. Surely someone else will come along and complain that we broke their TDX setup if we change its behavior.

Maybe you should just pay the one-time cost and move your whole fleet over to tsx=off if you truly believe nobody is using it.

author
#6 lore raw
Quoted 10 lines · Dave Hansen
Quoted 5 lines · Jon Kohler

So my theory here is to extend the logical effort of the microcode driven automatic disablement as well as the tsx=auto automatic disablement and have tsx=on force abort all transactions on X86_BUG_TAA SKUs, but leave the CPU features enumerated to maintain live migration.

Quoted 7 lines
Quoted 3 lines

This would still leave TSX totally good on Ice Lake / non-buggy systems.

If it would help, I'm working up an RFC patch, and we could discuss there?

Sure. But, it sounds like you really want a new tdx=something rather than to muck with tsx=on behavior. Surely someone else will come along and complain that we broke their TDX setup if we change its behavior.

Good point, there will always be a squeaky wheel. I’ll work that into the RFC, I’ll do something like tsx=compat and see how it shapes up.

To be fair though, this commit I’m patching with this series would break setups as they apply 5.14+ and the microcode update, but you have a good point for certain.

Quoted 4 lines

Maybe you should just pay the one-time cost and move your whole fleet over to tsx=off if you truly believe nobody is using it.

Trust me, I’d love to do that; however:
We’ve thousands of hosts across thousands of unique customers, which aren't managed as a centralized service (customers manage them directly), so doing that would require each individual customer to organize a full power cycle for all of their VMs prior to an upgrade to tsx=off hosts.

That said, we are marching in that direction, we're shipping a control plane update that will mask HLE and RTM after power cycles, but that requires customers to apply that control plane update, then power cycle everything. Just means that we've begun the feature deprecation now, it will take years to fully bleed off without having customers to micro manage full power cycles.

#7 lore raw
Quoted 12 lines · Jon Kohler
Quoted 12 lines · Dave Hansen
Quoted 7 lines · Jon Kohler

So my theory here is to extend the logical effort of the microcode driven automatic disablement as well as the tsx=auto automatic disablement and have tsx=on force abort all transactions on X86_BUG_TAA SKUs, but leave the CPU features enumerated to maintain live migration.

This would still leave TSX totally good on Ice Lake / non-buggy systems.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

If it would help, I'm working up an RFC patch, and we could discuss there?

Sure. But, it sounds like you really want a new tdx=something rather than to muck with tsx=on behavior. Surely someone else will come along and complain that we broke their TDX setup if we change its behavior.

Good point, there will always be a squeaky wheel. I’ll work that into the RFC, I’ll do something like tsx=compat and see how it shapes up.

FYI, the original series had tsx=fake, that would have taken care of this breakage.

https://lore.kernel.org/lkml/de6b97a567e273adff1f5268998692bad548aa10.1623272033.git-series.pawan.kumar.gupta@linux.intel.com/

For the lack of real world use-cases at that time, this patch was dropped.

Thanks,
Pawan

author
#8 lore raw
Quoted 23 lines · Pawan Gupta
Quoted 18 lines · Jon Kohler
Quoted 17 lines · Dave Hansen
Quoted 8 lines · Jon Kohler

So my theory here is to extend the logical effort of the microcode driven automatic disablement as well as the tsx=auto automatic disablement and have tsx=on force abort all transactions on X86_BUG_TAA SKUs, but leave the CPU features enumerated to maintain live migration.

This would still leave TSX totally good on Ice Lake / non-buggy systems.

If it would help, I'm working up an RFC patch, and we could discuss there?

Sure. But, it sounds like you really want a new tdx=something rather than to muck with tsx=on behavior. Surely someone else will come along and complain that we broke their TDX setup if we change its behavior.

Quoted 5 lines
Quoted 2 lines

Good point, there will always be a squeaky wheel. I’ll work that into the RFC, I’ll do something like tsx=compat and see how it shapes up.

FYI, the original series had tsx=fake, that would have taken care of this breakage.

Fake sounds way better than compat, which is what I had :)

My RFC code looks similar to your patch, I’ll combine the approaches and send it out shortly, almost done

Quoted 6 lines

https://urldefense.proofpoint.com/v2/url?u=https-3A__lore.kernel.org_lkml_de6b97a567e273adff1f5268998692bad548aa10.1623272033.git-2Dseries.pawan.kumar.gupta-40linux.intel.com_&d=DwIDaQ&c=s883GpUCOChKOHiocYtGcg&r=NGPRGGo37mQiSXgHKm5rCQ&m=AgPWHzCORdn5x5rYXE0QeJ2yf158HOjDA5Bn8udzp-m6i9V9s7S_jtSiLog-dk93&s=kR74kfovpa0zOK0tZ2Ss9xbg2aRLI5oocB_cp_6DLkg&e=
For the lack of real world use-cases at that time, this patch was dropped.

Thanks,
Pawan

#9 lore raw
In reply to Jon Kohler
Quoted 30 lines · Jon Kohler
Quoted 13 lines · Dave Hansen
Quoted 5 lines · Jon Kohler

Also, while I’ve got you, I’d also like to send out a patch to simply force abort all transactions even when tsx=on, and just be done with TSX. Now that we’ve had the patch that introduced this functionality I’m patching for roughly a year, combined with the microcode going out, it seems like TSX’s numbered days have come to an end.

Could you elaborate a little more here? Why would we ever want to force abort transactions that don't need to be aborted for some reason?

Sure, I'm talking specifically about when users of tsx=on (or CONFIG_X86_INTEL_TSX_MODE_ON) on X86_BUG_TAA CPU SKUs. In this situation, TSX features are enabled, as are TAA mitigations. Using our own use case as an example, we only do this because of legacy live migration reasons.

This is fine on Skylake (because we're signed up for MDS mitigation anyhow) and fine on Ice Lake because TAA_NO=1; however this is wicked painful on Cascade Lake, because MDS_NO=1 and TAA_NO=0, so we're still signed up for TAA mitigation by default. On CLX, this hits us on host syscalls as well as vmexits with the mds clear on every one :(

So tsx=on is this oddball for us, because if we switch to auto, we'll break live migration for some of our customers (but TAA overhead is gone), but if we leave tsx=on, we keep the feature enabled (but no one likely uses it) and still have to pay the TAA tax even if a customer doesn't use it.

Quoted 4 lines

So my theory here is to extend the logical effort of the microcode driven automatic disablement as well as the tsx=auto automatic disablement and have tsx=on force abort all transactions on X86_BUG_TAA SKUs, but leave the CPU features enumerated to maintain live migration.

This won't help on CLX as server parts did not get the microcode driven automatic disablement. On CLX CPUID.RTM_ALWAYS_ABORT will not be set.

What could work on CLX is TSX_CTRL_RTM_DISABLE=1 and TSX_CTRL_CPUID_CLEAR=0. This can be done for tsx=auto or with a new mode tsx=fake|compat. IMO, adding a new mode would be better, otherwise tsx=auto behavior will differ depending on the kernel version.

Provided that software using TSX is following below guidance [*]:

When Intel TSX is disabled at runtime using TSX_CTRL, but the CPUID enumeration of Intel TSX is not cleared, existing software using RTM may see aborts for every transaction. The abort will always return a 0 status code in EAX after XBEGIN. When the software does a number of transaction retries, it should never retry for a 0 status value, but go to the nontransactional fall back path immediately.

Thanks,
Pawan

[*] TAA document: section -> Implications on Intel TSX software
https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/intel-tsx-asynchronous-abort.html

author
#10 lore raw
Quoted 40 lines · Pawan Gupta
Quoted 30 lines · Jon Kohler
Quoted 13 lines · Dave Hansen
Quoted 5 lines · Jon Kohler

Also, while I’ve got you, I’d also like to send out a patch to simply force abort all transactions even when tsx=on, and just be done with TSX. Now that we’ve had the patch that introduced this functionality I’m patching for roughly a year, combined with the microcode going out, it seems like TSX’s numbered days have come to an end.

Could you elaborate a little more here? Why would we ever want to force abort transactions that don't need to be aborted for some reason?

Sure, I'm talking specifically about when users of tsx=on (or CONFIG_X86_INTEL_TSX_MODE_ON) on X86_BUG_TAA CPU SKUs. In this situation, TSX features are enabled, as are TAA mitigations. Using our own use case as an example, we only do this because of legacy live migration reasons.

This is fine on Skylake (because we're signed up for MDS mitigation anyhow) and fine on Ice Lake because TAA_NO=1; however this is wicked painful on Cascade Lake, because MDS_NO=1 and TAA_NO=0, so we're still signed up for TAA mitigation by default. On CLX, this hits us on host syscalls as well as vmexits with the mds clear on every one :(

So tsx=on is this oddball for us, because if we switch to auto, we'll break live migration for some of our customers (but TAA overhead is gone), but if we leave tsx=on, we keep the feature enabled (but no one likely uses it) and still have to pay the TAA tax even if a customer doesn't use it.

Quoted 4 lines

So my theory here is to extend the logical effort of the microcode driven automatic disablement as well as the tsx=auto automatic disablement and have tsx=on force abort all transactions on X86_BUG_TAA SKUs, but leave the CPU features enumerated to maintain live migration.

Quoted 7 lines

This won't help on CLX as server parts did not get the microcode driven automatic disablement. On CLX CPUID.RTM_ALWAYS_ABORT will not be set.

What could work on CLX is TSX_CTRL_RTM_DISABLE=1 and TSX_CTRL_CPUID_CLEAR=0. This can be done for tsx=auto or with a new mode tsx=fake|compat. IMO, adding a new mode would be better, otherwise tsx=auto behavior will differ depending on the kernel version.

Thanks for the guidance, Pawan, I appreciate it. This is exactly the approach my other patch is taking. Need to do a bit more review and testing and ill get the RFC out

Quoted 15 lines

Provided that software using TSX is following below guidance [*]:

When Intel TSX is disabled at runtime using TSX_CTRL, but the CPUID enumeration of Intel TSX is not cleared, existing software using RTM may see aborts for every transaction. The abort will always return a 0 status code in EAX after XBEGIN. When the software does a number of transaction retries, it should never retry for a 0 status value, but go to the nontransactional fall back path immediately.

Thanks,
Pawan

[*] TAA document: section -> Implications on Intel TSX software
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.intel.com_content_www_us_en_developer_articles_technical_software-2Dsecurity-2Dguidance_technical-2Ddocumentation_intel-2Dtsx-2Dasynchronous-2Dabort.html&d=DwIDaQ&c=s883GpUCOChKOHiocYtGcg&r=NGPRGGo37mQiSXgHKm5rCQ&m=-yy3gpUOG7W2s79bE3KTnzd9h32x038M5CkPkhFsUW22MWWzcf3SoX6An2835zrn&s=t85c0qBMosrY_UvEVGzkR4j125aGfHju3SFEEPAImpQ&e=

author
#11 lore raw

PATCH v2 1/1 x86/tsx: fix KVM guest live migration for tsx=on

In reply to Jon Kohler

Move automatic disablement for TSX microcode deprecation from tsx_init() to x86_get_tsx_auto_mode(), such that systems with tsx=on will continue to see the TSX CPU features (HLE, RTM) even on updated microcode.

KVM live migration could be possibly be broken in 5.14+ commit 293649307ef9 ("x86/tsx: Clear CPUID bits when TSX always force aborts"). Consider the following scenario:

1. KVM hosts clustered in a live migration capable setup.
2. KVM guests have TSX CPU features HLE and/or RTM presented.
3. One of the three maintenance events occur:
3a. An existing host running kernel >= 5.14 in the pool updated with the
new microcode.
3b. A new host running kernel >= 5.14 is commissioned that already has the
microcode update preloaded.
3c. All hosts are running kernel < 5.14 with microcode update already
loaded and one existing host gets updated to kernel >= 5.14.
4. After maintenance event, the impacted host will not have HLE and RTM exposed, and live migrations with guests with TSX features might not migrate.

Users using tsx=on or CONFIG_X86_INTEL_TSX_MODE_ON should always see HLE and RTM on capable Intel SKUs, even if microcode has been clubbed to prevent functionality.

Users using tsx=auto get or CONFIG_X86_INTEL_TSX_MODE_AUTO get to roll the dice with whatever the kernel believes the appropriate default is, which includes the feature disappearing after a kernel and/or microcode update. These users should consider masking HLE and RTM at a higher control plane level, e.g. qemu or libvirt, such that guests on TSX enabled systems do not see HLE/RTM and therefore do not enable TAA mitigation.

Fixes 293649307ef9 ("x86/tsx: Clear CPUID bits when TSX always force aborts")
Cc Pawan Gupta <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Tony Luck <[email protected]>
Cc Neelima Krishnan <[email protected]>

---
v1 -> v2:
- Addressed comments on approach from Dave.

 arch/x86/kernel/cpu/tsx.c | 30 +++++++++++++++---------------
 1 file changed, 15 insertions(+), 15 deletions(-)
arch/x86/kernel/cpu/tsx.c +15 -15
index 9c7a5f049292..4b701fa64869 100644
@@ -78,6 +78,10 @@ static bool __init tsx_ctrl_is_supported(void)
 
 static enum tsx_ctrl_states x86_get_tsx_auto_mode(void)
 {
+	if (boot_cpu_has(X86_FEATURE_RTM_ALWAYS_ABORT) &&
+	    boot_cpu_has(X86_FEATURE_TSX_FORCE_ABORT))
+		return TSX_CTRL_RTM_ALWAYS_ABORT;
+
 	if (boot_cpu_has_bug(X86_BUG_TAA))
 		return TSX_CTRL_DISABLE;
 
@@ -105,21 +109,6 @@ void __init tsx_init(void)
 	char arg[5] = {};
 	int ret;
 
-	/*
-	 * Hardware will always abort a TSX transaction if both CPUID bits
-	 * RTM_ALWAYS_ABORT and TSX_FORCE_ABORT are set. In this case, it is
-	 * better not to enumerate CPUID.RTM and CPUID.HLE bits. Clear them
-	 * here.
-	 */
-	if (boot_cpu_has(X86_FEATURE_RTM_ALWAYS_ABORT) &&
-	    boot_cpu_has(X86_FEATURE_TSX_FORCE_ABORT)) {
-		tsx_ctrl_state = TSX_CTRL_RTM_ALWAYS_ABORT;
-		tsx_clear_cpuid();
-		setup_clear_cpu_cap(X86_FEATURE_RTM);
-		setup_clear_cpu_cap(X86_FEATURE_HLE);
-		return;
-	}
-
 	if (!tsx_ctrl_is_supported()) {
 		tsx_ctrl_state = TSX_CTRL_NOT_SUPPORTED;
 		return;
@@ -173,5 +162,16 @@ void __init tsx_init(void)
 		 */
 		setup_force_cpu_cap(X86_FEATURE_RTM);
 		setup_force_cpu_cap(X86_FEATURE_HLE);
+	} else if (tsx_ctrl_state == TSX_CTRL_RTM_ALWAYS_ABORT) {
+
+		/*
+		 * Hardware will always abort a TSX transaction if both CPUID bits
+		 * RTM_ALWAYS_ABORT and TSX_FORCE_ABORT are set. In this case, it is
+		 * better not to enumerate CPUID.RTM and CPUID.HLE bits. Clear them
+		 * here.
+		 */
+		tsx_clear_cpuid();
+		setup_clear_cpu_cap(X86_FEATURE_RTM);
+		setup_clear_cpu_cap(X86_FEATURE_HLE);
 	}
 }

--
2.30.1 (Apple Git-130)

#12 lore raw
Quoted 3 lines · Jon Kohler

Move automatic disablement for TSX microcode deprecation from tsx_init() to x86_get_tsx_auto_mode(), such that systems with tsx=on will continue to see the TSX CPU features (HLE, RTM) even on updated microcode.

This patch needs to be based on recent changes in TSX handling (due to Feb 2022 microcode update). These patches were recently merged in tip tree:

https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86/urgent

Specifically these patches:

x86/tsx: Use MSR_TSX_CTRL to clear CPUID bits [1] x86/tsx: Disable TSX development mode at boot [2]

Thanks,
Pawan

[1] https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?id=258f3b8c3210b03386e4ad92b4bd8652b5c1beb3
[2] https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?id=400331f8ffa3bec5c561417e5eec6848464e9160

#13 lore raw
In reply to Jon Kohler
Quoted 8 lines · Jon Kohler

Move automatic disablement for TSX microcode deprecation from tsx_init() to x86_get_tsx_auto_mode(), such that systems with tsx=on will continue to see the TSX CPU features (HLE, RTM) even on updated microcode.

KVM live migration could be possibly be broken in 5.14+ commit 293649307ef9 ("x86/tsx: Clear CPUID bits when TSX always force aborts"). Consider the following scenario:

Quoted 12 lines

1. KVM hosts clustered in a live migration capable setup.
2. KVM guests have TSX CPU features HLE and/or RTM presented.
3. One of the three maintenance events occur:
3a. An existing host running kernel >= 5.14 in the pool updated with the
new microcode.
3b. A new host running kernel >= 5.14 is commissioned that already has the
microcode update preloaded.
3c. All hosts are running kernel < 5.14 with microcode update already
loaded and one existing host gets updated to kernel >= 5.14.
4. After maintenance event, the impacted host will not have HLE and RTM exposed, and live migrations with guests with TSX features might not migrate.

Which part was this reproduced on? AFAIK server parts(except for some Intel Xeon E3s) did not get such microcode update.

Thanks,
Pawan