Skip to contentSkip to conversation

x86, mce: Don't clear global error reporting banks during cpu_offline

PATCH In review Ashok Raj 3 participants 3 messages Sep 4, 2015 – Sep 29, 2015 x86

lore
Share & tools
lore

Ashok Raj

author
#0 lore raw

PATCH 1/1 x86, mce: Don't clear global error reporting banks during cpu_offline

During CPU offline, or during suspend/resume operations, its not safe to clear MCi_CTL. These MSR's are either thread scoped (meaning private to thread), or core scoped (private to threads in that core only), or socket scope i.e visible and controllable from all threads in the socket.

When we turn off during CPU_OFFLINE, just offlining a single CPU will stop signaling for all the socket wide resources, such as LLC, iMC for e.g.

It is true for Intel CPU's. But there seems some history that other processors may require to turn these off during every CPU offline.

Intel Secure Guard eXtentions (SGX) is worried that it might be possible to compromise integrity in a SGX system if the attacker has control of host system to inject errors which would be otherwise ignored when MCi_CTL bits are cleared. Hence on SGX enabled systems, if MCi_CTL is cleared SGX becomes not available anymore.

- Consolidated some code to use sharing
- Minor changes to some prototypes to fit usage.
- Left handling same for non-Intel CPU models to avoid any unknown regressions.
- Fixed review comments from Boris

---
 arch/x86/kernel/cpu/mcheck/mce.c | 30 ++++++++++++++++++++----------
 1 file changed, 20 insertions(+), 10 deletions(-)
arch/x86/kernel/cpu/mcheck/mce.c +20 -10
index d350858..69c7e3c 100644
@@ -2100,7 +2100,7 @@ int __init mcheck_init(void)
  * Disable machine checks on suspend and shutdown. We can't really handle
  * them later.
  */
-static int mce_disable_error_reporting(void)
+static void mce_disable_error_reporting(void)
 {
 	int i;
 
@@ -2110,17 +2110,32 @@ static int mce_disable_error_reporting(void)
 		if (b->init)
 			wrmsrl(MSR_IA32_MCx_CTL(i), 0);
 	}
-	return 0;
+	return;
+}
+
+static void vendor_disable_error_reporting(void)
+{
+	/*
+	 * Don't clear on Intel CPUs. Some of these MSRs are
+	 * socket wide. Disabling them for just a single CPU offline
+	 * is bad, since it will inhibit reporting for all shared
+	 * resources.. such as LLC, iMC for e.g.
+	 */
+	if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL)
+		return;
+
+	mce_disable_error_reporting();
 }
 
 static int mce_syscore_suspend(void)
 {
-	return mce_disable_error_reporting();
+	vendor_disable_error_reporting();
+	return 0;
 }
 
 static void mce_syscore_shutdown(void)
 {
-	mce_disable_error_reporting();
+	vendor_disable_error_reporting();
 }
 
 /*
@@ -2400,19 +2415,14 @@ static void mce_device_remove(unsigned int cpu)
 static void mce_disable_cpu(void *h)
 {
 	unsigned long action = *(unsigned long *)h;
-	int i;
 
 	if (!mce_available(raw_cpu_ptr(&cpu_info)))
 		return;
 
 	if (!(action & CPU_TASKS_FROZEN))
 		cmci_clear();
-	for (i = 0; i < mca_cfg.banks; i++) {
-		struct mce_bank *b = &mce_banks[i];
 
-		if (b->init)
-			wrmsrl(MSR_IA32_MCx_CTL(i), 0);
-	}
+	vendor_disable_error_reporting();
 }
 
 static void mce_reenable_cpu(void *h)
Signature · 1 line
2.4.3
maintainer Reviewed-by Tony Luck Tested-by Serge Ayoun on v1 1/1
#1 lore raw
Quoted 17 lines · Ashok Raj

During CPU offline, or during suspend/resume operations, its not safe to clear MCi_CTL. These MSR's are either thread scoped (meaning private to thread), or core scoped (private to threads in that core only), or socket scope i.e visible and controllable from all threads in the socket.

When we turn off during CPU_OFFLINE, just offlining a single CPU will stop signaling for all the socket wide resources, such as LLC, iMC for e.g.

It is true for Intel CPU's. But there seems some history that other processors may require to turn these off during every CPU offline.

Intel Secure Guard eXtentions (SGX) is worried that it might be possible to compromise integrity in a SGX system if the attacker has control of host system to inject errors which would be otherwise ignored when MCi_CTL bits are cleared. Hence on SGX enabled systems, if MCi_CTL is cleared SGX becomes not available anymore.

Quoted 4 lines

- Consolidated some code to use sharing
- Minor changes to some prototypes to fit usage.
- Left handling same for non-Intel CPU models to avoid any unknown regressions.
- Fixed review comments from Boris

I massaged the text a bit more and ended up committing this:

---
From: Ashok Raj <[email protected]>
Date: Fri, 4 Sep 2015 14:29:50 -0400
Subject: [PATCH] x86/mce: Don't clear shared banks on Intel when offlining
CPUs

It is not safe to clear global MCi_CTL banks during CPU offline or suspend/resume operations. These MSRs are either thread-scoped (meaning private to a thread), or core-scoped (private to threads in that core only), or with a socket scope: visible and controllable from all threads in the socket.

When we offline a single CPU, clearing those MCi_CTL bits will stop signaling for all the shared, i.e., socket-wide resources, such as LLC, iMC, etc.

In addition, it might be possible to compromise the integrity of an Intel Secure Guard eXtentions (SGX) system if the attacker has control of the host system and is able to inject errors which would be otherwise ignored when MCi_CTL bits are cleared.

Hence on SGX enabled systems, if MCi_CTL is cleared, SGX gets disabled.

---
 arch/x86/kernel/cpu/mcheck/mce.c | 30 ++++++++++++++++++++----------
 1 file changed, 20 insertions(+), 10 deletions(-)
arch/x86/kernel/cpu/mcheck/mce.c +20 -10
index 9d014b82a124..17b5ec6edcb6 100644
@@ -2042,7 +2042,7 @@ int __init mcheck_init(void)
  * Disable machine checks on suspend and shutdown. We can't really handle
  * them later.
  */
-static int mce_disable_error_reporting(void)
+static void mce_disable_error_reporting(void)
 {
 	int i;
 
@@ -2052,17 +2052,32 @@ static int mce_disable_error_reporting(void)
 		if (b->init)
 			wrmsrl(MSR_IA32_MCx_CTL(i), 0);
 	}
-	return 0;
+	return;
+}
+
+static void vendor_disable_error_reporting(void)
+{
+	/*
+	 * Don't clear on Intel CPUs. Some of these MSRs are socket-wide.
+	 * Disabling them for just a single offlined CPU is bad, since it will
+	 * inhibit reporting for all shared resources on the socket like the
+	 * last level cache (LLC), the integrated memory controller (iMC), etc.
+	 */
+	if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL)
+		return;
+
+	mce_disable_error_reporting();
 }
 
 static int mce_syscore_suspend(void)
 {
-	return mce_disable_error_reporting();
+	vendor_disable_error_reporting();
+	return 0;
 }
 
 static void mce_syscore_shutdown(void)
 {
-	mce_disable_error_reporting();
+	vendor_disable_error_reporting();
 }
 
 /*
@@ -2342,19 +2357,14 @@ static void mce_device_remove(unsigned int cpu)
 static void mce_disable_cpu(void *h)
 {
 	unsigned long action = *(unsigned long *)h;
-	int i;
 
 	if (!mce_available(raw_cpu_ptr(&cpu_info)))
 		return;
 
 	if (!(action & CPU_TASKS_FROZEN))
 		cmci_clear();
-	for (i = 0; i < mca_cfg.banks; i++) {
-		struct mce_bank *b = &mce_banks[i];
 
-		if (b->init)
-			wrmsrl(MSR_IA32_MCx_CTL(i), 0);
-	}
+	vendor_disable_error_reporting();
 }
 
 static void mce_reenable_cpu(void *h)
Signature · 8 lines
2.3.5


-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
#2 lore raw

[tip:ras/core] x86/mce: Don' t clear shared banks on Intel when offlining CPUs

In reply to Ashok Raj
x86/mce: Don't clear shared banks on Intel when offlining CPUs
Commit-ID:  6e06780a98f149f131d46c1108d4ae27f05a9357
Gitweb:     http://git.kernel.org/tip/6e06780a98f149f131d46c1108d4ae27f05a9357
Author:     Ashok Raj <[email protected]>
AuthorDate: Mon, 28 Sep 2015 09:21:43 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Mon, 28 Sep 2015 10:15:26 +0200

x86/mce: Don't clear shared banks on Intel when offlining CPUs

It is not safe to clear global MCi_CTL banks during CPU offline or suspend/resume operations. These MSRs are either thread-scoped (meaning private to a thread), or core-scoped (private to threads in that core only), or with a socket scope:
visible and controllable from all threads in the socket.

When we offline a single CPU, clearing those MCi_CTL bits will stop signaling for all the shared, i.e., socket-wide resources, such as LLC, iMC, etc.

In addition, it might be possible to compromise the integrity of an Intel Secure Guard eXtentions (SGX) system if the attacker has control of the host system and is able to inject errors which would be otherwise ignored when MCi_CTL bits are cleared.

Hence on SGX enabled systems, if MCi_CTL is cleared, SGX gets disabled.

[ Cleanup text. ]
Cc Andy Lutomirski <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Brian Gerst <[email protected]>
Cc Denys Vlasenko <[email protected]>
Cc H. Peter Anvin <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc linux-edac <[email protected]>
---
 arch/x86/kernel/cpu/mcheck/mce.c | 30 ++++++++++++++++++++----------
 1 file changed, 20 insertions(+), 10 deletions(-)
arch/x86/kernel/cpu/mcheck/mce.c +20 -10
index 9d014b82..17b5ec6 100644
@@ -2042,7 +2042,7 @@ int __init mcheck_init(void)
  * Disable machine checks on suspend and shutdown. We can't really handle
  * them later.
  */
-static int mce_disable_error_reporting(void)
+static void mce_disable_error_reporting(void)
 {
 	int i;
 
@@ -2052,17 +2052,32 @@ static int mce_disable_error_reporting(void)
 		if (b->init)
 			wrmsrl(MSR_IA32_MCx_CTL(i), 0);
 	}
-	return 0;
+	return;
+}
+
+static void vendor_disable_error_reporting(void)
+{
+	/*
+	 * Don't clear on Intel CPUs. Some of these MSRs are socket-wide.
+	 * Disabling them for just a single offlined CPU is bad, since it will
+	 * inhibit reporting for all shared resources on the socket like the
+	 * last level cache (LLC), the integrated memory controller (iMC), etc.
+	 */
+	if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL)
+		return;
+
+	mce_disable_error_reporting();
 }
 
 static int mce_syscore_suspend(void)
 {
-	return mce_disable_error_reporting();
+	vendor_disable_error_reporting();
+	return 0;
 }
 
 static void mce_syscore_shutdown(void)
 {
-	mce_disable_error_reporting();
+	vendor_disable_error_reporting();
 }
 
 /*
@@ -2342,19 +2357,14 @@ static void mce_device_remove(unsigned int cpu)
 static void mce_disable_cpu(void *h)
 {
 	unsigned long action = *(unsigned long *)h;
-	int i;
 
 	if (!mce_available(raw_cpu_ptr(&cpu_info)))
 		return;
 
 	if (!(action & CPU_TASKS_FROZEN))
 		cmci_clear();
-	for (i = 0; i < mca_cfg.banks; i++) {
-		struct mce_bank *b = &mce_banks[i];
 
-		if (b->init)
-			wrmsrl(MSR_IA32_MCx_CTL(i), 0);
-	}
+	vendor_disable_error_reporting();
 }
 
 static void mce_reenable_cpu(void *h)