Skip to contentSkip to conversation

x86: mce: Xeon75xx specific interface to get corrected memory error information v2

PATCH In review Andi Kleen 2 participants 6 messages Mar 24, 2010 – Mar 29, 2010 x86

lore
Share & tools
lore

Andi Kleen

author
#0 lore raw

PATCH 1/1 x86: mce: Xeon75xx specific interface to get corrected memory error information v2

x86: mce: Xeon75xx specific interface to get corrected memory error information v2

[This version addresses the previous comments. It does not change any interface to the outside and does not attempt to encode DIMMs or anything like that, but only passes out the physical address of u a corrected error in the standard ADDR register field.
So for the outside it looks exactly the same as if the CPU supported this natively, but no otherwise special interfaces.

I hope this addresses previous concerns. I guess the DIMM error reporting can be revisited once there's a new reporting interface. There are still some traces of DIMM parsing in there, but it's only used for debug purposes now.]

---

Xeon 75xx doesn't log physical addresses on corrected machine check events in the standard architectural MSRs. Instead the address has to be retrieved in a model specific way. This makes it impossible to do predictive failure analysis.

Implement cpu model specific code to do this in mce-xeon75xx.c using a new hook that is called from the generic poll code. The code retrieves
the physical addressof the last corrected error from the platform and makes the address look like a standard architectural MCA address for further processing.

There's no code to print this information on a panic because this only works for corrected errors, and corrected errors do not usually result in panics.

The act of retrieving the PA information can take some time, so this code has a rate limit to avoid taking too much CPU time on a error flood.

The whole thing can be loaded as a module and has suitable PCI-IDs so that it can be auto-loaded by a distribution. The code also checks explicitely for the expected CPU model number to make sure this code doesn't run anywhere else.

---
 arch/x86/Kconfig                          |    8 
 arch/x86/kernel/cpu/mcheck/Makefile       |    1 
 arch/x86/kernel/cpu/mcheck/mce-internal.h |    1 
 arch/x86/kernel/cpu/mcheck/mce-xeon75xx.c |  402 ++++++++++++++++++++++++++++++
 arch/x86/kernel/cpu/mcheck/mce.c          |   12 
 arch/x86/kernel/e820.c                    |    3 
 6 files changed, 426 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/mcheck/mce.c +12 -0
Index: linux-2.6.34-rc1-ak/arch/x86/kernel/cpu/mcheck/mce.c
===================================================================
@@ -88,6 +88,9 @@
 static DEFINE_PER_CPU(struct mce, mces_seen);
 static int			cpu_missing;
 
+void 				(*cpu_specific_poll)(struct mce *);
+EXPORT_SYMBOL_GPL(cpu_specific_poll);
+
 /*
  * CPU/chipset specific EDAC code can register a notifier call here to print
  * MCE errors in a human-readable form.
@@ -364,6 +367,11 @@
 	wrmsrl(msr, v);
 }
 
+static int under_injection(void)
+{
+	return __get_cpu_var(injectm).finished;
+}
+
 /*
  * Simple lockless ring to communicate PFNs from the exception handler with the
  * process context work function. This is vastly simplified because there's
@@ -567,6 +575,10 @@
 
 		if (!(flags & MCP_TIMESTAMP))
 			m.tsc = 0;
+
+		if (cpu_specific_poll && !under_injection() && !mce_dont_log_ce)
+			cpu_specific_poll(&m);
+
 		/*
 		 * Don't get the IP here because it's unlikely to
 		 * have anything to do with the actual error location.
arch/x86/kernel/cpu/mcheck/mce-internal.h +1 -0
Index: linux-2.6.34-rc1-ak/arch/x86/kernel/cpu/mcheck/mce-internal.h
===================================================================
@@ -28,3 +28,4 @@
 
 extern struct mce_bank *mce_banks;
 
+extern void (*cpu_specific_poll)(struct mce *);
arch/x86/kernel/cpu/mcheck/Makefile +1 -0
Index: linux-2.6.34-rc1-ak/arch/x86/kernel/cpu/mcheck/Makefile
===================================================================
@@ -2,6 +2,7 @@
 
 obj-$(CONFIG_X86_ANCIENT_MCE)	+= winchip.o p5.o
 obj-$(CONFIG_X86_MCE_INTEL)	+= mce_intel.o
+obj-$(CONFIG_X86_MCE_XEON75XX)	+= mce-xeon75xx.o
 obj-$(CONFIG_X86_MCE_AMD)	+= mce_amd.o
 obj-$(CONFIG_X86_MCE_THRESHOLD) += threshold.o
 obj-$(CONFIG_X86_MCE_INJECT)	+= mce-inject.o
arch/x86/kernel/cpu/mcheck/mce-xeon75xx.c new file +402 -0

402 lines not on this page · raw message

arch/x86/kernel/e820.c +2 -1
Index: linux-2.6.34-rc1-ak/arch/x86/kernel/e820.c
===================================================================
@@ -71,7 +71,7 @@
  * Note: this function only works correct if the e820 table is sorted and
  * not-overlapping, which is the case
  */
-int __init e820_all_mapped(u64 start, u64 end, unsigned type)
+int e820_all_mapped(u64 start, u64 end, unsigned type)
 {
 	int i;
 
@@ -98,6 +98,7 @@
 	}
 	return 0;
 }
+EXPORT_SYMBOL_GPL(e820_all_mapped);
 
 /*
  * Add a memory region to the kernel e820 map.
arch/x86/Kconfig +8 -0
Index: linux-2.6.34-rc1-ak/arch/x86/Kconfig
===================================================================
@@ -864,6 +864,14 @@
 	   Additional support for intel specific MCE features such as
 	   the thermal monitor.
 
+config X86_MCE_XEON75XX
+	tristate "Intel Xeon 7500 series corrected memory error driver"
+	depends on X86_MCE_INTEL
+	---help---
+	   Add support for a Intel Xeon 7500 series specific memory error driver.
+	   This allows to report the DIMM and physical address on a corrected
+	   memory error machine check event.
+
 config X86_MCE_AMD
 	def_bool y
 	prompt "AMD MCE features"
author
#1 lore raw
Quoted 1 line · Andi Kleen

x86: mce: Xeon75xx specific interface to get corrected memory error information v2

Ping? Please review this patch. Thanks.
-Andi

Quoted 19 lines

[This version addresses the previous comments. It does not change any interface to the outside and does not attempt to encode DIMMs or anything like that, but only passes out the physical address of u a corrected error in the standard ADDR register field.
So for the outside it looks exactly the same as if the CPU supported this natively, but no otherwise special interfaces.

I hope this addresses previous concerns. I guess the DIMM error reporting can be revisited once there's a new reporting interface. There are still some traces of DIMM parsing in there, but it's only used for debug purposes now.]

---

Xeon 75xx doesn't log physical addresses on corrected machine check events in the standard architectural MSRs. Instead the address has to be retrieved in a model specific way. This makes it impossible to do predictive failure analysis.

Signature · 1 line
[email protected] -- Speaking for myself only.
#2 lore raw
Quoted 26 lines · Andi Kleen
Quoted 1 line · Andi Kleen

x86: mce: Xeon75xx specific interface to get corrected memory error information v2

Ping? Please review this patch. Thanks.
-Andi

Quoted 19 lines

[This version addresses the previous comments. It does not change any interface to the outside and does not attempt to encode DIMMs or anything like that, but only passes out the physical address of u a corrected error in the standard ADDR register field.
So for the outside it looks exactly the same as if the CPU supported this natively, but no otherwise special interfaces.

I hope this addresses previous concerns. I guess the DIMM error reporting can be revisited once there's a new reporting interface. There are still some traces of DIMM parsing in there, but it's only used for debug purposes now.]

---

Xeon 75xx doesn't log physical addresses on corrected machine check events in the standard architectural MSRs. Instead the address has to be retrieved in a model specific way. This makes it impossible to do predictive failure analysis.

Could you point proper specification or datasheet to know/check what you are going to do here?

Thanks,
H.Seto

author
#3 lore raw
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

Xeon 75xx doesn't log physical addresses on corrected machine check events in the standard architectural MSRs. Instead the address has to be retrieved in a model specific way. This makes it impossible to do predictive failure analysis.

Could you point proper specification or datasheet to know/check what you are going to do here?

You mean how the model specific interface works?

There's currently no public specification for the interface, but it should be reasonably clear from reading the driver how it works.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#4 lore raw
Quoted 8 lines · Andi Kleen
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

Xeon 75xx doesn't log physical addresses on corrected machine check events in the standard architectural MSRs. Instead the address has to be retrieved in a model specific way. This makes it impossible to do predictive failure analysis.

Could you point proper specification or datasheet to know/check what you are going to do here?

Quoted 7 lines

You mean how the model specific interface works?

There's currently no public specification for the interface, but it should be reasonably clear from reading the driver how it works.

-Andi

It looks like overengineered...

I have some questions: Is it impossible to get the address after polling handler have processed? e.g. Is it possible to implement this module as mcelog's add-on that hooked & invoked immediately after reading /dev/mcelog? I guess there are some limitation/restriction to call pfa_command().

Are there any alternative way to get the address?
Polling like edac_i7 doesn't help this?

You pointed "This makes it impossible to do predictive failure analysis", but I guess we could do rough-but-enough analysis that requires coarse resolution like sockets. Or we should not expect that one of DIMMs connected to the socket is broken if the socket reports corrected memory errors many time?

Thanks,
H.Seto

author
#5 lore raw
Quoted 5 lines · Hidetoshi Seto

I have some questions: Is it impossible to get the address after polling handler have processed? e.g. Is it possible to implement this module as mcelog's add-on that hooked & invoked immediately after reading /dev/mcelog? I guess there are some limitation/restriction to call pfa_command().

You need to be in ring 0. In theory you could do it later, but then you risk losing the address if there is another error.

Quoted 1 line

Are there any alternative way to get the address?

No.

Quoted 1 line

Polling like edac_i7 doesn't help this?

No.

Quoted 4 lines

You pointed "This makes it impossible to do predictive failure analysis", but I guess we could do rough-but-enough analysis that requires coarse resolution like sockets. Or we should not expect

The main predictive failure analysis interesting here is bad page offlining and for that you need an address.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.