Skip to contentSkip to conversation

re-work MCA telemetry internals; use common code for Intel/AMD MCA

PATCH In review Frank van der Linden 5 participants 10 messages Mar 16, 2009 – Mar 18, 2009

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 re-work MCA telemetry internals; use common code for Intel/AMD MCA

The following patch reworks the MCA error telemetry handling inside Xen,
and shares code between the Intel and AMD implementations as much as
possible.

I've had this patch sitting around for a while, but it wasn't ported to -unstable yet. I finished porting and testing it, and am submitting it now, because the Intel folks want to go ahead and submit their new changes, so we agreed that I should push our changes first.

Brief explanation of the telemetry part: previously, the telemetry was accessed in a global array, with index variables used to access it. There were some issues with that: race conditions with regard to new machine checks (or CMCIs) coming in while handling the telemetry, and interaction with domains having been notified or not, which was a bit hairy. Our changes (I should say: Gavin Maltby's changes, as he did the bulk of this work for our 3.1 based tree, I merely ported/extended it to 3.3 and beyond) make telemetry access transactional (think of a database). Also, the internal database updates are atomic, since the final commit is done by a pointer swap. There is a brief explanation of the mechanism in mctelem.h.This patch also removes dom0->domU notification, which is ok, since Intel's upcoming changes will replace domU notification with a vMCE mechanism anyway.

The common code part is pretty much what it says. It defines a common MCE handler, with a few hooks for the special needs of the specific CPUs.

I've been told that Intel's upcoming patch will need to make some parts of the common code specific to the Intel CPU again, but we'll work together to use as much common code as possible.

- Frank

Replace hypervisor MCA telemetry structures with something more robust and designed to make terminal error telemetry available to the dom0 panic flow for diagnosis on reboot.

Use common code for a lot of the AMD and Intel MCE handling code.

xen/arch/x86/cpu/mcheck/Makefile +1 -0
@@ -2,6 +2,7 @@ obj-y += k7.o
 obj-y += k7.o
 obj-y += amd_k8.o
 obj-y += amd_f10.o
+obj-y += mctelem.o
 obj-y += mce.o
 obj-y += mce_intel.o
 obj-y += non-fatal.o
xen/arch/x86/cpu/mcheck/amd_f10.c +16 -21
@@ -49,20 +49,21 @@
 #include "x86_mca.h"
 
 
-static int amd_f10_handler(struct mc_info *mi, uint16_t bank, uint64_t status)
+static enum mca_extinfo
+amd_f10_handler(struct mc_info *mi, uint16_t bank, uint64_t status)
 {
 	struct mcinfo_extended mc_ext;
 
 	/* Family 0x10 introduced additional MSR that belong to the
 	 * northbridge bank (4). */
-	if (bank != 4)
-		return 0;
+	if (mi == NULL || bank != 4)
+		return MCA_EXTINFO_IGNORED;
 
 	if (!(status & MCi_STATUS_VAL))
-		return 0;
+		return MCA_EXTINFO_IGNORED;
 
 	if (!(status & MCi_STATUS_MISCV))
-		return 0;
+		return MCA_EXTINFO_IGNORED;
 
 	memset(&mc_ext, 0, sizeof(mc_ext));
 	mc_ext.common.type = MC_TYPE_EXTENDED;
@@ -78,23 +79,25 @@ static int amd_f10_handler(struct mc_inf
 	rdmsrl(MSR_F10_MC4_MISC3, mc_ext.mc_msr[2].value);
 	
 	x86_mcinfo_add(mi, &mc_ext);
-	return 1;
+	return MCA_EXTINFO_LOCAL;
 }
 
 
 extern void k8_machine_check(struct cpu_user_regs *regs, long error_code);
 
 /* AMD Family10 machine check */
-void amd_f10_mcheck_init(struct cpuinfo_x86 *c) 
+int amd_f10_mcheck_init(struct cpuinfo_x86 *c) 
 { 
 	uint64_t value;
 	uint32_t i;
 	int cpu_nr;
 
-	machine_check_vector = k8_machine_check;
-	mc_callback_bank_extended = amd_f10_handler;
+	if (!cpu_has(c, X86_FEATURE_MCA))
+		return 0;
+
+	x86_mce_vector_register(k8_machine_check);
+	x86_mce_callback_register(amd_f10_handler);
 	cpu_nr = smp_processor_id();
-	wmb();
 
 	rdmsrl(MSR_IA32_MCG_CAP, value);
 	if (value & MCG_CTL_P)	/* Control register present ? */
@@ -104,18 +107,9 @@ void amd_f10_mcheck_init(struct cpuinfo_
 	for (i = 0; i < nr_mce_banks; i++) {
 		switch (i) {
 		case 4: /* Northbridge */
-			/* Enable error reporting of all errors,
-			 * enable error checking and
-			 * disable sync flooding */
-			wrmsrl(MSR_IA32_MC4_CTL, 0x02c3c008ffffffffULL);
+			/* Enable error reporting of all errors */
+			wrmsrl(MSR_IA32_MC4_CTL, 0xffffffffffffffffULL);
 			wrmsrl(MSR_IA32_MC4_STATUS, 0x0ULL);
-
-			/* XXX: We should write the value 0x1087821UL into
-			 * to register F3x180 here, which sits in
-			 * the PCI extended configuration space.
-			 * Since this is not possible here, we can only hope,
-			 * Dom0 is doing that.
-			 */
 			break;
 
 		default:
@@ -128,4 +122,5 @@ void amd_f10_mcheck_init(struct cpuinfo_
 
 	set_in_cr4(X86_CR4_MCE);
 	printk("CPU%i: AMD Family10h machine check reporting enabled.\n", cpu_nr);
+	return 1;
 }
xen/arch/x86/cpu/mcheck/amd_k8.c +11 -218

259 lines not on this page · raw message

xen/arch/x86/cpu/mcheck/amd_nonfatal.c +47 -99

201 lines not on this page · raw message

xen/arch/x86/cpu/mcheck/k7.c +8 -3
@@ -68,13 +68,16 @@ static fastcall void k7_machine_check(st
 
 
 /* AMD K7 machine check */
-void amd_k7_mcheck_init(struct cpuinfo_x86 *c)
+int amd_k7_mcheck_init(struct cpuinfo_x86 *c)
 {
 	u32 l, h;
 	int i;
 
-	machine_check_vector = k7_machine_check;
-	wmb();
+	/* Check for PPro style MCA; our caller has confirmed MCE support. */
+	if (!cpu_has(c, X86_FEATURE_MCA))
+		return 0;
+
+	x86_mce_vector_register(k7_machine_check);
 
 	rdmsr (MSR_IA32_MCG_CAP, l, h);
 	if (l & (1<<8))	/* Control register present ? */
@@ -92,4 +95,6 @@ void amd_k7_mcheck_init(struct cpuinfo_x
 	set_in_cr4 (X86_CR4_MCE);
 	printk (KERN_INFO "CPU%d: AMD K7 machine check reporting enabled.\n",
 		smp_processor_id());
+
+	return 1;
 }
xen/arch/x86/cpu/mcheck/mce.c +597 -404

1,160 lines not on this page · raw message

xen/arch/x86/cpu/mcheck/mce.h +79 -19

117 lines not on this page · raw message

xen/arch/x86/cpu/mcheck/mce_intel.c +94 -297

456 lines not on this page · raw message

xen/arch/x86/cpu/mcheck/mctelem.c new file +443 -0

443 lines not on this page · raw message

xen/arch/x86/cpu/mcheck/mctelem.h new file +71 -0

71 lines not on this page · raw message

xen/arch/x86/cpu/mcheck/non-fatal.c +60 -27

134 lines not on this page · raw message

xen/arch/x86/cpu/mcheck/p5.c +6 -9
@@ -16,7 +16,7 @@
 #include "x86_mca.h"
 
 /* Machine check handler for Pentium class Intel */
-static fastcall void pentium_machine_check(struct cpu_user_regs * regs, long error_code)
+static void pentium_machine_check(struct cpu_user_regs * regs, long error_code)
 {
 	u32 loaddr, hi, lotype;
 	rdmsr(MSR_IA32_P5_MC_ADDR, loaddr, hi);
@@ -28,19 +28,14 @@ static fastcall void pentium_machine_che
 }
 
 /* Set up machine check reporting for processors with Intel style MCE */
-void intel_p5_mcheck_init(struct cpuinfo_x86 *c)
+int intel_p5_mcheck_init(struct cpuinfo_x86 *c)
 {
 	u32 l, h;
 	
-	/*Check for MCE support */
-	if( !cpu_has(c, X86_FEATURE_MCE) )
-		return;	
-
 	/* Default P5 to off as its often misconnected */
 	if(mce_disabled != -1)
-		return;
-	machine_check_vector = pentium_machine_check;
-	wmb();
+		return 0;
+	x86_mce_vector_register(pentium_machine_check);
 
 	/* Read registers before enabling */
 	rdmsr(MSR_IA32_P5_MC_ADDR, l, h);
@@ -50,4 +45,6 @@ void intel_p5_mcheck_init(struct cpuinfo
  	/* Enable MCE */
 	set_in_cr4(X86_CR4_MCE);
 	printk(KERN_INFO "Intel old style machine check reporting enabled on CPU#%d.\n", smp_processor_id());
+
+	return 1;
 }
xen/arch/x86/cpu/mcheck/winchip.c +5 -3
@@ -16,22 +16,24 @@
 #include "mce.h"
 
 /* Machine check handler for WinChip C6 */
-static fastcall void winchip_machine_check(struct cpu_user_regs * regs, long error_code)
+static void winchip_machine_check(struct cpu_user_regs * regs, long error_code)
 {
 	printk(KERN_EMERG "CPU0: Machine Check Exception.\n");
 	add_taint(TAINT_MACHINE_CHECK);
 }
 
[… more lines in the raw message]

Lines of this file are not shown. View the raw message.

xen/arch/x86/cpu/mcheck/x86_mca.h +8 -0

Lines of this file are not shown. View the raw message.

xen/include/asm-x86/traps.h +1 -1

Lines of this file are not shown. View the raw message.

xen/include/public/arch-x86/xen-mca.h +29 -20

49 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Part of the diff is not shown here. View the raw message.

#1 lore raw
Quoted 28 lines · [email protected] <> wrote:

The following patch reworks the MCA error telemetry handling inside Xen,
and shares code between the Intel and AMD implementations as much as
possible.

I've had this patch sitting around for a while, but it wasn't ported to
-unstable yet. I finished porting and testing it, and am submitting it now, because the Intel folks want to go ahead and submit their new changes, so we agreed that I should push our changes first.

Brief explanation of the telemetry part: previously, the telemetry was accessed in a global array, with index variables used to access it. There were some issues with that: race conditions with regard to new machine checks (or CMCIs) coming in while handling the telemetry, and interaction with domains having been notified or not, which was a bit hairy. Our changes (I should say: Gavin Maltby's changes, as he did the
bulk of this work for our 3.1 based tree, I merely ported/extended it to
3.3 and beyond) make telemetry access transactional (think of a database). Also, the internal database updates are atomic, since the final commit is done by a pointer swap. There is a brief explanation of
the mechanism in mctelem.h.This patch also removes dom0->domU notification, which is ok, since Intel's upcoming changes will replace domU notification with a vMCE mechanism anyway.

Quoted 8 lines

The common code part is pretty much what it says. It defines a common MCE handler, with a few hooks for the special needs of the specific CPUs.

I've been told that Intel's upcoming patch will need to make some parts
of the common code specific to the Intel CPU again, but we'll work together to use as much common code as possible.

Yes, as shown in our previous patch, we do change the current MCA handler, the main changes are followed:

1) Most importantly, we implement a softIRQ mechanism for post MCE handler.
The reason is, the #MC can happen in any time, that means: Firstly it is spin-lock unsafe, some code like vcpu_schedule_lock_irq(v) in current MCA handler is sure to cause hang if that lock is already hold by a ISR; Secondly, the execution context is uncertain, the "current " value in current MCA handler maybe incorrect (if set_current is interrupted by #MC), the page ownership maybe wrong (if still in change under heap_lock protection) etc. I remember this So our patch handling #MC is in two step. The MCA handler, which depends on the execution context when MCA happen (like checking if it is in Xen context) and especially it will bring all CPU to softIRQ context. The softIRQ handler (i.e. post handler), which will be spin_lock safe, and all CPU is redenzvous, so it can take more actions.

2) We implement a mechanism to handle the shared MSR resources similar to what we have done in CMCI handler. As the Intel SDM stated, some MC resource is shared by multiple logical CPU, we implement a ownership check.

3) As stated in linux MCA handler, on Intel platforms machine check exceptions are always broadcast to all CPUs, we add such support also.

We have no idea how the issues for item 2 and 3 are handled on other platform, so we have no idea on how to do the common handler for it, hope Christoph can provide more suggestion, or we can just keep them different for different platform.

But I think for item 1, it is software related, so it can be a enhancement to the common handler, the only thing I'm not sure is, if we need bring all CPU to softIRQ context in all platform, maybe Christoph can give more idea.

Since we have get most consensus on the high level idea of MCA handler (i.e. Xen take action instead of dom0, use vMCE for guest MCA etc, check discussion with subject " [RFC] RAS(Part II)--MCA enalbing in XEN", the only thing left is the detail method of how to pass the recover action information to dom0), maybe we can turn to this second level discussion of how to enhance the (common) MCA handler.

Thanks
-- Yunhong Jiang

Quoted 2 lines

- Frank

#4 lore raw
In reply to Frank van der Linden

Hi, Frank

I am now doing some tests based on latest Intel platform for this patch since CMCI needs some owner_checking and only the owned CPU will report the error.

Without the patch, when CMCI happened, since CPU0 is the owner of bank8, so when do checking, Only CPU0 will report the error.

Below is the correct log
(XEN) CMCI: cmci_intr happen on CPU3
[root@lke-ep inject]# (XEN) CMCI: cmci_intr happen on CPU2
(XEN) CMCI: cmci_intr happen on CPU0
(XEN) CMCI: cmci_intr happen on CPU1
(XEN) mcheck_poll: bank8 CPU0 status[cc0000800001009f]
(XEN) mcheck_poll: CPU0, SOCKET0, CORE0, APICID[0], thread[0]
(XEN) MCE: The hardware reports a non fatal, correctable incident occured on CPU
0.

After applied your patch, I found all CPUs will report the error. Below is the log
(XEN) MCE: The hardware reports a non fatal, correctable i
ncident occured on CPU 0.
(XEN) MCE: The hardware reports a non fatal, correctable incident occured on CPU
2.
(XEN) MCE: The hardware reports a non fatal, correctable incident occured on CPU
3.
(XEN) MCE: The hardware reports a non fatal, correctable incident occured on CPU
1.
(XEN) Bank 8: cc0000c00001009f<1>Bank 8: 8c0000400001009f<1>Bank 8: cc0001c00001 009f<1>MCE: The hardware reports a non fatal, correctable incident occured on CP U 0.

I noticed your patch has passed in the cmci_owner mask, I can't see the reason since this is really a big patch. I need some time to figure it out.

Also we found the polling mechanism has some changes. My feeling is that this patch is really too big. We can't easily figured out the impaction to our checked-in codes right now.
Just wonder whether you could split this big patch into two parts :-)
part1: mce log telem mechanism and required mce_intel interfaces changes. So that we can verify easily whether the new interfaces works fine for our CMCI as well as non-fatal polling. I guess this should not be a big work, you can just modify the new telem interfaces machine_check_poll?
part2: common handler part. (including both CMCI parts and non-fatal polling parts).

How do you think about it :-)

Thanks a lot for your help!
Criping

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Frank van der Linden
Sent: 2009年3月17日 7:28
To: [email protected]
Subject: [Xen-devel] [PATCH] re-work MCA telemetry internals; use common code for Intel/AMD MCA

The following patch reworks the MCA error telemetry handling inside Xen,
and shares code between the Intel and AMD implementations as much as
possible.

I've had this patch sitting around for a while, but it wasn't ported to -unstable yet. I finished porting and testing it, and am submitting it now, because the Intel folks want to go ahead and submit their new changes, so we agreed that I should push our changes first.

Brief explanation of the telemetry part: previously, the telemetry was accessed in a global array, with index variables used to access it. There were some issues with that: race conditions with regard to new machine checks (or CMCIs) coming in while handling the telemetry, and interaction with domains having been notified or not, which was a bit hairy. Our changes (I should say: Gavin Maltby's changes, as he did the bulk of this work for our 3.1 based tree, I merely ported/extended it to 3.3 and beyond) make telemetry access transactional (think of a database). Also, the internal database updates are atomic, since the final commit is done by a pointer swap. There is a brief explanation of the mechanism in mctelem.h.This patch also removes dom0->domU notification, which is ok, since Intel's upcoming changes will replace domU notification with a vMCE mechanism anyway.

The common code part is pretty much what it says. It defines a common MCE handler, with a few hooks for the special needs of the specific CPUs.

I've been told that Intel's upcoming patch will need to make some parts of the common code specific to the Intel CPU again, but we'll work together to use as much common code as possible.

- Frank

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#5 lore raw
In reply to Frank van der Linden

Hi, Frank

We did some small test here, found the CMCI problem is caused by the "mce_banks_owned" bitmap param passing. When CMCI happened, we print the bitmap (in smp_cmci_interrupt) value, it's correct (For cpu0, 16c). When passing into mcheck_mca_logout, it turned to be "0xFFFF~~FFF" which is wrong.
Only for your info -:)

Still, I suggest split the patch since this patch is realy big -:)

Thanks a lot for your help!
Criping

-----Original Message-----
From: Ke, Liping
Sent: 2009年3月17日 14:26
To: 'Frank van der Linden'; [email protected]
Subject: RE: [Xen-devel] [PATCH] re-work MCA telemetry internals; use common code for Intel/AMD MCA

Hi, Frank

I am now doing some tests based on latest Intel platform for this patch since CMCI needs some owner_checking and only the owned CPU will report the error.

Without the patch, when CMCI happened, since CPU0 is the owner of bank8, so when do checking, Only CPU0 will report the error.

Below is the correct log
(XEN) CMCI: cmci_intr happen on CPU3
[root@lke-ep inject]# (XEN) CMCI: cmci_intr happen on CPU2
(XEN) CMCI: cmci_intr happen on CPU0
(XEN) CMCI: cmci_intr happen on CPU1
(XEN) mcheck_poll: bank8 CPU0 status[cc0000800001009f]
(XEN) mcheck_poll: CPU0, SOCKET0, CORE0, APICID[0], thread[0]
(XEN) MCE: The hardware reports a non fatal, correctable incident occured on CPU
0.

After applied your patch, I found all CPUs will report the error. Below is the log
(XEN) MCE: The hardware reports a non fatal, correctable i
ncident occured on CPU 0.
(XEN) MCE: The hardware reports a non fatal, correctable incident occured on CPU
2.
(XEN) MCE: The hardware reports a non fatal, correctable incident occured on CPU
3.
(XEN) MCE: The hardware reports a non fatal, correctable incident occured on CPU
1.
(XEN) Bank 8: cc0000c00001009f<1>Bank 8: 8c0000400001009f<1>Bank 8: cc0001c00001 009f<1>MCE: The hardware reports a non fatal, correctable incident occured on CP U 0.

I noticed your patch has passed in the cmci_owner mask, I can't see the reason since this is really a big patch. I need some time to figure it out.

Also we found the polling mechanism has some changes. My feeling is that this patch is really too big. We can't easily figured out the impaction to our checked-in codes right now.
Just wonder whether you could split this big patch into two parts :-)
part1: mce log telem mechanism and required mce_intel interfaces changes. So that we can verify easily whether the new interfaces works fine for our CMCI as well as non-fatal polling. I guess this should not be a big work, you can just modify the new telem interfaces machine_check_poll?
part2: common handler part. (including both CMCI parts and non-fatal polling parts).

How do you think about it :-)

Thanks a lot for your help!
Criping

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Frank van der Linden
Sent: 2009年3月17日 7:28
To: [email protected]
Subject: [Xen-devel] [PATCH] re-work MCA telemetry internals; use common code for Intel/AMD MCA

The following patch reworks the MCA error telemetry handling inside Xen,
and shares code between the Intel and AMD implementations as much as
possible.

I've had this patch sitting around for a while, but it wasn't ported to -unstable yet. I finished porting and testing it, and am submitting it now, because the Intel folks want to go ahead and submit their new changes, so we agreed that I should push our changes first.

Brief explanation of the telemetry part: previously, the telemetry was accessed in a global array, with index variables used to access it. There were some issues with that: race conditions with regard to new machine checks (or CMCIs) coming in while handling the telemetry, and interaction with domains having been notified or not, which was a bit hairy. Our changes (I should say: Gavin Maltby's changes, as he did the bulk of this work for our 3.1 based tree, I merely ported/extended it to 3.3 and beyond) make telemetry access transactional (think of a database). Also, the internal database updates are atomic, since the final commit is done by a pointer swap. There is a brief explanation of the mechanism in mctelem.h.This patch also removes dom0->domU notification, which is ok, since Intel's upcoming changes will replace domU notification with a vMCE mechanism anyway.

The common code part is pretty much what it says. It defines a common MCE handler, with a few hooks for the special needs of the specific CPUs.

I've been told that Intel's upcoming patch will need to make some parts of the common code specific to the Intel CPU again, but we'll work together to use as much common code as possible.

- Frank

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#2 lore raw
In reply to Jiang, Yunhong
Quoted 61 lines · Yunhong
Quoted 28 lines · [email protected] <> wrote:

The following patch reworks the MCA error telemetry handling inside Xen,
and shares code between the Intel and AMD implementations as much as
possible.

I've had this patch sitting around for a while, but it wasn't ported to
-unstable yet. I finished porting and testing it, and am submitting it now, because the Intel folks want to go ahead and submit their new changes, so we agreed that I should push our changes first.

Brief explanation of the telemetry part: previously, the telemetry was accessed in a global array, with index variables used to access it. There were some issues with that: race conditions with regard to new machine checks (or CMCIs) coming in while handling the telemetry, and interaction with domains having been notified or not, which was a bit hairy. Our changes (I should say: Gavin Maltby's changes, as he did the
bulk of this work for our 3.1 based tree, I merely ported/extended it to
3.3 and beyond) make telemetry access transactional (think of a database). Also, the internal database updates are atomic, since the final commit is done by a pointer swap. There is a brief explanation of
the mechanism in mctelem.h.This patch also removes dom0->domU notification, which is ok, since Intel's upcoming changes will replace domU notification with a vMCE mechanism anyway.

Quoted 8 lines

The common code part is pretty much what it says. It defines a common MCE handler, with a few hooks for the special needs of the specific CPUs.

I've been told that Intel's upcoming patch will need to make some parts
of the common code specific to the Intel CPU again, but we'll work together to use as much common code as possible.

Yes, as shown in our previous patch, we do change the current MCA handler, the main changes are followed:

1) Most importantly, we implement a softIRQ mechanism for post MCE handler. The reason is, the #MC can happen in any time, that means: Firstly it is
spin-lock unsafe, some code like vcpu_schedule_lock_irq(v) in current MCA handler is sure to cause hang if that lock is already hold by a ISR; Secondly, the execution context is uncertain, the "current " value in current MCA handler maybe incorrect (if set_current is interrupted by #MC), the page ownership maybe wrong (if still in change under heap_lock protection) etc. I remember this So our patch handling #MC is in two step. The MCA handler, which depends on the execution context when MCA happen (like checking if it is in Xen context) and especially it will bring all CPU to softIRQ context. The softIRQ handler (i.e. post handler), which will be spin_lock safe, and all CPU is redenzvous, so it can take more actions.

2) We implement a mechanism to handle the shared MSR resources similar to what we have done in CMCI handler. As the Intel SDM stated, some MC resource is shared by multiple logical CPU, we implement a ownership check.

3) As stated in linux MCA handler, on Intel platforms machine check exceptions are always broadcast to all CPUs, we add such support also.

Quoted 8 lines

We have no idea how the issues for item 2 and 3 are handled on other platform, so we have no idea on how to do the common handler for it, hope Christoph can provide more suggestion, or we can just keep them different for different platform.

But I think for item 1, it is software related, so it can be a enhancement to the common handler, the only thing I'm not sure is, if we need bring all CPU to softIRQ context in all platform, maybe Christoph can give more idea.

The featureset of AMD Athlon K7 and Intel Pentium III are the common denominator on x86. This is what can go into the common code. In order to utilize features from newer cpus, allow to register function pointers and call them from the common code. Look into the amd_k8.c and amd_f10.c for example code. I register a function pointer to read the new MSRs. It can be easily extended to utilize features of coming CPUs.

Quoted 11 lines

Since we have get most consensus on the high level idea of MCA handler (i.e. Xen take action instead of dom0, use vMCE for guest MCA etc, check discussion with subject " [RFC] RAS(Part II)--MCA enalbing in XEN", the only thing left is the detail method of how to pass the recover action information to dom0), maybe we can turn to this second level discussion of how to enhance the (common) MCA handler.

Thanks
-- Yunhong Jiang

Quoted 1 line

- Frank

Signature · 6 lines
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Karl-Hammerschmidt-Str. 34, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Jochen Polster, Thomas M. McCoy, Giuliano Meroni
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632
#6 lore raw
In reply to Ke, Liping
Quoted 6 lines · Liping

Hi, Frank

We did some small test here, found the CMCI problem is caused by the "mce_banks_owned" bitmap param passing.
When CMCI happened, we print the bitmap (in smp_cmci_interrupt) value, it's correct (For cpu0, 16c). When passing into mcheck_mca_logout, it turned to be "0xFFFF~~FFF" which is wrong. Only for your info -:)

Still, I suggest split the patch since this patch is realy big -:)

Ok, I'll split up the patch.. the common code part will probably be still fairly big, though. Hopefully I can post it by the end of the day.

- Frank

#7 lore raw
In reply to Ke, Liping
Quoted 9 lines · Liping

Hi, Frank

We did some small test here, found the CMCI problem is caused by the "mce_banks_owned" bitmap param passing.
When CMCI happened, we print the bitmap (in smp_cmci_interrupt) value, it's correct (For cpu0, 16c). When passing into mcheck_mca_logout, it turned to be "0xFFFF~~FFF" which is wrong. Only for your info -:)

Still, I suggest split the patch since this patch is realy big -:)

Thanks a lot for your help!
Criping

Ok, since the patch is already in (thanks Keir!), let's work out any issues you may see with it.

When you say that the mce_banks_owned value is wrong in mcheck_mca_logout, do you mean that the parameter passing somehow went wrong? Is the value ok right before the call, but not in mcheck_mca_logout itself? That would be strange.

You mentioned the polling code: it didn't actually change much, in fact, the general polling code now is pretty much the same as the polling code that was originally in mce_intel.c. That code is very similar. Some AMD code contained AMD-specific things, and has not been touched (it still has its own poll function).

In general, the change is:

* machine_check_poll (going over the banks to look for non-fatal errors) and the loops inside the mcheck handlers that also walk the MC banks, have been combined in one function, which is mcheck_mca_logout. mcheck_mca_logout still takes a context argument, as machine_check_poll did.

* The mcheck handlers themselves mostly just call mcheck_cmn_handler, which calls mcheck_mca_logout, looks at the context, and then decides what to do.

* The mctelem structure as returned by mcheck_mca_logout, contains the relevant telemetry for the current error. If it should be put in the "database" for dom0 retrieval, use mctelem_commit to put it there. Otherwise, the usual course of action is to print the values and uses mctelem_dismiss to discard the information.

There is one thing that needs to be addressed: in your Intel-specific code, you do not reset the per-bank status in the case of a fatal error, so that the BIOS may look at the MSRs. mcheck_mca_logout does always reset the values. The best way to fix this is probably to pass in a flag to mcheck_mca_logout to tell it whether it should reset the status MSRs or not. The caller can then decide if they should be reset later.

Let me know if you have any more issues, I'll be happy to help merge your code and test it.

- Frank

#3 lore raw
In reply to Christoph Egger
Quoted 67 lines · Christoph Egger
Quoted 57 lines · Yunhong
Quoted 25 lines · [email protected] <> wrote:

The following patch reworks the MCA error telemetry handling inside Xen,
and shares code between the Intel and AMD implementations as much as
possible.

I've had this patch sitting around for a while, but it wasn't ported to -unstable yet. I finished porting and testing it, and am submitting it now, because the Intel folks want to go ahead and submit their new changes, so we agreed that I should push our changes first.

Brief explanation of the telemetry part: previously, the telemetry was accessed in a global array, with index variables used to access it. There were some issues with that: race conditions with regard to new machine checks (or CMCIs) coming in while handling the telemetry, and interaction with domains having been notified or not, which was a bit hairy. Our changes (I should say: Gavin Maltby's changes, as he did the
bulk of this work for our 3.1 based tree, I merely ported/extended it to
3.3 and beyond) make telemetry access transactional (think of a database). Also, the internal database updates are atomic, since the final commit is done by a pointer swap. There is a brief explanation of the mechanism in mctelem.h.This patch also removes dom0->domU notification, which is ok, since Intel's upcoming changes will replace domU notification with a vMCE mechanism anyway.

Quoted 7 lines

The common code part is pretty much what it says. It defines a common MCE handler, with a few hooks for the special needs of the specific CPUs.

I've been told that Intel's upcoming patch will need to make some parts
of the common code specific to the Intel CPU again, but we'll work together to use as much common code as possible.

Yes, as shown in our previous patch, we do change the current MCA handler, the main changes are followed:

1) Most importantly, we implement a softIRQ mechanism for post MCE handler. The reason is, the #MC can happen in any time, that means: Firstly it is
spin-lock unsafe, some code like vcpu_schedule_lock_irq(v) in current MCA handler is sure to cause hang if that lock is already hold by a ISR; Secondly, the execution context is uncertain, the "current " value in current MCA handler maybe incorrect (if set_current is interrupted by #MC), the page ownership maybe wrong (if still in change under heap_lock protection) etc. I remember this So our patch handling #MC is in two step. The MCA handler, which depends on the execution context when MCA happen (like checking if it is in Xen context) and especially it will bring all CPU to softIRQ context. The softIRQ handler (i.e. post handler), which will be spin_lock safe, and all CPU is redenzvous, so it can take more actions.

2) We implement a mechanism to handle the shared MSR resources similar to what we have done in CMCI handler. As the Intel SDM stated, some MC resource is shared by multiple logical CPU, we implement a ownership check.

3) As stated in linux MCA handler, on Intel platforms machine check exceptions are always broadcast to all CPUs, we add such support also.

Quoted 8 lines

We have no idea how the issues for item 2 and 3 are handled on other platform, so we have no idea on how to do the common handler for it, hope Christoph can provide more suggestion, or we can just keep them different for different platform.

But I think for item 1, it is software related, so it can be a enhancement to the common handler, the only thing I'm not sure is, if we need bring all CPU to softIRQ context in all platform, maybe Christoph can give more idea.

Quoted 7 lines

The featureset of AMD Athlon K7 and Intel Pentium III are the common denominator on x86. This is what can go into the common code. In order to utilize features from newer cpus, allow to
register function
pointers and call them from the common code. Look into the amd_k8.c and amd_f10.c for example code. I register a function pointer to read the new MSRs. It can be easily extended to utilize features of coming CPUs.

I suspect the difference between the mce_intel.c and amd_xxx.c will be far more than the difference between amd_k8.c and amd_f10.c. For example, the issue 2/3 listed above (will it exists on your side?)

And how about the softIRQ mechanism? How do you think about apply it to both side?

Thanks
Yunhong Jiang

Quoted 22 lines
Quoted 9 lines

Since we have get most consensus on the high level idea of MCA handler (i.e. Xen take action instead of dom0, use vMCE for guest MCA etc, check discussion with subject " [RFC] RAS(Part II)--MCA enalbing in XEN", the only thing left is the detail method of how to pass the recover action information to dom0), maybe we can turn to this second level discussion of how to enhance the (common) MCA handler.

Quoted 4 lines

Thanks
-- Yunhong Jiang

Quoted 1 line

- Frank

--
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Karl-Hammerschmidt-Str. 34, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Jochen Polster, Thomas M. McCoy, Giuliano Meroni
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen Registergericht Muenchen, HRB Nr. 43632

#8 lore raw
In reply to Frank van der Linden
Quoted 60 lines · [email protected] <> wrote:
Quoted 3 lines · Liping

Hi, Frank

We did some small test here, found the CMCI problem is

caused by the "mce_banks_owned" bitmap param passing.

Quoted 1 line

When CMCI happened, we print the bitmap (in

smp_cmci_interrupt) value, it's correct (For cpu0, 16c).

Quoted 7 lines

When passing into mcheck_mca_logout, it turned to be "0xFFFF~~FFF" which is wrong. Only for your info -:)

Still, I suggest split the patch since this patch is realy big -:)

Thanks a lot for your help!
Criping

Ok, since the patch is already in (thanks Keir!), let's work out any issues you may see with it.

When you say that the mce_banks_owned value is wrong in mcheck_mca_logout, do you mean that the parameter passing somehow went wrong? Is the value ok right before the call, but not in mcheck_mca_logout itself? That would be strange.

You mentioned the polling code: it didn't actually change much, in fact,
the general polling code now is pretty much the same as the polling code
that was originally in mce_intel.c. That code is very similar. Some AMD
code contained AMD-specific things, and has not been touched (it still has its own poll function).

In general, the change is:

* machine_check_poll (going over the banks to look for
non-fatal errors)
and the loops inside the mcheck handlers that also walk the MC banks, have been combined in one function, which is mcheck_mca_logout. mcheck_mca_logout still takes a context argument, as machine_check_poll did.

* The mcheck handlers themselves mostly just call mcheck_cmn_handler, which calls mcheck_mca_logout, looks at the context, and then decides what to do.

* The mctelem structure as returned by mcheck_mca_logout, contains the relevant telemetry for the current error. If it should be put in the "database" for dom0 retrieval, use mctelem_commit to put it there. Otherwise, the usual course of action is to print the values and uses mctelem_dismiss to discard the information.

There is one thing that needs to be addressed: in your Intel-specific code, you do not reset the per-bank status in the case of a fatal error,
so that the BIOS may look at the MSRs. mcheck_mca_logout does always reset the values. The best way to fix this is probably to pass in a flag
to mcheck_mca_logout to tell it whether it should reset the status MSRs
or not. The caller can then decide if they should be reset later.

Quoted 2 lines

Let me know if you have any more issues, I'll be happy to help merge your code and test it.

We will do more test for it and also check the per-bank status. Originally we hope:

a) settle down the interface in 3.4 release for MCA, so that it will not be dramatically changed anymore, including the telemetry interface for CE and the interface betwwen MCA handler and softIRQ handler (i.e. what we get consensused in the mailing list), and your telemetry changes that already in. since usually people based their work on some stable release.

b) If we can get consensus, finished the softIRQ mechanism so that it will get a solid base. The spin_lock() and "current" usage in MCA handler is really confusing.

But feature freeze now :-(

Or maybe someone can lobby Keir to accept these changes if Ke Liping and I can finish these patches this week. Hope it should not be big if we can remove the vMCE implementation cleanly.

Quoted 9 lines

- Frank

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#9 lore raw
Quoted 5 lines · Yunhong

But feature freeze now :-(

Or maybe someone can lobby Keir to accept these changes if Ke Liping and I can finish these patches this week. Hope it should not be big if we can remove the vMCE implementation cleanly.

If there's a strong argument for it, we can stretch the feature window. Particularly for something which is, frankly for now at least, a niche feature.

-- Keir