Skip to contentSkip to conversation

powerpc/powernv/mce: Don't silently restart the machine

RFC In review Balbir Singh 3 participants 4 messages Feb 21, 2018 PowerPC

lore
Share & tools
lore
#0 lore raw

RFC 1/1 powerpc/powernv/mce: Don't silently restart the machine

On MCE the current code will restart the machine with ppc_md.restart(). This case was extremely unlikely since prior to that a skiboot call is made and that resulted in a checkstop for analysis.

With newer skiboots, on P9 we don't checkstop the box by default, instead we return back to the kernel to extract useful information at the time of the MCE. While we still get this information, this patch converts the restart to a panic(), so that if configured a dump can be taken and we can track and probably debug the potential issue causing the MCE.

---
 arch/powerpc/platforms/powernv/opal.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
arch/powerpc/platforms/powernv/opal.c +4 -1
index 69b5263fc9e3..b510a6f41b00 100644
@@ -500,9 +500,12 @@ void pnv_platform_error_reboot(struct pt_regs *regs, const char *msg)
 	 *    opal to trigger checkstop explicitly for error analysis.
 	 *    The FSP PRD component would have already got notified
 	 *    about this error through other channels.
+	 * 4. We are running on a newer skiboot that by default does
+	 *    not cause a checkstop, drops us back to the kernel to
+	 *    extract context and state at the time of the error.
 	 */
 
-	ppc_md.restart(NULL);
+	panic("PowerNV Unrecovered Machine Check");
 }
 
 int opal_machine_check(struct pt_regs *regs)
Signature · 1 line
2.13.6
reviewer Reviewed-by on v1 1/1
#1 lore raw
Quoted 13 lines · On Wed, 21 Feb 2018 12:01:11 +1100 Balbir Singh <[email protected]> wrote:

On MCE the current code will restart the machine with ppc_md.restart(). This case was extremely unlikely since prior to that a skiboot call is made and that resulted in a checkstop for analysis.

With newer skiboots, on P9 we don't checkstop the box by default, instead we return back to the kernel to extract useful information at the time of the MCE. While we still get this information, this patch converts the restart to a panic(), so that if configured a dump can be taken and we can track and probably debug the potential issue causing the MCE.

Quoted 1 line

Seems like something we should be doing.

Quoted 22 lines
---
 arch/powerpc/platforms/powernv/opal.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
arch/powerpc/platforms/powernv/opal.c +4 -1
index 69b5263fc9e3..b510a6f41b00 100644
@@ -500,9 +500,12 @@ void pnv_platform_error_reboot(struct pt_regs *regs, const char *msg)
 	 *    opal to trigger checkstop explicitly for error analysis.
 	 *    The FSP PRD component would have already got notified
 	 *    about this error through other channels.
+	 * 4. We are running on a newer skiboot that by default does
+	 *    not cause a checkstop, drops us back to the kernel to
+	 *    extract context and state at the time of the error.
 	 */
 
-	ppc_md.restart(NULL);
+	panic("PowerNV Unrecovered Machine Check");
 }
 
 int opal_machine_check(struct pt_regs *regs)
#2 lore raw
In reply to Balbir Singh
Quoted 5 lines · Balbir Singh

On MCE the current code will restart the machine with ppc_md.restart(). This case was extremely unlikely since prior to that a skiboot call is made and that resulted in a checkstop for analysis.

Quoted 7 lines

With newer skiboots, on P9 we don't checkstop the box by default, instead we return back to the kernel to extract useful information at the time of the MCE. While we still get this information, this patch converts the restart to a panic(), so that if configured a dump can be taken and we can track and probably debug the potential issue causing the MCE.

I agree with the patch, although I'd be nervous stating that skiboot is going to keep this behaviour. In *theory* we should only ever get a platform error when there's actually something that isn't the kernel's fault.

Like any firmware promise though, it's slightly less reliable than one from a politician.

I'd say that in this case deferring to policy on what to do in event of panic() is the right thing.

Signature · 2 lines
Stewart Smith
OPAL Architect, IBM.
#3 lore raw
Quoted 19 lines · Stewart Smith
Quoted 5 lines · Balbir Singh

On MCE the current code will restart the machine with ppc_md.restart(). This case was extremely unlikely since prior to that a skiboot call is made and that resulted in a checkstop for analysis.

Quoted 7 lines

With newer skiboots, on P9 we don't checkstop the box by default, instead we return back to the kernel to extract useful information at the time of the MCE. While we still get this information, this patch converts the restart to a panic(), so that if configured a dump can be taken and we can track and probably debug the potential issue causing the MCE.

I agree with the patch, although I'd be nervous stating that skiboot is going to keep this behaviour. In *theory* we should only ever get a platform error when there's actually something that isn't the kernel's fault.

Quoted 6 lines

Like any firmware promise though, it's slightly less reliable than one from a politician.

I'd say that in this case deferring to policy on what to do in event of panic() is the right thing.

Your right, except that with NPUs and coherent device memory, things change. It is a platform error on the device side, the large issue is that the CPU touched this memory. Ideally we want the device and device driver to handle this error, but its racy as the CPU now treats it as a platform error on the system side. IOW, the definition of platform has grown and so has the definition of platform error and it can no longer be solely contained inside of one boxes firmware.

Balbir Singh.