Skip to contentSkip to conversation

powernv: Avoid checkstop on HMI and MCE

PATCH In review Michael Neuling 4 participants 5 messages Oct 24, 2017 – Oct 25, 2017 PowerPC

lore
Share & tools
lore

Michael Neuling

author
#0 lore raw

PATCH 1/1 powernv: Avoid checkstop on HMI and MCE

On an unrecoverable HMI or MCE only generate an checkstop (via PLATFORM ERROR opal reboot call) when panic_on_oops is set.

We currently generate an checkstop as an attempt for the FSP to grab a dump and then reboot us. Unfortunately this never works and no one I've talked to has ever seen a resulting dump, let alone got useful information from it.

Even worse, the checkstop gets in the way of debugging real problems. If we hit a software bug that results in this, we get no opportunity to debug it live. Similarly if the bug is due to hardware that is not in the dump (say PCI or NVLINK GPU), we get no information in the dump about that hardware.

So let's remove it unless someone sets panic_on_oops.

Signed-off-by Michael Neuling <[email protected]>
---
 arch/powerpc/platforms/powernv/opal-hmi.c | 6 ++++++
 arch/powerpc/platforms/powernv/opal.c     | 4 ++++
 2 files changed, 10 insertions(+)
arch/powerpc/platforms/powernv/opal-hmi.c +6 -0
index c9e1a4ff29..23780970d0 100644
@@ -29,6 +29,7 @@
 #include <asm/opal.h>
 #include <asm/cputable.h>
 #include <asm/machdep.h>
+#include <asm/bug.h>
 
 #include "powernv.h"
 

@@ -284,6 +285,11 @@ static void hmi_event_handler(struct work_struct *work)
 			print_hmi_event_info(hmi_evt);
 		}
 
+		if (!panic_on_oops) {
+			die("Unrecoverable HMI exception", NULL, SIGBUS);
+			return;
+		}
+
 		pnv_platform_error_reboot(NULL, "Unrecoverable HMI exception");
 	}
 }
arch/powerpc/platforms/powernv/opal.c +4 -0
index 65c79ecf5a..f208f222e4 100644
@@ -523,6 +523,10 @@ int opal_machine_check(struct pt_regs *regs)
 	if (opal_recover_mce(regs, &evt))
 		return 1;
 
+	if (!panic_on_oops) {
+		die("Unrecoverable Machine check", regs, SIGBUS);
+		return 0;
+	}
 	pnv_platform_error_reboot(regs, "Unrecoverable Machine Check exception");
 }
Signature · 1 line
2.14.1
reviewer
#1 lore raw
Quoted 5 lines · Michael Neuling

On an unrecoverable HMI or MCE only generate an checkstop (via PLATFORM ERROR opal reboot call) when panic_on_oops is set.

We currently generate an checkstop as an attempt for the FSP to grab a dump and then reboot us. Unfortunately this never works and no one

Never? WT#.

Quoted 10 lines

I've talked to has ever seen a resulting dump, let alone got useful information from it.

Even worse, the checkstop gets in the way of debugging real problems. If we hit a software bug that results in this, we get no opportunity to debug it live. Similarly if the bug is due to hardware that is not in the dump (say PCI or NVLINK GPU), we get no information in the dump about that hardware.

So let's remove it unless someone sets panic_on_oops.

Nick just rewrote pnv_platform_error_reboot(), so please talk to him to make sure you're not stepping on each other.

arch/powerpc/platforms/powernv/opal-hmi.c static void hmi_event_handler(struct work_struct *work) · line 290

Quoted 11 lines
arch/powerpc/platforms/powernv/opal-hmi.c +3 -0
index c9e1a4ff29..23780970d0 100644
@@ -284,6 +285,11 @@ static void hmi_event_handler(struct work_struct *work)
 			print_hmi_event_info(hmi_evt);
 		}
 
+		if (!panic_on_oops) {
+			die("Unrecoverable HMI exception", NULL, SIGBUS);
+			return;

I don't think we should return.

Otherwise we risk persisting corrupt data to disk and so on.

If we're getting unrecoverable HMI/MCEs that are not actually indicative of something bad happening then we need to filter those out somewhere.

cheers

author
#2 lore raw
Quoted 9 lines · Michael Ellerman

Michael Neuling <[email protected]> writes:
=20

Quoted 5 lines

On an unrecoverable HMI or MCE only generate an checkstop (via PLATFORM ERROR opal reboot call) when panic_on_oops is set. =20
We currently generate an checkstop as an attempt for the FSP to grab a dump and then reboot us. Unfortunately this never works and no one

=20
Never? WT#.

Well no one I've talked but I'm posting this so someone will stand up and s= ay
they want it.

Quoted 10 lines
Quoted 10 lines

I've talked to has ever seen a resulting dump, let alone got useful information from it.
=20
Even worse, the checkstop gets in the way of debugging real problems. If we hit a software bug that results in this, we get no opportunity to debug it live. Similarly if the bug is due to hardware that is not in the dump (say PCI or NVLINK GPU), we get no information in the dump about that hardware.
=20
So let's remove it unless someone sets panic_on_oops.

Quoted 3 lines

=20
Nick just rewrote pnv_platform_error_reboot(), so please talk to him to make sure you're not stepping on each other.

OK, will do.

Quoted 6 lines

arch/powerpc/platforms/powernv/opal-hmi.c static void hmi_event_handler(struct work_struct *=

Quoted 6 lines
arch/powerpc/platforms/powernv/opal-hmi.c +0 -0
index c9e1a4ff29..23780970d0 100644
@@ -284,6 +285,11 @@ static void hmi_event_handler(struct work_struct *=

work)

Quoted 10 lines

arch/powerpc/platforms/powernv/opal-hmi.c static void hmi_event_handler(struct work_struct *work) · line 290

Quoted 6 lines
arch/powerpc/platforms/powernv/opal-hmi.c +0 -0
 			print_hmi_event_info(hmi_evt);
 		}

=20

arch/powerpc/platforms/powernv/opal-hmi.c +3 -0
+		if (!panic_on_oops) {
+			die("Unrecoverable HMI exception", NULL, SIGBUS);
+			return;

=20
I don't think we should return.
=20
Otherwise we risk persisting corrupt data to disk and so on.

ok

Quoted 2 lines

If we're getting unrecoverable HMI/MCEs that are not actually indicative of something bad happening then we need to filter those out somewhere.

We hit this with some new HMIs for NVLINK and the Vector Load one, so we ne= ed to
handle them, and we have code that does (or is coming).

In the mean while, it's very hard to debug them once we xstop.

Mikey

#3 lore raw

cc'ing skiboot

Quoted 23 lines · On Wed, 25 Oct 2017 21:59:42 +1100 Michael Neuling <[email protected]> wrote:
Quoted 9 lines · Michael Ellerman
Quoted 5 lines · Michael Neuling

On an unrecoverable HMI or MCE only generate an checkstop (via PLATFORM ERROR opal reboot call) when panic_on_oops is set.

We currently generate an checkstop as an attempt for the FSP to grab a dump and then reboot us. Unfortunately this never works and no one

Never? WT#.

Well no one I've talked but I'm posting this so someone will stand up and say they want it.

Quoted 9 lines
Quoted 9 lines

I've talked to has ever seen a resulting dump, let alone got useful information from it.

Even worse, the checkstop gets in the way of debugging real problems. If we hit a software bug that results in this, we get no opportunity to debug it live. Similarly if the bug is due to hardware that is not in the dump (say PCI or NVLINK GPU), we get no information in the dump about that hardware.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

So let's remove it unless someone sets panic_on_oops.

Nick just rewrote pnv_platform_error_reboot(), so please talk to him to make sure you're not stepping on each other.

OK, will do.

I've got nothing really.

I couldn't find out how to do something useful with the crash. My patches were along the same line as this, which was to make more Linux crash data printed and try to avoid panic'ing the system when we can recover. Apparently it does set a firmware log.

Only thing I thought is that it's not really an oops, but a hardware error (except when it is a software error/oops). I thought it fits panic_on_unrecovered_nmi better (or we make our own one since that looks like it's x86 only).

We just need to go through and think about behaviour we want and make it all consistent. For example right now if we think we can recover the system because there's an unrecoverable machine check in process context, then we try to just kill the process (opal_recover_mce()). Someone who sets the panic_on_ here would like to panic in those cases too probably.

Quoted 20 lines
Quoted 10 lines
Quoted 10 lines
arch/powerpc/platforms/powernv/opal-hmi.c +0 -0
index c9e1a4ff29..23780970d0 100644
@@ -284,6 +285,11 @@ static void hmi_event_handler(struct work_struct *work)
 			print_hmi_event_info(hmi_evt);
 		}
Quoted 7 lines

arch/powerpc/platforms/powernv/opal-hmi.c static void hmi_event_handler(struct work_struct *work) · line 290

Quoted 3 lines
arch/powerpc/platforms/powernv/opal-hmi.c +3 -0
+		if (!panic_on_oops) {
+			die("Unrecoverable HMI exception", NULL, SIGBUS);
+			return;

I don't think we should return.

Otherwise we risk persisting corrupt data to disk and so on.

ok

Quoted 7 lines
Quoted 2 lines

If we're getting unrecoverable HMI/MCEs that are not actually indicative of something bad happening then we need to filter those out somewhere.

We hit this with some new HMIs for NVLINK and the Vector Load one, so we need to handle them, and we have code that does (or is coming).

In the mean while, it's very hard to debug them once we xstop.

The pnv_platform_error_reboot() wasn't able to print anything to console in that case? I'd like to try improve that if it's not working, because even in the panic case we should try to get something useful to console.

Thanks,
Nick

#4 lore raw
In reply to Michael Neuling
Quoted 35 lines · On Tue, 24 Oct 2017 20:20:05 +1100 Michael Neuling <[email protected]> wrote:

On an unrecoverable HMI or MCE only generate an checkstop (via PLATFORM ERROR opal reboot call) when panic_on_oops is set.

We currently generate an checkstop as an attempt for the FSP to grab a dump and then reboot us. Unfortunately this never works and no one I've talked to has ever seen a resulting dump, let alone got useful information from it.

Even worse, the checkstop gets in the way of debugging real problems. If we hit a software bug that results in this, we get no opportunity to debug it live. Similarly if the bug is due to hardware that is not in the dump (say PCI or NVLINK GPU), we get no information in the dump about that hardware.

So let's remove it unless someone sets panic_on_oops.

Signed-off-by Michael Neuling <[email protected]>
---
 arch/powerpc/platforms/powernv/opal-hmi.c | 6 ++++++
 arch/powerpc/platforms/powernv/opal.c     | 4 ++++
 2 files changed, 10 insertions(+)
arch/powerpc/platforms/powernv/opal-hmi.c +1 -0
index c9e1a4ff29..23780970d0 100644
@@ -29,6 +29,7 @@
 #include <asm/opal.h>
 #include <asm/cputable.h>
 #include <asm/machdep.h>
+#include <asm/bug.h>
 
 #include "powernv.h"

arch/powerpc/platforms/powernv/opal-hmi.c static void hmi_event_handler(struct work_struct *work) · line 292

Quoted 8 lines
arch/powerpc/platforms/powernv/opal-hmi.c +5 -0
@@ -284,6 +285,11 @@ static void hmi_event_handler(struct work_struct *work)
 			print_hmi_event_info(hmi_evt);
 		}
 
+		if (!panic_on_oops) {
+			die("Unrecoverable HMI exception", NULL, SIGBUS);
+			return;
+		}
+

If panic_on_oops is set, we checkstop, not panic! Passing NULL to die, will cause arch_uprobe_exception_notify() to complain.

We could respin this a bit and I can send an updated patch if there is interest

Balbir Singh.