Skip to contentSkip to conversation

panic: Don't print redundant backtraces on oops

PATCH In review Andi Kleen 4 participants 4 messages Dec 8, 2011 – Dec 19, 2011

lore
Share & tools
lore

Andi Kleen

author
#0 lore raw

PATCH 1/1 panic: Don't print redundant backtraces on oops

From: Andi Kleen <[email protected]>

When an oops causes a panic and panic prints another backtrace it's pretty common to have the original oops data be scrolled away on a 80x50 screen.

The second backtrace is quite redundant and not needed anyways.

So don't print the panic backtrace when oops_in_progress is true.

---
 kernel/panic.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
kernel/panic.c +2 -1
index b2659360..398412b 100644
@@ -78,7 +78,8 @@ NORET_TYPE void panic(const char * fmt, ...)
 	va_end(args);
 	printk(KERN_EMERG "Kernel panic - not syncing: %s\n",buf);
 #ifdef CONFIG_DEBUG_BUGVERBOSE
-	dump_stack();
+	if (!oops_in_progress)
+		dump_stack();
 #endif
 
 	/*
Signature · 1 line
1.7.4.4
Acked-by Chris Metcalf on v1 1/1
#1 lore raw
Quoted 20 lines · On Wed, 7 Dec 2011 16:36:43 -0800 Andi Kleen <[email protected]> wrote:

From: Andi Kleen <[email protected]>

When an oops causes a panic and panic prints another backtrace it's pretty common to have the original oops data be scrolled away on a 80x50 screen.

The second backtrace is quite redundant and not needed anyways.

So don't print the panic backtrace when oops_in_progress is true.

---
 kernel/panic.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
kernel/panic.c +0 -0
index b2659360..398412b 100644

kernel/panic.c NORET_TYPE void panic(const char * fmt, ...) · line 83

Quoted 7 lines
kernel/panic.c +2 -1
@@ -78,7 +78,8 @@ NORET_TYPE void panic(const char * fmt, ...)
 	va_end(args);
 	printk(KERN_EMERG "Kernel panic - not syncing: %s\n",buf);
 #ifdef CONFIG_DEBUG_BUGVERBOSE
-	dump_stack();
+	if (!oops_in_progress)
+		dump_stack();
 #endif

This is kinda related to Michael's kdump-fix-crash_kexec-smp_send_stop-race-in-panic.patch, below.

afacit Michael's patch will prevent panic-within-panic, and it does this by accident becasue we never thought about it. But it won't fix panic-within-other-oops.

Is there some clever way in which we can satisfy both requirements in one hit?

From: Michael Holzheu <[email protected]>
Subject: kdump: fix crash_kexec()/smp_send_stop() race in panic()

When two CPUs call panic at the same time there is a possible race condition that can stop kdump. The first CPU calls crash_kexec() and the second CPU calls smp_send_stop() in panic() before crash_kexec() finished on the first CPU. So the second CPU stops the first CPU and therefore kdump fails:

1st CPU:
panic()->crash_kexec()->mutex_trylock(&kexec_mutex)-> do kdump

2nd CPU:
panic()->crash_kexec()->kexec_mutex already held by 1st CPU

       ->smp_send_stop()-> stop 1st CPU (stop kdump)

This patch fixes the problem by introducing a spinlock in panic that allows only one CPU to process crash_kexec() and the subsequent panic code.

All other CPUs call the weak function panic_smp_self_stop() that stops the CPU itself. This function can be overloaded by architecture code. For example "tile" can use their lower-power "nap" instruction for that.

---

 kernel/panic.c |   18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)
kernel/panic.c renamed from kernel/panic.c~kdump-fix-crash_kexec-smp_send_stop-race-in-panic +17 -1
diff -puN kernel/panic.c~kdump-fix-crash_kexec-smp_send_stop-race-in-panic kernel/panic.c
@@ -49,6 +49,15 @@ static long no_blink(int state)
 long (*panic_blink)(int state);
 EXPORT_SYMBOL(panic_blink);
 
+/*
+ * Stop ourself in panic -- architecture code may override this
+ */
+void __weak panic_smp_self_stop(void)
+{
+	while (1)
+		cpu_relax();
+}
+
 /**
  *	panic - halt the system
  *	@fmt: The text string to print
@@ -59,6 +68,7 @@ EXPORT_SYMBOL(panic_blink);
  */
 void panic(const char *fmt, ...)
 {
+	static DEFINE_SPINLOCK(panic_lock);
 	static char buf[1024];
 	va_list args;
 	long i, i_next = 0;
@@ -68,8 +78,14 @@ void panic(const char *fmt, ...)
 	 * It's possible to come here directly from a panic-assertion and
 	 * not have preempt disabled. Some functions called from here want
 	 * preempt to be disabled. No point enabling it later though...
+	 *
+	 * Only one CPU is allowed to execute the panic code from here. For
+	 * multiple parallel invocations of panic, all other CPUs either
+	 * stop themself or will wait until they are stopped by the 1st CPU
+	 * with smp_send_stop().
 	 */
-	preempt_disable();
+	if (!spin_trylock(&panic_lock))
+		panic_smp_self_stop();
 
 	console_verbose();
 	bust_spinlocks(1);

_

#2 lore raw

Hello Andrew,

Quoted 2 lines · Andrew Morton

On Wed, 7 Dec 2011 16:36:43 -0800
Andi Kleen <[email protected]> wrote:

[snip]

Quoted 14 lines

kernel/panic.c NORET_TYPE void panic(const char * fmt, ...) · line 83

Quoted 10 lines
kernel/panic.c +2 -1
@@ -78,7 +78,8 @@ NORET_TYPE void panic(const char * fmt, ...)
 	va_end(args);
 	printk(KERN_EMERG "Kernel panic - not syncing: %s\n",buf);
 #ifdef CONFIG_DEBUG_BUGVERBOSE
-	dump_stack();
+	if (!oops_in_progress)
+		dump_stack();
 #endif

This is kinda related to Michael's kdump-fix-crash_kexec-smp_send_stop-race-in-panic.patch, below.

Quoted 6 lines

afacit Michael's patch will prevent panic-within-panic, and it does this by accident becasue we never thought about it. But it won't fix panic-within-other-oops.

Is there some clever way in which we can satisfy both requirements in one hit?

I think the two problems are not related. My problem was that only one CPU should execute panic. Andi's problem is that one (or more oops) messages are scrolled away by a subsequent panic.

But I am not sure if Andi's patch solves all his problems. What e.g. about panic_on_oom? Don't we have the same problem here?

Michael

#3 lore raw

it?

Quoted 3 lines

I think the two problems are not related. My problem was that only one CPU should execute panic. Andi's problem is that one (or more oops) messages are scrolled away by a subsequent panic.

Yes it's just for making panic less chatty.

Quoted 2 lines

But I am not sure if Andi's patch solves all his problems. What e.g. about panic_on_oom? Don't we have the same problem here?

Yes, it only solves it for this specific -- but common -- case. Or at least it was a case that annoyed me and prompted me to write the patch :)

I had a patch some time ago for a xpanic() where the caller could pass in additional flags, including one to disable the backtrace. But that was a much larger patch, this was a simpler alternative.

-Andi