Skip to contentSkip to conversation

overlaping printk

DISCUSSION Ingo Molnar 4 participants 7 messages May 20, 2004 – May 21, 2004 3 messages not in the archive

Subsystems (3)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw
Quoted 14 lines · Ingo Molnar
Quoted 2 lines · Ricky Beam

It looks like somewhere in the path of release_console_sem() more than one CPU is running the log. [...]

the problem is this code in printk:

        if (oops_in_progress) {
                /* If a crash is occurring, make sure we can't deadlock */
                spin_lock_init(&logbuf_lock);
                /* And make sure that we print immediately */
                init_MUTEX(&console_sem);
        }
Quoted 3 lines

so two crashes on two separate CPUs can go on in parallel. The problem is not constrained to the serial line - i've seen it on VGA too (albeit there it's much more rare).

One alternative way would be to use locks with timeouts for these two locks (e.g. checking the TSC on x86, since the timer interrupt may not be running anymore) and only break the lock when the wait time is too long.

Of course serial lines can be quite slow so even that may not help always (for unknown reasons far too many people use 9600 baud for their serial line)

-Andi

#4 lore raw
Quoted 8 lines · Andi Kleen

One alternative way would be to use locks with timeouts for these two locks (e.g. checking the TSC on x86, since the timer interrupt may not be running anymore) and only break the lock when the wait time is too long.

Of course serial lines can be quite slow so even that may not help always (for unknown reasons far too many people use 9600 baud for their serial line)

another solution would be to break the lock only once during the kernel's lifetime. The system is messed up anyway if it needs multiple lock breaks to get an oops out to the console. We dont care about followup oopses - the first oops is that matters.

Ingo

#5 lore raw
Quoted 4 lines · Ingo Molnar

another solution would be to break the lock only once during the kernel's lifetime. The system is messed up anyway if it needs multiple lock breaks to get an oops out to the console. We dont care about followup oopses - the first oops is that matters.

i.e. something like the attached patch, against BK-curr. (i've also attached a cleanup patch that gets rid of the many instances of bust_spinlocks() - we now have a generic one in lib/bust_spinlocks.c)

i consider any secondary lockup after the first oops has been printed a feature - sometimes the first oops gets washed away by the many followup oopses.

i've tested the patch with parallel SMP oopses - they seem to be serialized now. (but it's hard to time the oopses right.)

Ingo

kernel/printk.c renamed from kernel/printk.c.orig +4 -1
@@ -55,6 +55,9 @@ EXPORT_SYMBOL(console_printk);
 
 int oops_in_progress;
 
+/* zap spinlocks only once: */
+unsigned long zap_spinlocks = 1;
+
 /*
  * console_sem protects the console_drivers list, and also
  * provides serialisation for access to the entire console
@@ -493,7 +496,7 @@ asmlinkage int printk(const char *fmt, .
 	static char printk_buf[1024];
 	static int log_level_unknown = 1;
 
-	if (oops_in_progress) {
+	if (oops_in_progress && test_and_clear_bit(0, &zap_spinlocks)) {
 		/* If a crash is occurring, make sure we can't deadlock */
 		spin_lock_init(&logbuf_lock);
 		/* And make sure that we print immediately */
arch/i386/mm/fault.c renamed from arch/i386/mm/fault.c.orig +0 -26
@@ -31,32 +31,6 @@
 extern void die(const char *,struct pt_regs *,long);
 
 /*
- * Unlock any spinlocks which will prevent us from getting the
- * message out 
- */
-void bust_spinlocks(int yes)
-{
-	int loglevel_save = console_loglevel;
-
-	if (yes) {
-		oops_in_progress = 1;
-		return;
-	}
-#ifdef CONFIG_VT
-	unblank_screen();
-#endif
-	oops_in_progress = 0;
-	/*
-	 * OK, the message is on the console.  Now we call printk()
-	 * without oops_in_progress set so that printk will give klogd
-	 * a poke.  Hold onto your hats...
-	 */
-	console_loglevel = 15;		/* NMI oopser may have shut the console up */
-	printk(" ");
-	console_loglevel = loglevel_save;
-}
-
-/*
  * Return EIP plus the CS segment base.  The segment limit is also
  * adjusted, clamped to the kernel/user address space (whichever is
  * appropriate), and returned in *eip_limit.
arch/ia64/kernel/traps.c renamed from arch/ia64/kernel/traps.c.orig +0 -28
@@ -58,34 +58,6 @@ trap_init (void)
 		fpswa_interface = __va(ia64_boot_param->fpswa);
 }
 
-/*
- * Unlock any spinlocks which will prevent us from getting the message out (timerlist_lock
- * is acquired through the console unblank code)
- */
-void
-bust_spinlocks (int yes)
-{
-	int loglevel_save = console_loglevel;
-
-	if (yes) {
-		oops_in_progress = 1;
-		return;
-	}
-
-#ifdef CONFIG_VT
-	unblank_screen();
-#endif
-	oops_in_progress = 0;
-	/*
-	 * OK, the message is on the console.  Now we call printk() without
-	 * oops_in_progress set so that printk will give klogd a poke.  Hold onto
-	 * your hats...
-	 */
-	console_loglevel = 15;		/* NMI oopser may have shut the console up */
-	printk(" ");
-	console_loglevel = loglevel_save;
-}
-
 void
 die (const char *str, struct pt_regs *regs, long err)
 {
arch/x86_64/mm/fault.c renamed from arch/x86_64/mm/fault.c.orig +0 -21
@@ -34,27 +34,6 @@
 #include <asm/kdebug.h>
 #include <asm-generic/sections.h>
 
-void bust_spinlocks(int yes)
-{
-	int loglevel_save = console_loglevel;
-	if (yes) {
-		oops_in_progress = 1;
-	} else {
-#ifdef CONFIG_VT
-		unblank_screen();
-#endif
-		oops_in_progress = 0;
-		/*
-		 * OK, the message is on the console.  Now we call printk()
-		 * without oops_in_progress set so that printk will give klogd
-		 * a poke.  Hold onto your hats...
-		 */
-		console_loglevel = 15;		/* NMI oopser may have shut the console up */
-		printk(" ");
-		console_loglevel = loglevel_save;
-	}
-}
-
 /* Sometimes the CPU reports invalid exceptions on prefetch.
    Check that here and ignore.
    Opcode checker based on code by Richard Brunner */
arch/s390/mm/fault.c renamed from arch/s390/mm/fault.c.orig +0 -26
@@ -49,32 +49,6 @@ extern int sysctl_userprocess_debug;
 
 extern void die(const char *,struct pt_regs *,long);
 
-extern spinlock_t timerlist_lock;
-
-/*
- * Unlock any spinlocks which will prevent us from getting the
- * message out (timerlist_lock is acquired through the
- * console unblank code)
- */
-void bust_spinlocks(int yes)
-{
-	if (yes) {
-		oops_in_progress = 1;
-	} else {
-		int loglevel_save = console_loglevel;
-		oops_in_progress = 0;
-		console_unblank();
-		/*
-		 * OK, the message is on the console.  Now we call printk()
-		 * without oops_in_progress set so that printk will give klogd
-		 * a poke.  Hold onto your hats...
-		 */
-		console_loglevel = 15;
-		printk(" ");
-		console_loglevel = loglevel_save;
-	}
-}
-
 /*
  * Check which address space is addressed by the access
  * register in S390_lowcore.exc_access_id.
lib/bust_spinlocks.c renamed from lib/bust_spinlocks.c.orig +14 -13
@@ -17,23 +17,24 @@
 
 void bust_spinlocks(int yes)
 {
+	int loglevel_save = console_loglevel;
+
 	if (yes) {
 		oops_in_progress = 1;
-	} else {
-		int loglevel_save = console_loglevel;
+		return;
+	}
 #ifdef CONFIG_VT
-		unblank_screen();
+	unblank_screen();
 #endif
-		oops_in_progress = 0;
-		/*
-		 * OK, the message is on the console.  Now we call printk()
-		 * without oops_in_progress set so that printk() will give klogd
-		 * and the blanked console a poke.  Hold onto your hats...
-		 */
-		console_loglevel = 15;		/* NMI oopser may have shut the console up */
-		printk(" ");
-		console_loglevel = loglevel_save;
-	}
+	oops_in_progress = 0;
+	/*
+	 * OK, the message is on the console.  Now we call printk()
+	 * without oops_in_progress set so that printk() will give klogd
+	 * and the blanked console a poke.  Hold onto your hats...
+	 */
+	console_loglevel = 15;	/* NMI oopser may have shut the console up */
+	printk(" ");
+	console_loglevel = loglevel_save;
 }
 
 
#6 lore raw
Quoted 15 lines · Ingo Molnar
Quoted 4 lines · Ingo Molnar

another solution would be to break the lock only once during the kernel's lifetime. The system is messed up anyway if it needs multiple lock breaks to get an oops out to the console. We dont care about followup oopses - the first oops is that matters.

i.e. something like the attached patch, against BK-curr. (i've also attached a cleanup patch that gets rid of the many instances of bust_spinlocks() - we now have a generic one in lib/bust_spinlocks.c)

i consider any secondary lockup after the first oops has been printed a feature - sometimes the first oops gets washed away by the many followup oopses.

+unsigned long zap_spinlocks = 1;

[snip]

kernel/printk.c asmlinkage int printk(const char *fmt, . · line 499

Quoted 2 lines
kernel/printk.c renamed from kernel/printk.c.orig +1 -1
-	if (oops_in_progress) {
+	if (oops_in_progress && test_and_clear_bit(0, &zap_spinlocks)) {

It looks like this will cause problems if the user experiences an oops during boot, runs for a few hours, and then has another oops. Won't the second oops fail to break the lock, and end up just deadlocking?

Yeah, of course you shouldn't blithely ignore the first oops. But I bet it happens.

Perhaps zap_spinlocks could be reset after 10 minutes without an oops?

-andy

#7 lore raw
Quoted 3 lines · Andy Isaacson

It looks like this will cause problems if the user experiences an oops during boot, runs for a few hours, and then has another oops. Won't the second oops fail to break the lock, and end up just deadlocking?

i've attached a new patch that does what Andi suggested too - timestamping of the oopses. This way we will zap no sooner than 10 seconds after the first oops.

Ingo

kernel/printk.c renamed from kernel/printk.c.orig +22 -6
@@ -55,6 +55,9 @@ EXPORT_SYMBOL(console_printk);
 
 int oops_in_progress;
 
+/* zap spinlocks only once: */
+unsigned long oops_timestamp;
+
 /*
  * console_sem protects the console_drivers list, and also
  * provides serialisation for access to the entire console
@@ -472,6 +475,23 @@ static void emit_log_char(char c)
 }
 
 /*
+ * Zap console related locks when oopsing. Only zap at most once
+ * every 10 seconds, to leave time for slow consoles to print a
+ * full oops.
+ */
+static inline void zap_locks(void)
+{
+	if (!time_after(jiffies, oops_timestamp + 10*HZ))
+		return;
+	oops_timestamp = jiffies;
+
+	/* If a crash is occurring, make sure we can't deadlock */
+	spin_lock_init(&logbuf_lock);
+	/* And make sure that we print immediately */
+	init_MUTEX(&console_sem);
+}
+
+/*
  * This is printk.  It can be called from any context.  We want it to work.
  * 
  * We try to grab the console_sem.  If we succeed, it's easy - we log the output and
@@ -493,12 +513,8 @@ asmlinkage int printk(const char *fmt, .
 	static char printk_buf[1024];
 	static int log_level_unknown = 1;
 
-	if (oops_in_progress) {
-		/* If a crash is occurring, make sure we can't deadlock */
-		spin_lock_init(&logbuf_lock);
-		/* And make sure that we print immediately */
-		init_MUTEX(&console_sem);
-	}
+	if (unlikely(oops_in_progress))
+		zap_locks();
 
 	/* This stops the holder of console_sem just where we want him */
 	spin_lock_irqsave(&logbuf_lock, flags);
#8 lore raw
Quoted 4 lines · Ingo Molnar

i've attached a new patch that does what Andi suggested too - timestamping of the oopses. This way we will zap no sooner than 10 seconds after the first oops.

I think that will do the wrong thing between 23 and 47 days uptime because time_after() will return an incorrect answer.

How's this look?

- Bump the timeout to 30 seconds - 9600 baud is slow.

- Handle jiffy wraps: change the logic so that we only skip the lockbust if the current time is within 30 seconds of the previous lockbusting attempt.

---

 25-akpm/kernel/printk.c |   11 ++++++-----
 1 files changed, 6 insertions(+), 5 deletions(-)
kernel/printk.c renamed from kernel/printk.c~mangled-printk-oops-output-fix-tweaks +6 -5
diff -puN kernel/printk.c~mangled-printk-oops-output-fix-tweaks kernel/printk.c
@@ -55,9 +55,6 @@ EXPORT_SYMBOL(console_printk);
 
 int oops_in_progress;
 
-/* zap spinlocks only once: */
-unsigned long oops_timestamp;
-
 /*
  * console_sem protects the console_drivers list, and also
  * provides serialisation for access to the entire console
@@ -479,10 +476,14 @@ static void emit_log_char(char c)
  * every 10 seconds, to leave time for slow consoles to print a
  * full oops.
  */
-static inline void zap_locks(void)
+static void zap_locks(void)
 {
-	if (!time_after(jiffies, oops_timestamp + 10*HZ))
+	static unsigned long oops_timestamp;
+
+	if (time_after_eq(jiffies, oops_timestamp) &&
+			!time_after(jiffies, oops_timestamp + 30*HZ))
 		return;
+
 	oops_timestamp = jiffies;
 
 	/* If a crash is occurring, make sure we can't deadlock */

_

#9 lore raw
Quoted 11 lines · Andrew Morton
Quoted 4 lines · Ingo Molnar

i've attached a new patch that does what Andi suggested too - timestamping of the oopses. This way we will zap no sooner than 10 seconds after the first oops.

I think that will do the wrong thing between 23 and 47 days uptime because time_after() will return an incorrect answer.

How's this look?

kernel/printk.c static void emit_log_char(char c) · line 486

Quoted 6 lines
kernel/printk.c renamed from kernel/printk.c~mangled-printk-oops-output-fix-tweaks +5 -0
+	static unsigned long oops_timestamp;
+
+	if (time_after_eq(jiffies, oops_timestamp) &&
+			!time_after(jiffies, oops_timestamp + 30*HZ))
 		return;
+

looks good to me!

Ingo