Serialise oopses, BUGs, WARNs, dump_stack, soft lockups and hard lockups
Subsystems (5)
cover Serialise oopses, BUGs, WARNs, dump_stack, soft lockups and hard lockups
Every now and then I end up with an undebuggable issue because multiple CPUs hit something at the same time and everything is interleaved:
CR: 48000082 XER: 00000000
,RI
c0000003dc72fd10
,LE
d0000000065b84e8
Instruction dump:
MSR: 8000000100029033Very annoying.
Some architectures already have their own recursive locking for oopses and we have another version for serialising dump_stack.
Create a common version and use it everywhere (oopses, BUGs, WARNs, dump_stack, soft lockups and hard lockups). A few testcases were used to verify the series:
A trivial module to create concurrent WARNs, BUGs and oopses:
http://ozlabs.org/~anton/junkcode/warnstorm.tar.gz
And one to create concurrent soft and hard lockups:
http://ozlabs.org/~anton/junkcode/badguy.tar.gz
Anton Blanchard (7):
Add die_spin_lock_{irqsave,irqrestore}
powerpc: Use die_spin_lock_{irqsave,irqrestore}
arm: Use die_spin_lock_{irqsave,irqrestore}
x86: Use die_spin_lock_{irqsave,irqrestore}
watchdog: Serialise soft lockup errors with
die_spin_lock_{irqsave,irqrestore}
dump_stack: Serialise dump_stack with
die_spin_lock_{irqsave,irqrestore}
powerpc: Serialise BUG and WARNs with
die_spin_lock_{irqsave,irqrestore}arch/arm/kernel/traps.c | 26 ++--------------- arch/powerpc/kernel/traps.c | 68 ++++++++++++++++++++++++++------------------- arch/x86/kernel/dumpstack.c | 26 ++--------------- include/linux/die_lock.h | 23 +++++++++++++++ kernel/watchdog.c | 4 +++ lib/Makefile | 1 + lib/die_lock.c | 43 ++++++++++++++++++++++++++++ lib/dump_stack.c | 40 +++----------------------- 8 files changed, 120 insertions(+), 111 deletions(-) create mode 100644 include/linux/die_lock.h create mode 100644 lib/die_lock.c
Signature · 1 line
2.1.0
PATCH 1/7 Add die_spin_lock_{irqsave,irqrestore}
Many architectures have their own oops locking code that allows the lock to be taken recursively. Create a common version.
Avoid creating generic locking functions, so they can't be abused in other parts of the kernel.
--- include/linux/die_lock.h | 23 +++++++++++++++++++++++ lib/Makefile | 1 + lib/die_lock.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 67 insertions(+) create mode 100644 include/linux/die_lock.h create mode 100644 lib/die_lock.c
include/linux/die_lock.h new file +23 -0
@@ -0,0 +1,23 @@ +#ifndef __LINUX_DIE_LOCK_H +#define __LINUX_DIE_LOCK_H + +#include <linux/typecheck.h> + +/** +* die_spin_lock_irqsave - lock die spinlock +* @flags: interrupt state is saved here +* +* The die spinlock is used to serialise output during oopses, BUGs and +* WARNs. It can be taken recursively so that nested oopses will not +* lock up. +*/ +unsigned long __die_spin_lock_irqsave (void ); +#define die_spin_lock_irqsave (flags) \ + do { \ + typecheck(unsigned long, flags); \ + flags = __die_spin_lock_irqsave(); \ + } while (0) + +void die_spin_unlock_irqrestore (unsigned long flags); + +#endif /* __LINUX_DIE_LOCK_H */
lib/Makefile +1 -0
@@ -28,6 +28,7 @@ obj-y += bcd.o div64.o sort.o parser.o halfmd4.o debug_locks.o random32.o \ bsearch.o find_last_bit.o find_next_bit.o llist.o memweight.o kfifo.o \ percpu-refcount.o percpu_ida.o rhashtable.o reciprocal_div.o obj-y += string_helpers.o +obj-y += die_lock.o obj-$(CONFIG_TEST_STRING_HELPERS) += test-string_helpers.o obj-y += kstrtox.o obj-$(CONFIG_TEST_KSTRTOX) += test-kstrtox.o
lib/die_lock.c new file +43 -0
43 lines not on this page · raw message
Signature · 1 line
2.1.0
Re: [PATCH 1/7] Add die_spin_lock_{irqsave,irqrestore}
Quoted 6 lines · Anton Blanchard
lib/die_lock.c new file +6
-0+static arch_spinlock_t die_lock = __ARCH_SPIN_LOCK_UNLOCKED; +static int die_owner =-1 ; +static unsigned int die_nest_count; + +unsigned long __die_spin_lock_irqsave (void ) +{
lib/die_lock.c line 19
Quoted 10 lines
lib/die_lock.c new file +10
-0+unsigned long flags; +int cpu; + +/* racy, but better than risking deadlock. */ +raw_local_irq_save (flags); + + cpu =smp_processor_id (); +if (!arch_spin_trylock (&die_lock)) { +if (cpu != die_owner) +arch_spin_lock (&die_lock);
So why not trylock and time out here after a few seconds, instead of indefinitely supressing some potentially vital output due to some other CPU crashing/locking with the lock held?
lib/die_lock.c line 24
Quoted 5 lines
lib/die_lock.c new file +5
-0+ } + die_nest_count++; + die_owner = cpu; + +return flags;
I suspect this would work in most cases.
If we fix the deadlock potential, and get a true global ordering of various oopses/warnings as they triggered (or at least timestamping them), then I'm sold on this I guess, it will likely improve things.
Thanks,
Ingo
Re: [PATCH 1/7] Add die_spin_lock_{irqsave,irqrestore}
Quoted 24 lines · Ingo Molnar
Quoted 6 lines · Anton Blanchard
lib/die_lock.c new file +6
-0+static arch_spinlock_t die_lock = __ARCH_SPIN_LOCK_UNLOCKED; +static int die_owner =-1 ; +static unsigned int die_nest_count; + +unsigned long __die_spin_lock_irqsave (void ) +{lib/die_lock.c line 19
Quoted 10 lines
lib/die_lock.c new file +10
-0+unsigned long flags; +int cpu; + +/* racy, but better than risking deadlock. */ +raw_local_irq_save (flags); + + cpu =smp_processor_id (); +if (!arch_spin_trylock (&die_lock)) { +if (cpu != die_owner) +arch_spin_lock (&die_lock);So why not trylock and time out here after a few seconds, instead of indefinitely supressing some potentially vital output due to some other CPU crashing/locking with the lock held?
[...]
Quoted 3 lines
If we fix the deadlock potential, and get a true global ordering of various oopses/warnings as they triggered (or at least timestamping them), [...]
If we had a global 'trouble counter' we could use that to refine the spin-looping timeout: instead of using a pure timeout of a few seconds, we could say 'a timeout of a few seconds while the counter does not increase'.
I.e. only override the locking/ordering if the owner CPU does not seem to be able to make progress with printing the oops/warning.
Thanks,
Ingo
RE: [PATCH 1/7] Add die_spin_lock_{irqsave,irqrestore}
In reply to Ingo MolnarFrom: Ingo Molnar
...
Quoted 4 lines
So why not trylock and time out here after a few seconds, instead of indefinitely supressing some potentially vital output due to some other CPU crashing/locking with the lock held?
I've used that for status requests that usually lock a table to get a consistent view.
If trylock times out assume that the data is actually stable and access it anyway. Remember the pid doing the access and next time it tries to acquire the same lock do a trylock with no timeout.
That way if (when) there is a locking fubar (or a driver oops with a lock held) at least some of the relevant status commands will work.
David
PATCH 2/7 powerpc: Use die_spin_lock_{irqsave,irqrestore}
In reply to Anton BlanchardReplace the powerpc specific oops locking with the common one.
--- arch/powerpc/kernel/traps.c | 24 +++--------------------- 1 file changed, 3 insertions(+), 21 deletions(-)
arch/powerpc/kernel/traps.c +3 -21
@@ -36,6 +36,7 @@#include <linux/debugfs.h> #include <linux/ratelimit.h> #include <linux/context_tracking.h> +#include <linux/die_lock.h> #include <asm/emulated_ops.h> #include <asm/pgtable.h>
@@ -109,14 +110,10 @@ static void pmac_backlight_unblank(void)static inline void pmac_backlight_unblank (void ) { }#endif -static arch_spinlock_t die_lock = __ARCH_SPIN_LOCK_UNLOCKED; -static int die_owner =-1 ; -static unsigned int die_nest_count;static int die_counter;static unsigned __kprobes long oops_begin (struct pt_regs *regs) { -int cpu;unsigned long flags;if (debugger (regs))
@@ -124,17 +121,7 @@ static unsigned __kprobes long oops_begin(struct pt_regs *regs)oops_enter (); -/* racy, but better than risking deadlock. */ -raw_local_irq_save (flags); - cpu =smp_processor_id (); -if (!arch_spin_trylock (&die_lock)) { -if (cpu == die_owner) -/* nested oops. should stop eventually */ ; -else -arch_spin_lock (&die_lock); - } - die_nest_count++; - die_owner = cpu; +die_spin_lock_irqsave (flags);console_verbose ();bust_spinlocks (1 );if (machine_is (powermac))
@@ -146,15 +133,10 @@ static void __kprobes oops_end(unsigned long flags, struct pt_regs *regs, int signr) {bust_spinlocks (0 ); - die_owner =-1 ;add_taint (TAINT_DIE ,LOCKDEP_NOW_UNRELIABLE ); - die_nest_count--;oops_exit ();printk ("\n" ); -if (!die_nest_count) -/* Nest count reaches zero, release the lock. */ -arch_spin_unlock (&die_lock); -raw_local_irq_restore (flags); +die_spin_unlock_irqrestore (flags);crash_fadump (regs,"die oops" );
Signature · 1 line
2.1.0
PATCH 3/7 arm: Use die_spin_lock_{irqsave,irqrestore}
In reply to Anton BlanchardReplace the ARM specific oops locking with the common one.
--- arch/arm/kernel/traps.c | 26 +++----------------------- 1 file changed, 3 insertions(+), 23 deletions(-)
arch/arm/kernel/traps.c +3 -23
@@ -26,6 +26,7 @@#include <linux/init.h> #include <linux/sched.h> #include <linux/irq.h> +#include <linux/die_lock.h> #include <linux/atomic.h> #include <asm/cacheflush.h>
@@ -263,28 +264,12 @@ static int __die(const char *str, int err, struct pt_regs *regs)return 0 ; } -static arch_spinlock_t die_lock = __ARCH_SPIN_LOCK_UNLOCKED; -static int die_owner =-1 ; -static unsigned int die_nest_count; -static unsigned long oops_begin (void ) { -int cpu;unsigned long flags;oops_enter (); - -/* racy, but better than risking deadlock. */ -raw_local_irq_save (flags); - cpu =smp_processor_id (); -if (!arch_spin_trylock (&die_lock)) { -if (cpu == die_owner) -/* nested oops. should stop eventually */ ; -else -arch_spin_lock (&die_lock); - } - die_nest_count++; - die_owner = cpu; +die_spin_lock_irqsave (flags);console_verbose ();bust_spinlocks (1 );return flags;
@@ -296,13 +281,8 @@ static void oops_end(unsigned long flags, struct pt_regs *regs, int signr)crash_kexec (regs);bust_spinlocks (0 ); - die_owner =-1 ;add_taint (TAINT_DIE ,LOCKDEP_NOW_UNRELIABLE ); - die_nest_count--; -if (!die_nest_count) -/* Nest count reaches zero, release the lock. */ -arch_spin_unlock (&die_lock); -raw_local_irq_restore (flags); +die_spin_unlock_irqrestore (flags);oops_exit ();if (in_interrupt ())
Signature · 1 line
2.1.0
PATCH 4/7 x86: Use die_spin_lock_{irqsave,irqrestore}
In reply to Anton BlanchardReplace the x86 specific oops locking with the common one.
--- arch/x86/kernel/dumpstack.c | 26 +++----------------------- 1 file changed, 3 insertions(+), 23 deletions(-)
arch/x86/kernel/dumpstack.c +3 -23
@@ -15,6 +15,7 @@#include <linux/bug.h> #include <linux/nmi.h> #include <linux/sysfs.h> +#include <linux/die_lock.h> #include <asm/stacktrace.h>
@@ -196,28 +197,12 @@ void show_stack(struct task_struct *task, unsigned long *sp)show_stack_log_lvl (task,NULL , sp, bp,"" ); } -static arch_spinlock_t die_lock = __ARCH_SPIN_LOCK_UNLOCKED; -static int die_owner =-1 ; -static unsigned int die_nest_count; -unsigned long oops_begin (void ) { -int cpu;unsigned long flags;oops_enter (); - -/* racy, but better than risking deadlock. */ -raw_local_irq_save (flags); - cpu =smp_processor_id (); -if (!arch_spin_trylock (&die_lock)) { -if (cpu == die_owner) -/* nested oops. should stop eventually */ ; -else -arch_spin_lock (&die_lock); - } - die_nest_count++; - die_owner = cpu; +die_spin_lock_irqsave (flags);console_verbose ();bust_spinlocks (1 );return flags;
@@ -231,13 +216,8 @@ void oops_end(unsigned long flags, struct pt_regs *regs, int signr)crash_kexec (regs);bust_spinlocks (0 ); - die_owner =-1 ;add_taint (TAINT_DIE ,LOCKDEP_NOW_UNRELIABLE ); - die_nest_count--; -if (!die_nest_count) -/* Nest count reaches zero, release the lock. */ -arch_spin_unlock (&die_lock); -raw_local_irq_restore (flags); +die_spin_unlock_irqrestore (flags);oops_exit ();if (!signr)
Signature · 1 line
2.1.0
PATCH 5/7 watchdog: Serialise soft lockup errors with die_spin_lock_{irqsave,irqrestore}
In reply to Anton BlanchardA simple kernel module was used to create concurrent soft and hard lockups:
http://ozlabs.org/~anton/junkcode/badguy.tar.gz
--- kernel/watchdog.c | 4 ++++ 1 file changed, 4 insertions(+)
kernel/watchdog.c +4 -0
@@ -19,6 +19,7 @@#include <linux/sysctl.h> #include <linux/smpboot.h> #include <linux/sched/rt.h> +#include <linux/die_lock.h> #include <asm/irq_regs.h> #include <linux/kvm_para.h>
@@ -313,6 +314,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)struct pt_regs *regs =get_irq_regs ();int duration;int softlockup_all_cpu_backtrace = sysctl_softlockup_all_cpu_backtrace; +unsigned long flags;/* kick the hardlockup detector */ watchdog_interrupt_count ();
@@ -384,6 +386,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) } } +die_spin_lock_irqsave (flags);pr_emerg ("BUG: soft lockup - CPU#%d stuck for %us! [%s:%d]\n" ,smp_processor_id (), duration, current->comm,task_pid_nr (current));
@@ -394,6 +397,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)show_regs (regs); elsedump_stack (); +die_spin_unlock_irqrestore (flags);if (softlockup_all_cpu_backtrace) {/* Avoid generating two back traces for current
Signature · 1 line
2.1.0
PATCH 6/7 dump_stack: Serialise dump_stack with die_spin_lock_{irqsave,irqrestore}
In reply to Anton BlanchardRemove another version of a recursive lock in dump_stack.
--- lib/dump_stack.c | 40 ++++------------------------------------ 1 file changed, 4 insertions(+), 36 deletions(-)
lib/dump_stack.c +4 -36
@@ -7,7 +7,7 @@#include <linux/export.h> #include <linux/sched.h> #include <linux/smp.h> -#include <linux/atomic.h> +#include <linux/die_lock.h> static void __dump_stack (void ) {
@@ -20,44 +20,12 @@ static void __dump_stack(void)* * Architectures can override this implementation by implementing its own. */ -#ifdef CONFIG_SMP -static atomic_t dump_lock =ATOMIC_INIT (-1 ); - asmlinkage __visible void dump_stack(void) { -int was_locked; -int old; -int cpu; - -/* -* Permit this cpu to perform nested stack dumps while serialising -* against other CPUs -*/ -preempt_disable (); - -retry: - cpu =smp_processor_id (); - old =atomic_cmpxchg (&dump_lock,-1 , cpu); -if (old ==-1 ) { - was_locked =0 ; - }else if (old == cpu) { - was_locked =1 ; - }else { -cpu_relax (); - goto retry; - } - -__dump_stack (); +unsigned long flags; -if (!was_locked) -atomic_set (&dump_lock,-1 ); - -preempt_enable (); -} -#else -asmlinkage __visible void dump_stack(void) -{ +die_spin_lock_irqsave (flags);__dump_stack (); +die_spin_unlock_irqrestore (flags); } -#endif EXPORT_SYMBOL (dump_stack);
Signature · 1 line
2.1.0
PATCH 7/7 powerpc: Serialise BUG and WARNs with die_spin_lock_{irqsave,irqrestore}
In reply to Anton BlanchardA simple kernel module was used to create concurrent WARNs and BUGs:
http://ozlabs.org/~anton/junkcode/warnstorm.tar.gz
--- arch/powerpc/kernel/traps.c | 44 ++++++++++++++++++++++++++++++++++++-------- 1 file changed, 36 insertions(+), 8 deletions(-)
arch/powerpc/kernel/traps.c +36 -8
@@ -1114,6 +1114,39 @@ static int emulate_math(struct pt_regs *regs)static inline int emulate_math (struct pt_regs *regs) {return -1 ; }#endif +static bool __report_bug (unsigned long bugaddr,struct pt_regs *regs) +{ +const struct bug_entry *bug; +unsigned long flags; +int err; + +if (!is_valid_bugaddr (bugaddr)) +return false; + + bug =find_bug (bugaddr); +if (!bug) +return false; + +if (is_warning_bug (bug)) { +die_spin_lock_irqsave (flags); +report_bug (bugaddr, regs); +die_spin_unlock_irqrestore (flags); + + regs->nip +=4 ; + +return true; + } + + flags =oops_begin (regs); +report_bug (bugaddr, regs); + err =SIGTRAP ; +if (__die ("Exception in kernel mode" , regs, err)) + err =0 ; +oops_end (flags, regs, err); + +return true; +} +void __kprobes program_check_exception (struct pt_regs *regs) {enum ctx_state prev_state =exception_enter ();
@@ -1138,11 +1171,9 @@ void __kprobes program_check_exception(struct pt_regs *regs) == NOTIFY_STOP) goto bail; -if (!(regs->msr &MSR_PR ) &&/* not user-mode */ -report_bug (regs->nip, regs) ==BUG_TRAP_TYPE_WARN ) { - regs->nip +=4 ; +if (!user_mode (regs) &&__report_bug (regs->nip, regs)) goto bail; - } +_exception (SIGTRAP , regs,TRAP_BRKPT , regs->nip); goto bail; }
@@ -1157,11 +1188,8 @@ void __kprobes program_check_exception(struct pt_regs *regs)* - A tend is illegally attempted. * - writing a TM SPR when transactional. */ -if (!user_mode (regs) && -report_bug (regs->nip, regs) ==BUG_TRAP_TYPE_WARN ) { - regs->nip +=4 ; +if (!user_mode (regs) &&__report_bug (regs->nip, regs)) goto bail; - }/* If usermode caused this, it's done something illegal and * gets a SIGILL slap on the wrist. We call it an illegal * operand to distinguish from the instruction just being bad
Signature · 1 line
2.1.0
Quoted 12 lines · Anton Blanchard
Every now and then I end up with an undebuggable issue because multiple CPUs hit something at the same time and everything is interleaved:
CR: 48000082 XER: 00000000 ,RI c0000003dc72fd10 ,LE d0000000065b84e8 Instruction dump: MSR: 8000000100029033
Quoted 8 lines
Very annoying.
Some architectures already have their own recursive locking for oopses and we have another version for serialising
dump_stack.Create a common version and use it everywhere (oopses, BUGs, WARNs,
dump_stack, soft lockups and hard lockups).
Dunno. I've had cases where the simultaneity of the oopses (i.e. their garbled nature) gave me the clue about the type of race to expect.
To still get that information: instead of taking a serializing spinlock (or in addition to it), it would be nice to at least preserve the true time order of the incidents, at minimum by generating a global count for oopses/warnings (a bit like the oops count # currently), and to gather it first - before taking any spinlocks.
Thanks,
Ingo
Quoted 11 lines
Quoted 6 lines
Some architectures already have their own recursive locking for oopses and we have another version for serialising
dump_stack.Create a common version and use it everywhere (oopses, BUGs, WARNs,
dump_stack, soft lockups and hard lockups).Dunno. I've had cases where the simultaneity of the oopses (i.e. their garbled nature) gave me the clue about the type of race to expect.
one of the question is if you want to serialize, or if you just want to label. If you take a cookie (could just be a monotonic increasing number) at the start of the oops
and then prefix/postfix the stack printing with that number, you don't serialize (risk of locking up),
but you can pretty trivially see which line came from where..
if you do the monotonic increasing number approach, you even get an ordering out of it.
it does mean changing the dump_stack() and co function fingerprint to take an extra argument,
but that is not TOO insane.
Quoted 9 lines · Arjan van de Ven
one of the question is if you want to serialize, or if you just want to label. If you take a cookie (could just be a monotonic increasing number) at the start of the oops and then prefix/postfix the stack printing with that number, you don't serialize (risk of locking up), but you can pretty trivially see which line came from where.. if you do the monotonic increasing number approach, you even get an ordering out of it. it does mean changing the
dump_stack()and co function fingerprint to take an extra argument, but that is not TOO insane.
I like that idea, but it relies on ensuring that each line is printed by one printk() statement - which in itself is a good idea.
I'd actually like a version of print_hex_dump() which we could use for stack and code dumping - the existing print_hex_dump() assumes that it's fine to dereference the pointer, whereas for stack and code dumping, we can't always make that assumption. That's a separate issue though.
Signature · 2 lines
FTTC broadband for 0.8mile line: currently at 10.5Mbps down 400kbps up according to speedtest.net.