Skip to contentSkip to conversation

crash_stop: Summary

PATCH In review Keith Owens 2 participants 15 messages Nov 9, 2006 – Nov 13, 2006

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover crash_stop: Summary

All the kernel debug style tools (kdb, kgdb, nlkd, netdump, lkcd, crash, kdump etc.) have a common requirement, they need to do a crash stop of the systems. This means stopping all the cpus, even if some of the cpus are spinning disabled. In addition, each cpu has to save enough state to start the diagnosis of the problem.

* Each debug style tool has written its own code for interrupting the other cpus and for saving cpu state.

* Some tools try a normal IPI first then send a non-maskable interrupt after a delay.

* Some tools always send a NMI first, which can result in incomplete or wrong machine state if NMI arrives at the wrong time.

* Most of the tools do not know how to cope with the IA64 architecture defined rendezvous algorithm, which interferes with an OS driven rendezvous.

* Needless to say, every single patch set conflicts with all the others, which makes it very difficult to install more than one of the tools at a time.

The solution is to define a common crash_stop API that can be used by _all_ of the debug style tools, without reinventing the wheel each time. The following crash_stop patches implement this API for i386, x86_64 and ia64. It correctly handles the complicated ia64 algorithm for MCA and INIT, unlike almost every current debug style tool.

Adding other architectures is a fairly simple matter, define the IPI and NMI routines (the crash_stop_$(ARCH)_handlers patch), intercept the events that indicate that the system is dying (the crash_stop_$(ARCH) patch), update the Kconfig entry for CRASH_STOP to add the new $(ARCH).

Most of the design documentation is in the crash_stop_common patch. Please read that before replying.

crash_stop_header The architecture independent header.

crash_stop_common The architecture independent code.

crash_stop_i386_handlers i386 specific code to send and respond
to the crash_stop IPI and NMI.

crash_stop_i386 i386 specific code to intercept events
that indicate that the system is dying.

crash_stop_x86_64_nmiwatchdog i386 creates an event for NMI watchdog,
it is missing from x86_64. Add DIE_NMIWATCHDOG to x86_64.

crash_stop_x86_64_handlers x86_64 specific code to send and
respond to the crash_stop IPI and NMI.

crash_stop_x86_64 x86_64 specific code to intercept
events that indicate that the system is dying.

crash_stop_ia64_handlers ia64 specific code to send and respond
to the crash_stop IPI and NMI.

crash_stop_ia64 ia64 specific code to intercept events
that indicate that the system is dying.

crash_stop_common_Kconfig Add crash_stop to the config system.
Only for i386, x86_64 and ia64 at the moment, extend as new architectures are added.

crash_stop_demo A demonstration of using crash_stop in
a debug style tool. Not for inclusion in the kernel.

crash_stop_test Test the crash_stop code. Not for
inclusion in the kernel.

#1 lore raw

PATCH 1/12 crash_stop: common header

crash_stop() common header.

---
 include/linux/crash_stop.h |   71 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 71 insertions(+)
include/linux/crash_stop.h new file +71 -0
Index: linux/include/linux/crash_stop.h
===================================================================
@@ -0,0 +1,71 @@
+#ifndef _LINUX_CRASH_STOP_H
+
+#include <linux/cpumask.h>
+#include <linux/errno.h>
+#include <linux/ptrace.h>
+
+typedef asmlinkage int (*printk_t)(const char * fmt, ...)
+	__attribute__ ((format (printf, 1, 2)));
+#define _LINUX_CRASH_STOP_H
+
+#ifdef	CONFIG_CRASH_STOP_SUPPORTED
+
+#include <asm/crash_stop.h>
+
+/* These six entries are the only ones used by code outside crash_stop itself.
+ * Anything starting with 'crash_stop' is part of the external ABI, anything
+ * starting with'cs_' is only to be used by internal crash_stop code.
+ */
+extern int crash_stop(void (*callback)(int monarch, void *data),
+		      void *data, printk_t print,
+		      struct pt_regs *regs, const char *text);
+extern void crash_stop_recovered(void);
+#ifdef	CONFIG_SMP
+extern void crash_stop_slave(void);
+extern int crash_stop_sent_nmi(void);
+#else	/* !CONFIG_SMP */
+#define crash_stop_slave() do {} while(0)
+#define crash_stop_sent_nmi() 0
+#endif	/* CONFIG_SMP */
+extern int crash_stop_slaves(void);
+struct crash_stop_running_process {
+	struct task_struct *p;
+	struct pt_regs *regs;
+	struct crash_stop_running_process *prev;
+	struct crash_stop_running_process_arch arch;
+};
+
+extern void cs_common_ipi(void);
+extern void cs_arch_send_ipi(int);
+extern void cs_arch_send_nmi(int);
+
+extern void cs_arch_cpu(int, struct crash_stop_running_process *);
+extern void cs_common_cpu(int);
+
+extern void cs_notify_chain_start(struct pt_regs *);
+extern void cs_notify_chain_end(void);
+
+struct cs_global {
+	void (*callback)(int monarch, void *data);
+	void *data;
+	printk_t print;
+};
+extern struct cs_global cs_global;
+
+#else	/* !CONFIG_CRASH_STOP_SUPPORTED */
+
+static inline
+int crash_stop(void (*callback)(int monarch, void *data),
+		     void *data, printk_t print,
+		     struct pt_regs *regs, const char *text)
+{
+	return -ENOSYS;
+}
+#define crash_stop_recovered() do {} while(0)
+#define crash_stop_slave() do {} while(0)
+#define crash_stop_sent_nmi() 0
+#define crash_stop_slaves() 0
+
+#endif	/* CONFIG_CRASH_STOP_SUPPORTED */
+
+#endif	/* _LINUX_CRASH_STOP_H */
#2 lore raw

PATCH 2/12 crash_stop: common code

In reply to Keith Owens

The common crash_stop code, including the design level documentation.

---
 kernel/Makefile     |    1 
 kernel/crash_stop.c |  843 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 kernel/sys.c        |    5 
 3 files changed, 848 insertions(+), 1 deletion(-)
kernel/Makefile +1 -0
Index: linux/kernel/Makefile
===================================================================
@@ -51,6 +51,7 @@ obj-$(CONFIG_RELAY) += relay.o
 obj-$(CONFIG_UTS_NS) += utsname.o
 obj-$(CONFIG_TASK_DELAY_ACCT) += delayacct.o
 obj-$(CONFIG_TASKSTATS) += taskstats.o tsacct.o
+obj-$(CONFIG_CRASH_STOP_SUPPORTED) += crash_stop.o
 
 ifneq ($(CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER),y)
 # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is
kernel/crash_stop.c new file +843 -0

843 lines not on this page · raw message

kernel/sys.c +4 -1
Index: linux/kernel/sys.c
===================================================================
@@ -29,6 +29,7 @@
 #include <linux/signal.h>
 #include <linux/cn_proc.h>
 #include <linux/getcpu.h>
+#include <linux/crash_stop.h>
 
 #include <linux/compat.h>
 #include <linux/syscalls.h>
@@ -144,8 +145,10 @@ static int __kprobes notifier_call_chain
 	while (nb) {
 		next_nb = rcu_dereference(nb->next);
 		ret = nb->notifier_call(nb, val, v);
-		if ((ret & NOTIFY_STOP_MASK) == NOTIFY_STOP_MASK)
+		if ((ret & NOTIFY_STOP_MASK) == NOTIFY_STOP_MASK) {
+			cs_notify_chain_end();
 			break;
+		}
 		nb = next_nb;
 	}
 	return ret;
#3 lore raw

PATCH 3/12 crash_stop: i386 interrupt handlers

In reply to Keith Owens

Define the i386 crash_stop() interrupt handler and associated routines.

CRASH_STOP_VECTOR conflicts with the lkcd vector. This is deliberate, one aim of the crash_stop() API is to remove all the interrupt code from the various kernel debug patches and use the common crash_stop code instead.

Note 1: I have patched visw but I cannot test it, no hardware.

Note 2: This patch does not cover i386 voyager. I do not understand
the voyager interrupt mechanism well enough to define its interrupt handlers. Somebody with voyager equipment needs to define the cs_arch_send_ipi() and cs_arch_send_nmi() routines, plus the gate for the voyager CRASH_STOP_VECTOR.

---
 arch/i386/kernel/smp.c                      |   23 +++++++++++++++++++++++
 arch/i386/kernel/smpboot.c                  |    4 ++++
 include/asm-i386/hw_irq.h                   |    1 +
 include/asm-i386/mach-default/entry_arch.h  |    3 +++
 include/asm-i386/mach-default/irq_vectors.h |    1 +
 include/asm-i386/mach-visws/entry_arch.h    |    3 +++
 6 files changed, 35 insertions(+)
arch/i386/kernel/smp.c +23 -0
Index: linux/arch/i386/kernel/smp.c
===================================================================
@@ -20,6 +20,7 @@
 #include <linux/interrupt.h>
 #include <linux/cpu.h>
 #include <linux/module.h>
+#include <linux/crash_stop.h>
 
 #include <asm/mtrr.h>
 #include <asm/tlbflush.h>
@@ -733,3 +734,25 @@ int safe_smp_processor_id(void)
 
 	return cpuid >= 0 ? cpuid : 0;
 }
+
+#ifdef	CONFIG_CRASH_STOP_SUPPORTED
+void
+cs_arch_send_ipi(int cpu)
+{
+	send_IPI_mask(cpumask_of_cpu(cpu), CRASH_STOP_VECTOR);
+}
+
+void
+cs_arch_send_nmi(int cpu)
+{
+	send_IPI_mask(cpumask_of_cpu(cpu), NMI_VECTOR);
+}
+
+fastcall void smp_crash_stop_interrupt(struct pt_regs *regs)
+{
+	struct pt_regs *old_regs = set_irq_regs(regs);
+	ack_APIC_irq();
+	cs_common_ipi();
+	set_irq_regs(old_regs);
+}
+#endif	/* CONFIG_CRASH_STOP_SUPPORTED */
arch/i386/kernel/smpboot.c +4 -0
Index: linux/arch/i386/kernel/smpboot.c
===================================================================
@@ -1497,6 +1497,10 @@ void __init smp_intr_init(void)
 
 	/* IPI for generic function call */
 	set_intr_gate(CALL_FUNCTION_VECTOR, call_function_interrupt);
+
+#ifdef	CONFIG_CRASH_STOP_SUPPORTED
+	set_intr_gate(CRASH_STOP_VECTOR, crash_stop_interrupt);
+#endif	/* CONFIG_CRASH_STOP_SUPPORTED */
 }
 
 /*
include/asm-i386/hw_irq.h +1 -0
Index: linux/include/asm-i386/hw_irq.h
===================================================================
@@ -32,6 +32,7 @@ extern void (*interrupt[NR_IRQS])(void);
 fastcall void reschedule_interrupt(void);
 fastcall void invalidate_interrupt(void);
 fastcall void call_function_interrupt(void);
+fastcall void crash_stop_interrupt(void);
 #endif
 
 #ifdef CONFIG_X86_LOCAL_APIC
include/asm-i386/mach-default/entry_arch.h +3 -0
Index: linux/include/asm-i386/mach-default/entry_arch.h
===================================================================
@@ -13,6 +13,9 @@
 BUILD_INTERRUPT(reschedule_interrupt,RESCHEDULE_VECTOR)
 BUILD_INTERRUPT(invalidate_interrupt,INVALIDATE_TLB_VECTOR)
 BUILD_INTERRUPT(call_function_interrupt,CALL_FUNCTION_VECTOR)
+#ifdef	CONFIG_CRASH_STOP_SUPPORTED
+BUILD_INTERRUPT(crash_stop_interrupt,CRASH_STOP_VECTOR)
+#endif	/* CONFIG_CRASH_STOP_SUPPORTED */
 #endif
 
 /*
include/asm-i386/mach-default/irq_vectors.h +1 -0
Index: linux/include/asm-i386/mach-default/irq_vectors.h
===================================================================
@@ -48,6 +48,7 @@
 #define INVALIDATE_TLB_VECTOR	0xfd
 #define RESCHEDULE_VECTOR	0xfc
 #define CALL_FUNCTION_VECTOR	0xfb
+#define CRASH_STOP_VECTOR	0xfa
 
 #define THERMAL_APIC_VECTOR	0xf0
 /*
include/asm-i386/mach-visws/entry_arch.h +3 -0
Index: linux/include/asm-i386/mach-visws/entry_arch.h
===================================================================
@@ -7,6 +7,9 @@
 BUILD_INTERRUPT(reschedule_interrupt,RESCHEDULE_VECTOR)
 BUILD_INTERRUPT(invalidate_interrupt,INVALIDATE_TLB_VECTOR)
 BUILD_INTERRUPT(call_function_interrupt,CALL_FUNCTION_VECTOR)
+#ifdef	CONFIG_CRASH_STOP_SUPPORTED
+BUILD_INTERRUPT(crash_stop_interrupt,CRASH_STOP_VECTOR)
+#endif	/* CONFIG_CRASH_STOP_SUPPORTED */
 #endif
 
 /*
#4 lore raw

PATCH 4/12 crash_stop: i386 specific code

In reply to Keith Owens

Add the i386 specific crash_stop code. This contains routines that are called from the common crash_stop code and from the i386 notify_die chain.

Note: Not tested on visw nor voyager, no hardware.

---
 arch/i386/kernel/Makefile     |    1 
 arch/i386/kernel/crash_stop.c |  127 ++++++++++++++++++++++++++++++++++++++++++
 include/asm-i386/crash_stop.h |   14 ++++
 3 files changed, 142 insertions(+)
arch/i386/kernel/Makefile +1 -0
Index: linux/arch/i386/kernel/Makefile
===================================================================
@@ -39,6 +39,7 @@ obj-$(CONFIG_VM86)		+= vm86.o
 obj-$(CONFIG_EARLY_PRINTK)	+= early_printk.o
 obj-$(CONFIG_HPET_TIMER) 	+= hpet.o
 obj-$(CONFIG_K8_NB)		+= k8.o
+obj-$(CONFIG_CRASH_STOP_SUPPORTED)	+= crash_stop.o
 
 EXTRA_AFLAGS   := -traditional
 
arch/i386/kernel/crash_stop.c new file +127 -0

127 lines not on this page · raw message

include/asm-i386/crash_stop.h new file +14 -0
Index: linux/include/asm-i386/crash_stop.h
===================================================================
@@ -0,0 +1,14 @@
+#ifndef _ASM_CRASH_STOP_H
+#define _ASM_CRASH_STOP_H
+
+/* CONFIG_4KSTACKS means that the registers (including eip) at the time of the
+ * interrupt can be on one stack while the crash_stop code is running on
+ * another stack.  We have to save the current esp and eip.
+ */
+struct crash_stop_running_process_arch
+{
+	unsigned long esp;
+	unsigned long eip;
+};
+
+#endif	/* _ASM_CRASH_STOP_H */
#5 lore raw

PATCH 5/12 crash_stop: add DIE_NMIWATCHDOG to x86_64

In reply to Keith Owens

x86_64 defines DIE_NMIWATCHDOG but does not use it. Add an x86_64 call to notify_die when the NMI watchdog is about to go off, as i386 does.

---
 arch/x86_64/kernel/traps.c |    6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
arch/x86_64/kernel/traps.c +5 -1
Index: linux/arch/x86_64/kernel/traps.c
===================================================================
@@ -613,7 +613,11 @@ void die(const char * str, struct pt_reg
 
 void __kprobes die_nmi(char *str, struct pt_regs *regs, int do_panic)
 {
-	unsigned long flags = oops_begin();
+	unsigned long flags;
+	if (notify_die(DIE_NMIWATCHDOG, str, regs, 0, 2, SIGINT) ==
+	    NOTIFY_STOP)
+		return;
+	flags = oops_begin();
 
 	/*
 	 * We are in trouble anyway, lets at least try
#6 lore raw

PATCH 6/12 crash_stop: x86_64 interrupt handlers

In reply to Keith Owens

Define the x86_64 crash_stop() interrupt handler and associated routines.

CRASH_STOP_VECTOR conflicts with the KDB vector. This is deliberate, one aim of the crash_stop() API is to remove all the interrupt code from the various kernel debug patches and use the common crash_stop code instead.

---
 arch/x86_64/kernel/entry.S  |    6 ++++++
 arch/x86_64/kernel/i8259.c  |    4 ++++
 arch/x86_64/kernel/smp.c    |   23 +++++++++++++++++++++++
 include/asm-x86_64/hw_irq.h |    3 +--
 4 files changed, 34 insertions(+), 2 deletions(-)
arch/x86_64/kernel/entry.S +6 -0
Index: linux/arch/x86_64/kernel/entry.S
===================================================================
@@ -699,6 +699,12 @@ END(invalidate_interrupt\num)
 ENTRY(call_function_interrupt)
 	apicinterrupt CALL_FUNCTION_VECTOR,smp_call_function_interrupt
 END(call_function_interrupt)
+
+#ifdef CONFIG_CRASH_STOP_SUPPORTED
+ENTRY(crash_stop_interrupt)
+	apicinterrupt CRASH_STOP_VECTOR,smp_crash_stop_interrupt
+END(crash_stop_interrupt)
+#endif	/* CONFIG_CRASH_STOP_SUPPORTED */
 #endif
 
 ENTRY(apic_timer_interrupt)
arch/x86_64/kernel/i8259.c +4 -0
Index: linux/arch/x86_64/kernel/i8259.c
===================================================================
@@ -458,6 +458,7 @@ void invalidate_interrupt6(void);
 void invalidate_interrupt7(void);
 void thermal_interrupt(void);
 void threshold_interrupt(void);
+void crash_stop_interrupt(void);
 void i8254_timer_resume(void);
 
 static void setup_timer_hardware(void)
@@ -541,6 +542,9 @@ void __init init_IRQ(void)
 
 	/* IPI for generic function call */
 	set_intr_gate(CALL_FUNCTION_VECTOR, call_function_interrupt);
+#ifdef CONFIG_CRASH_STOP_SUPPORTED
+	set_intr_gate(CRASH_STOP_VECTOR, crash_stop_interrupt);
+#endif	/* CONFIG_CRASH_STOP_SUPPORTED */
 #endif	
 	set_intr_gate(THERMAL_APIC_VECTOR, thermal_interrupt);
 	set_intr_gate(THRESHOLD_APIC_VECTOR, threshold_interrupt);
arch/x86_64/kernel/smp.c +23 -0
Index: linux/arch/x86_64/kernel/smp.c
===================================================================
@@ -19,6 +19,7 @@
 #include <linux/kernel_stat.h>
 #include <linux/mc146818rtc.h>
 #include <linux/interrupt.h>
+#include <linux/crash_stop.h>
 
 #include <asm/mtrr.h>
 #include <asm/pgalloc.h>
@@ -522,3 +523,25 @@ asmlinkage void smp_call_function_interr
 	}
 }
 
+#ifdef	CONFIG_CRASH_STOP_SUPPORTED
+void
+cs_arch_send_ipi(int cpu)
+{
+	send_IPI_mask(cpumask_of_cpu(cpu), CRASH_STOP_VECTOR);
+}
+
+void
+cs_arch_send_nmi(int cpu)
+{
+	send_IPI_mask(cpumask_of_cpu(cpu), NMI_VECTOR);
+}
+
+asmlinkage void
+smp_crash_stop_interrupt(struct pt_regs *regs)
+{
+	struct pt_regs *old_regs = set_irq_regs(regs);
+	ack_APIC_irq();
+	cs_common_ipi();
+	set_irq_regs(old_regs);
+}
+#endif	/* CONFIG_CRASH_STOP_SUPPORTED */
include/asm-x86_64/hw_irq.h +1 -2
Index: linux/include/asm-x86_64/hw_irq.h
===================================================================
@@ -47,8 +47,7 @@
 #define ERROR_APIC_VECTOR	0xfe
 #define RESCHEDULE_VECTOR	0xfd
 #define CALL_FUNCTION_VECTOR	0xfc
-/* fb free - please don't readd KDB here because it's useless
-   (hint - think what a NMI bit does to a vector) */
+#define CRASH_STOP_VECTOR	0xfb
 #define THERMAL_APIC_VECTOR	0xfa
 #define THRESHOLD_APIC_VECTOR   0xf9
 /* f8 free */
#7 lore raw

PATCH 7/12 crash_stop: x86_64 specific code

In reply to Keith Owens

Add the x86_64 specific crash_stop code. This contains routines that are called from the common crash_stop code and from the x86_64 notify_die chain.

---
 arch/x86_64/kernel/Makefile     |    1 
 arch/x86_64/kernel/crash_stop.c |  128 ++++++++++++++++++++++++++++++++++++++++
 include/asm-x86_64/crash_stop.h |   14 ++++
 3 files changed, 143 insertions(+)
arch/x86_64/kernel/Makefile +1 -0
Index: linux/arch/x86_64/kernel/Makefile
===================================================================
@@ -37,6 +37,7 @@ obj-$(CONFIG_X86_PM_TIMER)	+= pmtimer.o
 obj-$(CONFIG_X86_VSMP)		+= vsmp.o
 obj-$(CONFIG_K8_NB)		+= k8.o
 obj-$(CONFIG_AUDIT)		+= audit.o
+obj-$(CONFIG_CRASH_STOP_SUPPORTED)	+= crash_stop.o
 
 obj-$(CONFIG_MODULES)		+= module.o
 obj-$(CONFIG_PCI)		+= early-quirks.o
arch/x86_64/kernel/crash_stop.c new file +128 -0

128 lines not on this page · raw message

include/asm-x86_64/crash_stop.h new file +14 -0
Index: linux/include/asm-x86_64/crash_stop.h
===================================================================
@@ -0,0 +1,14 @@
+#ifndef _ASM_CRASH_STOP_H
+#define _ASM_CRASH_STOP_H
+
+/* x86_64 uses multiple stacks so the registers (including rip) at the time of
+ * the interrupt can be on one stack while the crash_stop code is running on
+ * another stack.  We have to save the current rsp and rip.
+ */
+struct crash_stop_running_process_arch
+{
+	unsigned long rsp;
+	unsigned long rip;
+};
+
+#endif	/* _ASM_CRASH_STOP_H */
#8 lore raw

PATCH 8/12 crash_stop: ia64 interrupt handlers

In reply to Keith Owens

Define the ia64 crash_stop() interrupt handler and associated routines.

IPI_CRASH_STOP conflicts with the KDB vector. This is deliberate, one aim of the crash_stop() API is to remove all the interrupt code from the various kernel debug patches and use the common crash_stop code instead.

---
 arch/ia64/kernel/smp.c |   35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)
arch/ia64/kernel/smp.c +35 -0
Index: linux/arch/ia64/kernel/smp.c
===================================================================
@@ -30,6 +30,7 @@
 #include <linux/delay.h>
 #include <linux/efi.h>
 #include <linux/bitops.h>
+#include <linux/crash_stop.h>
 
 #include <asm/atomic.h>
 #include <asm/current.h>
@@ -66,6 +67,7 @@ static volatile struct call_data_struct 
 
 #define IPI_CALL_FUNC		0
 #define IPI_CPU_STOP		1
+#define IPI_CRASH_STOP		2
 
 /* This needs to be cacheline aligned because it is written to by *other* CPUs.  */
 static DEFINE_PER_CPU(u64, ipi_operation) ____cacheline_aligned;
@@ -156,6 +158,16 @@ handle_IPI (int irq, void *dev_id)
 				stop_this_cpu();
 				break;
 
+#ifdef	CONFIG_CRASH_STOP_SUPPORTED
+			      case IPI_CRASH_STOP:
+				{
+					extern void smp_crash_stop_interrupt(
+						    struct pt_regs *regs);
+					smp_crash_stop_interrupt(get_irq_regs());
+				}
+				break;
+#endif	/* CONFIG_CRASH_STOP_SUPPORTED */
+
 			      default:
 				printk(KERN_CRIT "Unknown IPI on CPU %d: %lu\n", this_cpu, which);
 				break;
@@ -379,3 +391,26 @@ setup_profiling_timer (unsigned int mult
 {
 	return -EINVAL;
 }
+
+#ifdef	CONFIG_CRASH_STOP_SUPPORTED
+void
+cs_arch_send_ipi(int cpu)
+{
+	send_IPI_single(cpu, IPI_CRASH_STOP);
+}
+
+void
+cs_arch_send_nmi(int cpu)
+{
+	set_mb(cs_arch_monarch_cpu, smp_processor_id());
+	platform_send_ipi(cpu, 0, IA64_IPI_DM_INIT, 0);
+}
+
+void
+smp_crash_stop_interrupt(struct pt_regs *regs)
+{
+	struct pt_regs *old_regs = set_irq_regs(regs);
+	cs_common_ipi();
+	set_irq_regs(old_regs);
+}
+#endif	/* CONFIG_CRASH_STOP_SUPPORTED */
#9 lore raw

PATCH 9/12 crash_stop: ia64 specific code

In reply to Keith Owens

Add the ia64 specific crash_stop code. This contains routines that are called from the common crash_stop code and from the ia64 notify_die chain.

---
 arch/ia64/kernel/Makefile     |    1 
 arch/ia64/kernel/crash_stop.c |  237 ++++++++++++++++++++++++++++++++++++++++++
 include/asm-ia64/crash_stop.h |   11 +
 3 files changed, 249 insertions(+)
arch/ia64/kernel/Makefile +1 -0
Index: linux/arch/ia64/kernel/Makefile
===================================================================
@@ -31,6 +31,7 @@ obj-$(CONFIG_KPROBES)		+= kprobes.o jpro
 obj-$(CONFIG_IA64_UNCACHED_ALLOCATOR)	+= uncached.o
 obj-$(CONFIG_AUDIT)		+= audit.o
 obj-$(CONFIG_PCI_MSI)		+= msi_ia64.o
+obj-$(CONFIG_CRASH_STOP_SUPPORTED)	+= crash_stop.o
 mca_recovery-y			+= mca_drv.o mca_drv_asm.o
 
 obj-$(CONFIG_IA64_ESI)		+= esi.o
arch/ia64/kernel/crash_stop.c new file +237 -0

237 lines not on this page · raw message

include/asm-ia64/crash_stop.h new file +11 -0
Index: linux/include/asm-ia64/crash_stop.h
===================================================================
@@ -0,0 +1,11 @@
+#ifndef _ASM_CRASH_STOP_H
+#define _ASM_CRASH_STOP_H
+
+struct crash_stop_running_process_arch
+{
+	struct switch_stack *sw;
+};
+
+extern int cs_arch_monarch_cpu;
+
+#endif	/* _ASM_CRASH_STOP_H */
#10 lore raw

PATCH 10/12 crash_stop: add to config system

In reply to Keith Owens

Add CONFIG_CRASH_STOP and CONFIG_CRASH_STOP_SUPPORTED. Debug code will select CRASH_STOP. That in turn will select CRASH_STOP_SUPPORTED if the config supports crash_stop(). If the supported architecture list changes then only the CRASH_STOP entry needs to be updated, none of the tools that select CRASH_STOP have to worry about arch dependent details.

---
 lib/Kconfig.debug |    7 +++++++
 1 file changed, 7 insertions(+)
lib/Kconfig.debug +7 -0
Index: linux/lib/Kconfig.debug
===================================================================
@@ -412,3 +412,10 @@ config LKDTM
 
 	Documentation on how to use the module can be found in
 	drivers/misc/lkdtm.c
+
+config CRASH_STOP
+	bool
+	select CRASH_STOP_SUPPORTED if ( IA64 || (X86 && !X86_VOYAGER) )
+
+config CRASH_STOP_SUPPORTED
+	bool
#11 lore raw

PATCH 11/12 crash_stop: demonstration code

In reply to Keith Owens

A quick and dirty crash_stop() demo program.

Notice that this demo is almost completely architecture independent. It has no code to do per architecture IPI/NMI, nor does it need to save any per cpu state - all of that is handled by the common code. IOW, a debug style tool can concentrate on its own requirements, leaving all the complicated code to the crash_stop API.

The base patch does not export any crash_stop() symbols. They can be added later if any debug style code can be installed in a module. Since the demo is best used as a module, this patch temporarilly exports some symbols.

No signed-off-by, this code is not going into the kernel.

---
 kernel/Makefile          |    1 
 kernel/crash_stop.c      |    6 +-
 kernel/crash_stop_demo.c |  134 +++++++++++++++++++++++++++++++++++++++++++++++
 lib/Kconfig.debug        |   11 +++
 4 files changed, 151 insertions(+), 1 deletion(-)
kernel/Makefile +1 -0
Index: linux/kernel/Makefile
===================================================================
@@ -52,6 +52,7 @@ obj-$(CONFIG_UTS_NS) += utsname.o
 obj-$(CONFIG_TASK_DELAY_ACCT) += delayacct.o
 obj-$(CONFIG_TASKSTATS) += taskstats.o tsacct.o
 obj-$(CONFIG_CRASH_STOP_SUPPORTED) += crash_stop.o
+obj-$(CONFIG_CRASH_STOP_DEMO) += crash_stop_demo.o
 
 ifneq ($(CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER),y)
 # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is
kernel/crash_stop.c +5 -1
Index: linux/kernel/crash_stop.c
===================================================================
@@ -197,6 +197,7 @@
 #include <linux/crash_stop.h>
 #include <linux/interrupt.h>
 #include <linux/kernel.h>
+#include <linux/module.h>		/* used by crash_stop_demo */
 #include <linux/ptrace.h>
 #include <linux/nmi.h>
 #include <linux/spinlock.h>
@@ -335,7 +336,7 @@ retry:
 	set_mb(cs_notify_chain_owner, cpu);
 	set_mb(cs_lock_owner, -1);
 	spin_unlock(&cs_lock);
-	crash_stop(cs_notify_callback, NULL, NULL, regs, __FUNCTION__);
+	crash_stop(cs_notify_callback, NULL, printk, regs, __FUNCTION__);
 }
 
 /* Called by the arch specific crash_stop code, when they reach the end of a
@@ -463,6 +464,7 @@ crash_stop_sent_nmi(void)
 {
 	return cpu_isset(smp_processor_id(), cs_sent_nmi);
 }
+EXPORT_SYMBOL(crash_stop_sent_nmi);	/* used by crash_stop_demo */
 #endif	/* CONFIG_SMP */
 
 /* Should only be called by the arch specific crash_stop code, after they have
@@ -752,6 +754,7 @@ retry:
 	set_mb(cs_leaving, 0);
 	return 0;
 }
+EXPORT_SYMBOL(crash_stop);		/* used by crash_stop_demo */
 
 /**
  * crash_stop_recovered: - Release any slaves in crash_stop state.
@@ -841,3 +844,4 @@ crash_stop_slaves(void)
 	else
 		return 0;
 }
+EXPORT_SYMBOL(crash_stop_slaves);	/* used by crash_stop_demo */
kernel/crash_stop_demo.c new file +134 -0

134 lines not on this page · raw message

lib/Kconfig.debug +11 -0
Index: linux/lib/Kconfig.debug
===================================================================
@@ -419,3 +419,14 @@ config CRASH_STOP
 
 config CRASH_STOP_SUPPORTED
 	bool
+
+config CRASH_STOP_DEMO
+	tristate "Demonstrate the use of crash_stop"
+	default m
+	select CRASH_STOP
+	help
+          Code to demonstrate the use of crash_stop.  Build it as a
+          module and load it.  It will make one cpu spin disabled then
+          call crash_stop.  All slave cpus bar one will get a normal
+          IPI, the spinning cpu will get NMI.  You need at least 3 cpus
+          to run crash_stop_demo.
#12 lore raw

PATCH 12/12 crash_stop: test code

In reply to Keith Owens

A quick and dirty crash_stop() test program. Most of the code is to get the machine into a suitable state for testing both the normal IPI and NMI code. The interesting crash_stop bits are cs_test_callback*() and simulate_crash_stop_event().

No signed-off-by, this code is not going into the kernel.

---
 kernel/Makefile          |    1 
 kernel/crash_stop_test.c |  177 +++++++++++++++++++++++++++++++++++++++++++++++
 lib/Kconfig.debug        |   11 ++
 3 files changed, 189 insertions(+)
kernel/Makefile +1 -0
Index: linux/kernel/Makefile
===================================================================
@@ -53,6 +53,7 @@ obj-$(CONFIG_TASK_DELAY_ACCT) += delayac
 obj-$(CONFIG_TASKSTATS) += taskstats.o tsacct.o
 obj-$(CONFIG_CRASH_STOP_SUPPORTED) += crash_stop.o
 obj-$(CONFIG_CRASH_STOP_DEMO) += crash_stop_demo.o
+obj-$(CONFIG_CRASH_STOP_TEST) += crash_stop_test.o
 
 ifneq ($(CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER),y)
 # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is
kernel/crash_stop_test.c new file +177 -0

177 lines not on this page · raw message

lib/Kconfig.debug +11 -0
Index: linux/lib/Kconfig.debug
===================================================================
@@ -430,3 +430,14 @@ config CRASH_STOP_DEMO
           call crash_stop.  All slave cpus bar one will get a normal
           IPI, the spinning cpu will get NMI.  You need at least 3 cpus
           to run crash_stop_demo.
+
+config CRASH_STOP_TEST
+	tristate "Test crash_stop"
+	default m
+	help
+          Code to test the use of crash_stop.  Build it as a module and
+          load it.  It will make one cpu spin disabled then generate an
+          oops or NMI.  All slave cpus bar one will get a normal IPI,
+          the spinning cpu will get NMI.  You need at least 3 cpus to
+          run crash_stop_test.  You can also test the NMI watchdog and
+          oops handling of crash_stop, see kernel/crash_stop_test.c.
#13 lore raw
In reply to Keith Owens

On Thu, Nov 09, 2006 at 03:04:18PM +1100, Keith Owens wrote:
Hi Keith,

Quoted 6 lines

All the kernel debug style tools (kdb, kgdb, nlkd, netdump, lkcd, crash, kdump etc.) have a common requirement, they need to do a crash stop of the systems. This means stopping all the cpus, even if some of the cpus are spinning disabled. In addition, each cpu has to save enough state to start the diagnosis of the problem.

Quoted 9 lines

* Each debug style tool has written its own code for interrupting the other cpus and for saving cpu state.

* Some tools try a normal IPI first then send a non-maskable interrupt after a delay.

* Some tools always send a NMI first, which can result in incomplete or wrong machine state if NMI arrives at the wrong time.

What kind of problem one can run into if NMI is sent directly instead of trying an normal IPI first?

On a general note, I am not sure how well suited this infrastructure is for crash dump needs. We are trying to follow one theme and that is run a bare minimal code after a system crash to increase reliabitity.

Avoid taking locks avoid relying on crashed task's stack etc. (if possible). Of course it is an ideal situation and we have not achieved that state but roughly that seems to be the long term goal. Looking at the patches it looks like it introduces lots of code to be run after crash and also uses smp_processor_id() which introduces a dependency on stack. This poses problem in stack overflow cases. Fernando from valinux, introduced safe_smp_processor_id() call to read apic id from LAPIC instead of relying on the thread's stack. As of today, the crash path is no safe from stack overflow but we hope that someday it would be.

So I think crash dump will be a little special case and we need to be careful while introducing extra code that new code will be able to run even if system is in very bad state. (for example stack overflow).

Thanks
Vivek

#14 lore raw
Quoted 15 lines · Vivek Goyal (on Fri, 10 Nov 2006 20:45:05 -0500) wrote:

On Thu, Nov 09, 2006 at 03:04:18PM +1100, Keith Owens wrote:
Hi Keith,

Quoted 12 lines

All the kernel debug style tools (kdb, kgdb, nlkd, netdump, lkcd, crash, kdump etc.) have a common requirement, they need to do a crash stop of the systems. This means stopping all the cpus, even if some of the cpus are spinning disabled. In addition, each cpu has to save enough state to start the diagnosis of the problem.

* Each debug style tool has written its own code for interrupting the other cpus and for saving cpu state.

* Some tools try a normal IPI first then send a non-maskable interrupt after a delay.

Quoted 6 lines
Quoted 3 lines

* Some tools always send a NMI first, which can result in incomplete or wrong machine state if NMI arrives at the wrong time.

What kind of problem one can run into if NMI is sent directly instead of trying an normal IPI first?

Incomplete cpu state on the cpus that are hit with NMI. By definition, NMI can be delivered at any time, with the cpu in any state. It can be in the middle of saving the state for a previous interrupt when NMI is delivered. On IA64, a cpu can even be in physical mode instead of virtual mode when INIT is delivered. All of which means that you have incomplete or misleading information in your dump.

Sending a normal interrupt first, waiting a short while, then sending an NMI later maximises the chance that we get good state for every cpu.

Quoted 3 lines

On a general note, I am not sure how well suited this infrastructure is for crash dump needs. We are trying to follow one theme and that is run a bare minimal code after a system crash to increase reliabitity.

Agreed. But you still have to stop all the existing cpus _and_ capture their state before switching to the second kernel. There is no point in switching to a new kernel if you cannot get information about all the cpus from the failing kernel.

Quoted 9 lines

Avoid taking locks avoid relying on crashed task's stack etc. (if possible). Of course it is an ideal situation and we have not achieved that state but roughly that seems to be the long term goal. Looking at the patches it looks like it introduces lots of code to be run after crash and also uses smp_processor_id() which introduces a dependency on stack. This poses problem in stack overflow cases. Fernando from valinux, introduced safe_smp_processor_id() call to read apic id from LAPIC instead of relying on the thread's stack. As of today, the crash path is no safe from stack overflow but we hope that someday it would be.

Good point. I will look at converting crash_stop to use safe_smp_processor_id.