Skip to contentSkip to conversation

Common API for bring the system to a crash_stop state

RFC New Keith Owens 2 participants 10 messages Oct 13, 2006

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

This initial mail is going to a wide distribution because there are several different people and groups who are working on kernel debug style tools. These tools include debuggers such as kdb, kgdb, nlkd. There are also kernel dump tools like netdump, lkcd, crash, kexec/kdump and others. To cut down the cross list noise, I have arbitrarily designated [email protected] as the only list to receive and discuss the patches that follow this initial mail.

Reply-To is set to [email protected], please honour it and trim the rest of the cc: list.

-------------------------------------------------------------------------------------

All the kernel debug style tools (kdb, kgdb, nlkd, netdump, lkcd, crash, kdump etc.) have a common requirement, they need to do a crash stop of the systems. This means stopping all the cpus, even if some of the cpus are spinning disabled. In addition, each cpu has to save enough state to start diagnosis of the problem.

* Each debug style tool has written its own code for interrupting the other cpus and for saving cpu state.

* Some tools try a normal IPI first then send a non-maskable interrupt after a delay.

* Some tools always send a NMI first, which can result in incomplete machine state if it arrives at the wrong time.

* Most of the tools do not know how to cope with the IA64 architecture defined rendezvous algorithm, which interferes with an OS driven rendezvous.

* Needless to say, every single patch set conflicts with all the others, which makes it very difficult to install more than one of the tools at a time.

The solution is to define a common crash_stop API that can be used by _all_ of the debug style tools, without reinventing the wheel each time.

The following crash_stop patches will only appear on linux-arch.

crash_stop_headers The common and i386 crash_stop.h files

crash_stop_i386_handler Add the crash_stop i386 interrupt handlers.
This patch changes existing i386 files. It needs testing on visw and updating for voyager.

crash_stop_i386 I386 specific crash_stop code.

crash_stop_common Architecture independent crash_stop code.

crash_stop_common_Kconfig Kconfig change to activate crash_stop.

crash_stop_demo A demo module to test crash_stop().

This is a work in progress, it does most of the job on i386. x86_64 will be easy once i386 is working. I have an incomplete patch for ia64, coexxisting with the MCA/INIT rendezvous algorithm is non-trivial. At the moment, I am more interested in feedback on the design of the API, to ensure that it suits everybody's requirements.

Most of the design documentation is in the crash_stop_common patch. Please read that before replying.

#1 lore raw

RFC 1/1 crash_stop: crash_stop_headers

crash_stop() headers. The common header defines the API. The architecture headers define arch dependent state that is saved on each cpu, this is typically the data that is required to get a decent backtrace.

---
 include/asm-i386/crash_stop.h |   14 ++++++++++
 include/linux/crash_stop.h    |   57 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 71 insertions(+)
include/asm-i386/crash_stop.h new file +14 -0
Index: linux/include/asm-i386/crash_stop.h
===================================================================
@@ -0,0 +1,14 @@
+#ifndef _ASM_CRASH_STOP_H
+#define _ASM_CRASH_STOP_H
+
+/* CONFIG_4KSTACKS means that the registers (including eip) at the time of the
+ * interrupt can be on one stack while the crash_stop code is running on
+ * another stack.  We have to save the current esp and eip.
+ */
+struct crash_stop_running_process_arch
+{
+	unsigned long esp;
+	unsigned long eip;
+};
+
+#endif	/* _ASM_CRASH_STOP_H */
include/linux/crash_stop.h new file +57 -0

57 lines not on this page · raw message

#2 lore raw

RFC 1/1 crash_stop: crash_stop_i386_handler

In reply to Keith Owens

Define the i386 crash_stop() interrupt handler and associated routines.

Yes, I know that CRASH_STOP_VECTOR conflicts with the lkcd vector. This is deliberate, one aim of the crash_stop() API is to remove all the interrupt code from the various kernel debug patches.

Note 1: I have patched visw but I cannot test it, no hardware.

Note 2: This patch does not cover i386 voyager. I do not understand
the voyager interrupt mechanism well enough to define its interrupt handlers (hint, hint).

---
 arch/i386/kernel/smp.c                      |   21 +++++++++++++++++++++
 arch/i386/kernel/smpboot.c                  |    4 ++++
 include/asm-i386/hw_irq.h                   |    1 +
 include/asm-i386/mach-default/entry_arch.h  |    3 +++
 include/asm-i386/mach-default/irq_vectors.h |    1 +
 include/asm-i386/mach-visws/entry_arch.h    |    3 +++
 6 files changed, 33 insertions(+)
arch/i386/kernel/smp.c +21 -0
Index: linux/arch/i386/kernel/smp.c
===================================================================
@@ -20,6 +20,7 @@
 #include <linux/interrupt.h>
 #include <linux/cpu.h>
 #include <linux/module.h>
+#include <linux/crash_stop.h>
 
 #include <asm/mtrr.h>
 #include <asm/tlbflush.h>
@@ -727,3 +728,23 @@ int safe_smp_processor_id(void)
 
 	return cpuid >= 0 ? cpuid : 0;
 }
+
+#ifdef	CONFIG_CRASH_STOP_SUPPORTED
+void
+cs_arch_send_ipi(int cpu)
+{
+	send_IPI_mask(cpumask_of_cpu(cpu), CRASH_STOP_VECTOR);
+}
+
+void
+cs_arch_send_nmi(int cpu)
+{
+	send_IPI_mask(cpumask_of_cpu(cpu), NMI_VECTOR);
+}
+
+fastcall void smp_crash_stop_interrupt(struct pt_regs *regs)
+{
+	ack_APIC_irq();
+	cs_common_ipi(regs);
+}
+#endif	/* CONFIG_CRASH_STOP_SUPPORTED */
arch/i386/kernel/smpboot.c +4 -0
Index: linux/arch/i386/kernel/smpboot.c
===================================================================
@@ -1497,6 +1497,10 @@ void __init smp_intr_init(void)
 
 	/* IPI for generic function call */
 	set_intr_gate(CALL_FUNCTION_VECTOR, call_function_interrupt);
+
+#ifdef	CONFIG_CRASH_STOP_SUPPORTED
+	set_intr_gate(CRASH_STOP_VECTOR, crash_stop_interrupt);
+#endif	/* CONFIG_CRASH_STOP_SUPPORTED */
 }
 
 /*
include/asm-i386/hw_irq.h +1 -0
Index: linux/include/asm-i386/hw_irq.h
===================================================================
@@ -35,6 +35,7 @@ extern void (*interrupt[NR_IRQS])(void);
 fastcall void reschedule_interrupt(void);
 fastcall void invalidate_interrupt(void);
 fastcall void call_function_interrupt(void);
+fastcall void crash_stop_interrupt(void);
 #endif
 
 #ifdef CONFIG_X86_LOCAL_APIC
include/asm-i386/mach-default/entry_arch.h +3 -0
Index: linux/include/asm-i386/mach-default/entry_arch.h
===================================================================
@@ -13,6 +13,9 @@
 BUILD_INTERRUPT(reschedule_interrupt,RESCHEDULE_VECTOR)
 BUILD_INTERRUPT(invalidate_interrupt,INVALIDATE_TLB_VECTOR)
 BUILD_INTERRUPT(call_function_interrupt,CALL_FUNCTION_VECTOR)
+#ifdef	CONFIG_CRASH_STOP_SUPPORTED
+BUILD_INTERRUPT(crash_stop_interrupt,CRASH_STOP_VECTOR)
+#endif	/* CONFIG_CRASH_STOP_SUPPORTED */
 #endif
 
 /*
include/asm-i386/mach-default/irq_vectors.h +1 -0
Index: linux/include/asm-i386/mach-default/irq_vectors.h
===================================================================
@@ -48,6 +48,7 @@
 #define INVALIDATE_TLB_VECTOR	0xfd
 #define RESCHEDULE_VECTOR	0xfc
 #define CALL_FUNCTION_VECTOR	0xfb
+#define CRASH_STOP_VECTOR	0xfa
 
 #define THERMAL_APIC_VECTOR	0xf0
 /*
include/asm-i386/mach-visws/entry_arch.h +3 -0
Index: linux/include/asm-i386/mach-visws/entry_arch.h
===================================================================
@@ -7,6 +7,9 @@
 BUILD_INTERRUPT(reschedule_interrupt,RESCHEDULE_VECTOR)
 BUILD_INTERRUPT(invalidate_interrupt,INVALIDATE_TLB_VECTOR)
 BUILD_INTERRUPT(call_function_interrupt,CALL_FUNCTION_VECTOR)
+#ifdef	CONFIG_CRASH_STOP_SUPPORTED
+BUILD_INTERRUPT(crash_stop_interrupt,CRASH_STOP_VECTOR)
+#endif	/* CONFIG_CRASH_STOP_SUPPORTED */
 #endif
 
 /*
#3 lore raw

Re: [RFC] crash_stop: crash_stop_i386_handler

Quoted 3 lines · Keith Owens

Note 2: This patch does not cover i386 voyager. I do not understand
the voyager interrupt mechanism well enough to define its interrupt handlers (hint, hint).

Voyager actually doesn't need a software based crash dump at all.

The system itself has a type of GSP that runs SUS (sort of a bios) continuously. SUS will dump a complete memory and state image to a nominated dump slice (usually swap) if the OS becomes unresponsive or requests it. So all I really have to do is come up with a user space tool that converts from this raw image format to your crash tool format.

James

#4 lore raw

Re: [RFC] crash_stop: crash_stop_i386_handler

Quoted 6 lines · James Bottomley (on Fri, 13 Oct 2006 08:55:35 -0500) wrote:
Quoted 3 lines · Keith Owens

Note 2: This patch does not cover i386 voyager. I do not understand
the voyager interrupt mechanism well enough to define its interrupt handlers (hint, hint).

Voyager actually doesn't need a software based crash dump at all.

This API is designed for any kernel debug tool, not just crash dump. This includes run time debuggers such as kdb, kgdb and nlkd.

Quoted 5 lines

The system itself has a type of GSP that runs SUS (sort of a bios) continuously. SUS will dump a complete memory and state image to a nominated dump slice (usually swap) if the OS becomes unresponsive or requests it. So all I really have to do is come up with a user space tool that converts from this raw image format to your crash tool format.

Not my crash format, other people maintain the crash dump tools.

#5 lore raw

RFC 1/1 crash_stop: crash_stop_i386

In reply to Keith Owens

Add the i386 specific crash_stop code. This contains routines that are called from the common crash_stop code and from the i386 notify_die chain.

Note 1: Not tested on visw nor voyager, no hardware.

Note 2: It is missing the code to handle a notify_chain (see scenario 4
in kernel/crash_stop.c in a later patch). I want to make sure that the API design is right before adding that code.

---
 arch/i386/kernel/Makefile     |    1 
 arch/i386/kernel/crash_stop.c |   75 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 76 insertions(+)
arch/i386/kernel/Makefile +1 -0
Index: linux/arch/i386/kernel/Makefile
===================================================================
@@ -39,6 +39,7 @@ obj-$(CONFIG_VM86)		+= vm86.o
 obj-$(CONFIG_EARLY_PRINTK)	+= early_printk.o
 obj-$(CONFIG_HPET_TIMER) 	+= hpet.o
 obj-$(CONFIG_K8_NB)		+= k8.o
+obj-$(CONFIG_CRASH_STOP_SUPPORTED)	+= crash_stop.o
 
 EXTRA_AFLAGS   := -traditional
 
arch/i386/kernel/crash_stop.c new file +75 -0

75 lines not on this page · raw message

#6 lore raw

RFC 1/1 crash_stop: crash_stop_common

In reply to Keith Owens

The common crash_stop code, including the design level documentation.

This patch is missing the code to handle a notify_chain (see scenario 4 in kernel/crash_stop.c). I want to make sure that the API design is right before adding that code.

---
 kernel/Makefile     |    1 
 kernel/crash_stop.c |  563 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 564 insertions(+)
kernel/Makefile +1 -0
Index: linux/kernel/Makefile
===================================================================
@@ -51,6 +51,7 @@ obj-$(CONFIG_RELAY) += relay.o
 obj-$(CONFIG_UTS_NS) += utsname.o
 obj-$(CONFIG_TASK_DELAY_ACCT) += delayacct.o
 obj-$(CONFIG_TASKSTATS) += taskstats.o tsacct.o
+obj-$(CONFIG_CRASH_STOP_SUPPORTED) += crash_stop.o
 
 ifneq ($(CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER),y)
 # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is
kernel/crash_stop.c new file +563 -0

563 lines not on this page · raw message

#7 lore raw

RFC 1/1 crash_stop: crash_stop_common_Kconfig

In reply to Keith Owens

Add CONFIG_CRASH_STOP and CONFIG_CRASH_STOP_SUPPORTED. Debug code will select CRASH_STOP. That in turn will select CRASH_STOP_SUPPORTED if the config supports crash_stop(). If the supported architecture list changes then only the CRASH_STOP entty needs to be updated, all the tools that select CRASH_STOP do not have to worry about arch dependent details.

lib/Kconfig.debug +7 -0
Index: linux/lib/Kconfig.debug
===================================================================
@@ -398,3 +398,10 @@ config LKDTM
 
 	Documentation on how to use the module can be found in
 	drivers/misc/lkdtm.c
+
+config CRASH_STOP
+	bool
+	select CRASH_STOP_SUPPORTED if SMP && ( (X86 && !X86_VOYAGER) )
+
+config CRASH_STOP_SUPPORTED
+	bool
#8 lore raw

RFC 1/1 crash_stop: crash_stop_demo

In reply to Keith Owens

A quick and dirty crash_stop() demo program. Most of the code is to get the machine into a suitable state for testing both the normal IPI and NMI code. The interesting crash_stop bits are cs_demo_callback*() and simulate_crash_stop_event().

The base patch does not export any crash_stop() symbols. They can be added later if any debug style code can be installed in a module. Since the demo is best used as a module, this patch temporarilly exports some symbols.

---
 kernel/Makefile          |    1 
 kernel/crash_stop.c      |    5 +
 kernel/crash_stop_demo.c |  160 +++++++++++++++++++++++++++++++++++++++++++++++
 lib/Kconfig.debug        |   12 +++
 4 files changed, 177 insertions(+), 1 deletion(-)
kernel/Makefile +1 -0
Index: linux/kernel/Makefile
===================================================================
@@ -52,6 +52,7 @@ obj-$(CONFIG_UTS_NS) += utsname.o
 obj-$(CONFIG_TASK_DELAY_ACCT) += delayacct.o
 obj-$(CONFIG_TASKSTATS) += taskstats.o tsacct.o
 obj-$(CONFIG_CRASH_STOP_SUPPORTED) += crash_stop.o
+obj-$(CONFIG_CRASH_STOP_DEMO) += crash_stop_demo.o
 
 ifneq ($(CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER),y)
 # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is
kernel/crash_stop.c +4 -1
Index: linux/kernel/crash_stop.c
===================================================================
@@ -134,6 +134,7 @@
 #include <linux/delay.h>
 #include <linux/crash_stop.h>
 #include <linux/kernel.h>
+#include <linux/module.h>		/* used by crash_stop_demo */
 #include <linux/ptrace.h>
 #include <linux/nmi.h>
 #include <linux/spinlock.h>
@@ -255,6 +256,7 @@ crash_stop_sent_nmi(void)
 {
 	return cpu_isset(smp_processor_id(), cs_sent_nmi);
 }
+EXPORT_SYMBOL(crash_stop_sent_nmi);	/* used by crash_stop_demo */
 
 /* Should only be called by the arch specific crash_stop code, after they have
  * saved any arch specific state.  The call chain is :-
@@ -307,7 +309,7 @@ cs_wait_for_cpus(void)
 {
 	int count, prev_count = 0, sent_nmi = 0, t, wait_secs, slaves, cpu;
 	slaves = num_online_cpus() - 1;
-	wait_secs = min(3, (slaves * 100) / 1000);
+	wait_secs = 3 + (slaves * 100) / 1000;
 	cs_mdelay(100);
 	for (t = 0; t < wait_secs; ++t) {
 		count = 0;
@@ -495,6 +497,7 @@ retry:
 	spin_unlock(&cs_lock);
 	return 0;
 }
+EXPORT_SYMBOL(crash_stop);		/* used by crash_stop_demo */
 
 /**
  * crash_stop_recovered: - Release any slaves in crash_stop state.
kernel/crash_stop_demo.c new file +160 -0

160 lines not on this page · raw message

lib/Kconfig.debug +12 -0
Index: linux/lib/Kconfig.debug
===================================================================
@@ -405,3 +405,15 @@ config CRASH_STOP
 
 config CRASH_STOP_SUPPORTED
 	bool
+
+config CRASH_STOP_DEMO
+	tristate "Demonstrate the use of crash_stop"
+	default m
+	depends on SMP
+	select CRASH_STOP
+	help
+          Code to demonstrate the use of crash_stop.  Build it as a
+          module and load it.  It will make one cpu spin disabled then
+          call crash_stop.  All slave cpus bar one will get a normal
+          IPI, the spinning cpu will get NMI.  You need at least 3 cpus
+          to run crash_stop_demo.
#9 lore raw
In reply to Keith Owens
Quoted 5 lines · Keith Owens (on Fri, 13 Oct 2006 23:23:35 +1000) wrote:

The solution is to define a common crash_stop API that can be used by _all_ of the debug style tools, without reinventing the wheel each time.

The following crash_stop patches will only appear on linux-arch.

Forgot to mention that these patches are against 2.6.19-rc1. They will have to change for 2.6.19-rc2 due to the pt_regs clean up.