kdump's dump capture kernel fails (APICs & linux-2.6.15-rc1)
Hi Eric,
It seems there are still some problems with the initialization of the APICs in the second kernel. As you can see from the serial line captures below, early during the boot process the kernel is flooded with "unexpected IRQ trap at vector ##" errors and fails to boot.
Some basic details about the machine:
- Dual Xeon Hyper-Threading 2.4GHz
- 1GB RAM
In this particular machine the problem could be solved by reintroducing the APICs shutdown patch. The drawback with this approach is that LAPIC/IOAPIC registers setup for the legacy PIC mode differs a lot between mother boards and the somewhat hackish shutdown code fails to set the APICs properly under certain configurations.
In "mkdump" we take a different method that solves these problems: save the original state of the APICs (i.e. as configured by the BIOS) before they are reconfigured by the kernel, and restore the APICs to that original state before kexecing the second kernel.
I feel this is the safest (and most paranoid) solution. But, of course, I concede that it would be preferable to avoid touching the APICs in the reboot patch to the second kernel if possible. I hope we can get the "moving apic init in init_IRQs" patches to work reliably.
The APICs save/restore approach was originally proposed by Jan Kratochvil in the Fastboot mailing list:
http://lists.osdl.org/pipermail/fastboot/2005-May/001383.html
I have prepared "LAPIC/IO-APIC save/restore" patches against "linux-2.6.15-rc1" (attached).
I would appreciate comments on this problematic regarding the APICs.
Regards,
Fernando
******** WITH ACPI ENABLED
# kexec -p linux-2.6.15-rc1-cptr/vmlinux --args-linux --append="root=/dev/sda1 init 1 irqpoll console=ttyS0,38400 console=tty0" # SysRq : Trigger a crashdump Linux version 2.6.15-rc1-cptr (fewy@nexus) (gcc version 4.0.3 20051023 (prerelease) (Debian 4.0.2-3)) #1 PREEMPT Thu Nov 17 11:19:24 JST 2005 BIOS-provided physical RAM map: BIOS-e820: 0000000000000100 - 000000000009f800 (usable) BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved) BIOS-e820: 0000000000100000 - 000000003fee0000 (usable) BIOS-e820: 000000003fee0000 - 000000003fee3000 (ACPI NVS) BIOS-e820: 000000003fee3000 - 000000003fef0000 (ACPI data) BIOS-e820: 000000003fef0000 - 000000003ff00000 (reserved) BIOS-e820: 00000000fec00000 - 0000000100000000 (reserved) user-defined physical RAM map: user: 0000000000000000 - 00000000000a0000 (usable) user: 0000000001000000 - 000000000128a000 (usable) user: 000000000132a400 - 0000000005000000 (usable) 0MB HIGHMEM available. 80MB LOWMEM available. found SMP MP-table at 000f5750 DMI 2.3 present. ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) Processor #0 15:2 APIC version 20 ACPI: LAPIC (acpi_id[0x01] lapic_id[0x06] enabled) Processor #6 15:2 APIC version 20 WARNING: NR_CPUS limit of 1 reached. Processor ignored. ACPI: LAPIC (acpi_id[0x02] lapic_id[0x07] enabled) Processor #7 15:2 APIC version 20 WARNING: NR_CPUS limit of 1 reached. Processor ignored. ACPI: LAPIC (acpi_id[0x03] lapic_id[0x01] enabled) Processor #1 15:2 APIC version 20 WARNING: NR_CPUS limit of 1 reached. Processor ignored. ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1]) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1]) ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1]) ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1]) ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0]) IOAPIC[0]: apic_id 4, version 32, address 0xfec00000, GSI 0-23 ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) Enabling APIC mode: Flat. Using 1 I/O APICs Using ACPI (MADT) for SMP configuration information Allocating PCI resources starting at 10000000 (gap: 05000000:fb000000) Built 1 zonelists Kernel command line: root=/dev/sda1 init 1 irqpoll console=ttyS0,38400 console=tty0 memmap=exactmap memmap=640K@0K memmap=2600K@16384K memmap=62295K@19625K elfcorehdr=19624K Misrouted IRQ fixup and polling support enabled This may significantly impact system performance Initializing CPU#0 CPU 0 irqstacks, hard=c124e000 soft=c124d000 PID hash table entries: 512 (order: 9, 8192 bytes) Detected 2405.933 MHz processor. Using tsc for high-res timesource Console: colour VGA+ 80x25 unexpected IRQ trap at vector 89 unexpected IRQ trap at vector 29 unexpected IRQ trap at vector 89 unexpected IRQ trap at vector 19 .....................
******** WITH ACPI DISABLED
# kexec -p linux-2.6.15-rc1-cptr/vmlinux --args-linux
--append="root=/dev/sda1 init 1 irqpoll console=ttyS0,38400
console=tty0"
# SysRq : Trigger a crashdump
Linux version 2.6.15-rc1-cptr (fewy@nexus) (gcc version 4.0.3 20051023
(prerelease) (Debian 4.0.2-3)) #1 Thu Nov 17 15:46:25 JST 2005
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000100 - 000000000009f800 (usable)
BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)
BIOS-e820: 0000000000100000 - 000000003fee0000 (usable)
BIOS-e820: 000000003fee0000 - 000000003fee3000 (ACPI NVS)
BIOS-e820: 000000003fee3000 - 000000003fef0000 (ACPI data)
BIOS-e820: 000000003fef0000 - 000000003ff00000 (reserved)
BIOS-e820: 00000000fec00000 - 0000000100000000 (reserved)
user-defined physical RAM map:
user: 0000000000000000 - 00000000000a0000 (usable)
user: 0000000001000000 - 000000000123c000 (usable)
user: 00000000012dc400 - 0000000005000000 (usable)
0MB HIGHMEM available.
80MB LOWMEM available.
found SMP MP-table at 000f5750
DMI 2.3 present.
Intel MultiProcessor Specification v1.4
Virtual Wire compatibility mode.
OEM ID: OEM00000 Product ID: PROD00000000 APIC at: 0xFEE00000
Processor #0 15:2 APIC version 17
Processor #6 15:2 APIC version 17
WARNING: NR_CPUS limit of 1 reached. Processor ignored.
I/O APIC #4 Version 17 at 0xFEC00000.
Enabling APIC mode: Flat. Using 1 I/O APICs
Processors: 1
Allocating PCI resources starting at 10000000 (gap: 05000000:fb000000)
Built 1 zonelists
Kernel command line: root=/dev/sda1 init 1 irqpoll console=ttyS0,38400
console=tty0 memmap=exactmap memmap=640K@0K memmap=2288K@16384K
memmap=62607K@19313K elfcorehdr=19312K
Misrouted IRQ fixup and polling support enabled
This may significantly impact system performance
Initializing CPU#0
PID hash table entries: 512 (order: 9, 8192 bytes)
Detected 2405.819 MHz processor.
Using tsc for high-res timesource
Console: colour VGA+ 80x25
unexpected IRQ trap at vector 89
unexpected IRQ trap at vector 29
unexpected IRQ trap at vector 89
.....................Signature · 1 line
Fernando Luis Vázquez Cao @ NTT Data Intellilink
arch/i386/kernel/crash.c +44 -1
@@ -22,7 +22,8 @@#include <asm/nmi.h> #include <asm/hw_irq.h> #include <mach_ipi.h> - +#include <asm/apic.h> +#include <asm/io_apic.h> note_buf_t crash_notes[NR_CPUS ];/* This keeps a track of which one is crashing cpu. */
@@ -127,6 +128,39 @@ static void crash_save_self(struct pt_recrash_save_this_cpu (®s, cpu); } +#ifdef CONFIG_X86_IO_APIC +static void ioapic_kexec_restore_once (void ) +{ +static int tried[NR_CPUS ]; +int cpu =smp_processor_id (); + +/* We may crash inside: ioapic_kexec_restore() */ +if (tried[cpu]) +return ; + tried[cpu] =1 ; + +ioapic_kexec_restore ();/* errors ignored */ +} +#endif + +#ifdef CONFIG_X86_LOCAL_APIC +void lapic_kexec_restore_once (void ) +{ +static int tried[NR_CPUS ]; +int cpu =smp_processor_id (); + +if (!cpu_has_apic) +return ; + +/* We may crash inside: lapic_kexec_restore() */ +if (tried[cpu]) +return ; + tried[cpu] =1 ; + +lapic_kexec_restore ();/* errors ignored */ +} +#endif +#ifdef CONFIG_SMP static atomic_t waiting_for_crash_ipi;
@@ -147,6 +181,9 @@ static int crash_nmi_callback(struct pt_ regs = &fixed_regs; }crash_save_this_cpu (regs, cpu); +#if defined(CONFIG_X86_LOCAL_APIC ) +lapic_kexec_restore_once (); +#endif atomic_dec (&waiting_for_crash_ipi);/* Assume hlt works */ halt ();
@@ -210,5 +247,11 @@ void machine_crash_shutdown(struct pt_re/* Make a note of crashing cpu. Will be used in NMI callback.*/ crashing_cpu =smp_processor_id ();nmi_shootdown_cpus (); +#ifdef CONFIG_X86_IO_APIC +ioapic_kexec_restore_once (); +#endif /* CONFIG_X86_IO_APIC */ +#ifdef CONFIG_X86_LOCAL_APIC +lapic_kexec_restore_once (); +#endif crash_save_self (regs); }
arch/i386/kernel/io_apic.c +98 -46
192 lines not on this page · raw message
include/asm-i386/io_apic.h +2 -0
@@ -209,5 +209,7 @@ extern int (*ioapic_renumber_irq)(int io#endif extern int assign_irq_vector (int irq); +extern int ioapic_kexec_restore (void ); +extern int ioapic_kdump_restore (void );#endif
arch/x86_64/kernel/io_apic.c +98 -46
192 lines not on this page · raw message
include/asm-x86_64/io_apic.h +2 -0
@@ -214,6 +214,8 @@ extern int sis_apic_bug; /* dummy */#endif extern int assign_irq_vector (int irq); +extern int ioapic_kexec_restore (void ); +extern int ioapic_kdump_restore (void );void enable_NMI_through_LVT0 (void * dummy);
arch/i386/kernel/apic.c +90 -77
236 lines not on this page · raw message
include/asm-i386/apic.h +7 -1
@@ -100,7 +100,7 @@ extern void (*wait_timer_tick)(void);extern int get_maxlvt (void );extern void clear_local_APIC (void );extern void connect_bsp_APIC (void ); -extern void disconnect_bsp_APIC (int virt_wire_setup); +extern void disconnect_bsp_APIC (void );extern void disable_local_APIC (void );extern void lapic_shutdown (void );extern int verify_local_APIC (void );
@@ -121,6 +121,9 @@ extern void nmi_watchdog_tick (struct ptextern int APIC_init_uniprocessor (void );extern void disable_APIC_timer (void );extern void enable_APIC_timer (void ); +#ifdef CONFIG_KEXEC +extern int lapic_kexec_restore (void ); +#endif /* CONFIG_KEXEC */ extern void enable_NMI_through_LVT0 (void * dummy);
@@ -134,6 +137,9 @@ extern int disable_timer_pin_1;#else /* !CONFIG_X86_LOCAL_APIC */ static inline void lapic_shutdown (void ) { } +#ifdef CONFIG_KEXEC +static inline int lapic_kexec_restore (void ) {return 0 ; } +#endif /* CONFIG_KEXEC */ #endif /* !CONFIG_X86_LOCAL_APIC */
arch/x86_64/kernel/apic.c +92 -77
236 lines not on this page · raw message
include/asm-x86_64/apic.h +9 -2
@@ -77,7 +77,7 @@ static inline void ack_APIC_irq(void)extern int get_maxlvt (void );extern void clear_local_APIC (void );extern void connect_bsp_APIC (void ); -extern void disconnect_bsp_APIC (int virt_wire_setup); +extern void disconnect_bsp_APIC (void );extern void disable_local_APIC (void );extern int verify_local_APIC (void );extern void cache_APIC_registers (void );
@@ -98,6 +98,9 @@ extern int APIC_init_uniprocessor (void)extern void disable_APIC_timer (void );extern void enable_APIC_timer (void );extern void clustered_apic_check (void ); +#ifdef CONFIG_KEXEC +extern int lapic_kexec_restore (void ); +#endif /* CONFIG_KEXEC */ extern void nmi_watchdog_default (void );extern int setup_nmi_watchdog (char *);
@@ -111,7 +114,11 @@ extern unsigned int nmi_watchdog;extern int disable_timer_pin_1; -#endif /* CONFIG_X86_LOCAL_APIC */ +#else /* !CONFIG_X86_LOCAL_APIC */ +#ifdef CONFIG_KEXEC +static inline int lapic_kexec_restore (void ) {return 0 ; } +#endif /* CONFIG_KEXEC */ +#endif /* !CONFIG_X86_LOCAL_APIC */ extern unsigned boot_cpu_id;
arch/i386/kernel/reboot.c +11 -3
@@ -304,11 +304,19 @@ void machine_shutdown(void)smp_send_stop ();#endif /* CONFIG_SMP */ -lapic_shutdown (); -#ifdef CONFIG_X86_IO_APIC +#ifndef CONFIG_KEXEC disable_IO_APIC (); -#endif +#else /* CONFIG_KEXEC */ +ioapic_kexec_restore ();/* errors ignored */ +#endif /* CONFIG_KEXEC */ +#endif /* CONFIG_X86_IO_APIC */ + +#ifndef CONFIG_KEXEC +lapic_shutdown (); +#else /* CONFIG_KEXEC */ +lapic_kexec_restore ();/* errors ignored */ +#endif /* CONFIG_KEXEC */ }void machine_emergency_restart (void )
arch/x86_64/kernel/reboot.c +11 -3
@@ -100,11 +100,19 @@ void machine_shutdown(void)local_irq_disable (); +#ifndef CONFIG_KEXEC +disable_IO_APIC (); +#else /* CONFIG_KEXEC */ +ioapic_kexec_restore ();/* errors ignored */ +#endif /* CONFIG_KEXEC */ +#ifndef CONFIG_SMP +#ifndef CONFIG_KEXEC disable_local_APIC (); -#endif - -disable_IO_APIC (); +#else /* CONFIG_KEXEC */ +lapic_kexec_restore ();/* errors ignored */ +#endif /* CONFIG_KEXEC */ +#endif /* CONFIG_SMP */ local_irq_enable (); }
Quoted 11 lines · Fernando Luis Vazquez Cao
Hi Eric,
It seems there are still some problems with the initialization of the APICs in the second kernel. As you can see from the serial line captures below, early during the boot process the kernel is flooded with "unexpected IRQ trap at vector ##" errors and fails to boot.
Some basic details about the machine:
- Dual Xeon Hyper-Threading 2.4GHz
- 1GB RAM
Quoted 5 lines
In this particular machine the problem could be solved by reintroducing the APICs shutdown patch. The drawback with this approach is that LAPIC/IOAPIC registers setup for the legacy PIC mode differs a lot between mother boards and the somewhat hackish shutdown code fails to set the APICs properly under certain configurations.
I have just realized that the "move apic init in init_IRQs" patches by Eric were reverted back. Besides, APICs shutdown code made it back into linux-2.6.15-rc2.
This is a desirable stopgap solution that will solve the problem for some machines and configurations.
Anyway, up to now, the "LAPIC/IO-APIC save/restore" solution is the only one that has worked reliably for us (even with badly broken BIOSes).
I will test Eric's patches and report the results.
Regards,
Fernando