Kdump does not work when panic triggered due to MCE
Subsystems (2)
Hi All,
I wanted to test the behaviour of kdump when panic is triggered
due to MCE on x86 and found that kdump is not captured.
While the kdump service is configured and running and non-MCE panics (such as those triggered through to /proc/sysrq-trigger) successfully capture a kdump, any fatal MCE error injected through the mce-inject tool causes a reboot of the machine.
The code has been traced (using early_serial_putc()) to enter the kexec path i.e. panic()->crash_kexec()->machine_kexec()->relocate_kernel() but is untraceable further.
Kdump works fine when the same the similar test is carried out inside a KVM guest.
Has anybody tested this before? Or have found kdump working when fatal MCEs have actually occurred?
Thanks,
K.Prasad
Relevant Screen logs
---------------------
login: root
Password:
Last login: Fri May 6 11:16:52 from 9.77.122.190
# uname -a Linux elm3a97.beaverton.ibm.com 2.6.39-rc6.prasad_kdump+ #1 SMP Fri May 6 07:47:31 EDT 2011 i686 i686 i386 GNU/Linux # lsmod | grep mce mce_inject 2355 0 [permanent] # service kdump status Kdump is operational # mce-inject /home/prasadkr/mce/mce-test/cases/soft-inj/panic_ucr/data/srar_over Triggering MCE exception on CPU 0 Disabling lock debugging due to kernel taint [Hardware Error]: CPU 0: Machine Check Exception: 6 Bank 2: f580000000000000 [Hardware Error]: RIP 73:<000000001eadbabe> [Hardware Error]: TSC 21dde8717030 ADDR 1234 [Hardware Error]: PROCESSOR 0:106a5 TIME 1304696989 SOCKET 0 APIC 0 [Hardware Error]: No human readable MCE decoding support on this CPU type. [Hardware Error]: Run the message through 'mcelog --ascii' to decode. [Hardware Error]: Machine check: Overflowed uncorrected Kernel panic - not syncing: Fatal Machine check Pid: 0, comm: kworker/0:0 Tainted: G M W 2.6.39-rc6.prasad_kdump+ #1 ------------[ cut here ]------------ kernel BUG at arch/x86/kernel/traps.c:436! invalid opcode: 0000 [#1] SMP last sysfs file: /sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map Modules linked in: mce_inject autofs4 cpufreq_ondemand acpi_cpufreq mperf ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log dm_mod cdc_ether usbnet mii microcode sg i2c_i801 serio_raw pcspkr iTCO_wdt iTCO_vendor_support bnx2x libcrc32c mdio ioatdma dca i7core_edac edac_core bnx2 ext4 jbd2 sd_mod crc_t10dif pata_acpi ata_generic ata_piix mptsas mptscsih mptbase scsi_transport_sas [last unloaded: scsi_wait_scan]
Pid: 0, comm: kworker/0:1 Tainted: G M W 2.6.39-rc6.prasad_kdump+ #1 IBM IBM System x -[7839AC1]-/46C7890
EIP: 0060:[<c0860da9>] EFLAGS: 00010006 CPU: 12
EIP is at do_nmi+0x89/0xa0
EAX: e9ba9c9c EBX: e9ba9c9c ECX: 04010000 EDX: e9ba8000
ESI: 0000000c EDI: 00000af0 EBP: e9ba9c94 ESP: e9ba9c90
DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Process kworker/0:1 (pid: 0, ti=e9ba8000 task=e9ba70d0 task.ti=e9ba8000)
Stack: 0000000c e9ba9ce8 c0860678 0000000c 00000010 000021dd 0000000c 00000af0 e9ba9ce8 f0eca855 c0aa007b 0000007b e9ba00d8 c08600e0 f0eca855 c06048fd 00000060 00000246 f0eca5dd 004b9910 0000000c 0000000c e9ba9cf0 c06048be Call Trace: [<c0860678>] nmi_stack_correct+0x2f/0x34 [<c08600e0>] ? invalidate_interrupt23+0x3c/0x3c [<c06048fd>] ? delay_tsc+0x3d/0x70 [<c06048be>] __const_udelay+0x1e/0x20 [<c041c9b5>] wait_for_panic+0x25/0x50 [<c041cd78>] mce_timed_out+0x48/0x90 [<c041cf39>] mce_end+0x59/0x100 [<c041db2b>] do_machine_check+0x3db/0x6a0 [<c0476600>] ? __hrtimer_start_range_ns+0xa0/0x470 [<f7e15064>] raise_exception+0x34/0xa0 [mce_inject] [<c040f978>] ? sched_clock+0x8/0x10 [<c0478735>] ? sched_clock_cpu+0x145/0x190 [<c0489cc0>] ? __lock_acquire+0x2c0/0x490 [<f7e151a1>] mce_raise_notify+0x61/0x70 [mce_inject] [<c0863393>] notifier_call_chain+0x43/0x60 [<c086340b>] __atomic_notifier_call_chain+0x5b/0x80 [<c08633b0>] ? notifier_call_chain+0x60/0x60 [<c086344a>] atomic_notifier_call_chain+0x1a/0x20 [<c086347d>] notify_die+0x2d/0x30 [<c0860ac2>] default_do_nmi+0x32/0x290 [<c048a312>] ? __lock_release+0x72/0x180 [<c04809ea>] ? clockevents_notify+0x3a/0xf0 [<c0860da7>] do_nmi+0x87/0xa0 [<c0860678>] nmi_stack_correct+0x2f/0x34 [<c048007b>] ? leaps_between+0x3b/0x90 [<c065521c>] ? intel_idle+0x8c/0x100 [<c077a32d>] cpuidle_idle_call+0x8d/0x210 [<c0409b0b>] cpu_idle+0x9b/0xd0 [<c0858714>] start_secondary+0xdd/0xe3 Code: 5e 2f c2 ff 89 e0 25 00 e0 ff ff 8b 50 14 f7 c2 00 00 00 04 74 1e 81 ea 00 00 01 04 89 50 14 5b 5d c3 89 d8 e8 e9 fc ff ff eb c2 <0f> 0b 90 8d 74 26 00 eb f9 0f 0b eb fe 8d 76 00 8d bc 27 00 00 EIP: [<c0860da9>] do_nmi+0x89/0xa0 SS:ESP 0068:e9ba9c90 Call Trace: [<c085c638>] panic+0x57/0x165 [<c041cd10>] mce_panic+0x1c0/0x1e0 [<c041ced0>] mce_reign+0x110/0x120 [<c041cfca>] mce_end+0xea/0x100 [<c041db2b>] do_machine_check+0x3db/0x6a0 [<c0476600>] ? __hrtimer_start_range_ns+0xa0/0x470 [<f7e15064>] raise_exception+0x34/0xa0 [mce_inject] [<c040f978>] ? sched_clock+0x8/0x10 [<c0478735>] ? sched_clock_cpu+0x145/0x190 [<c0489cc0>] ? __lock_acquire+0x2c0/0x490 [<f7e151a1>] mce_raise_notify+0x61/0x70 [mce_inject] [<c0863393>] nx30 [<c0860ac2>] default_do_nmi+0x32/0x290 [<c048a312>] ? __lock_release+0x72/0x180 [<c04809ea>] ? clockevents_notify+0x3a/0xf0 [<c0860da7>] do_nmi+0x87/0xa0 [<c0860678>] nmi_stack_correct+0x2f/0x34 [<c048007b>] ? leaps_between+0x3b/0x90 [<c065521c>] ? intel_idle+0x8c/0x100 [<c077a32d>] cpuidle_idle_call+0x8d/0x210 [<c0409b0b>] cpu_idle+0x9b/0xd0 [<c0858714>] start_secondary+0xdd/0xe3
Rebooting in 1 seconds..
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 2 lines
Has anybody tested this before? Or have found kdump working when fatal MCEs have actually occurred?
Ying did some testing. mce-test has test cases for kdump.
My guess is you injected the error into some area used by the kexec code or boot up path of the kexec kernel.
-Andi
Quoted 5 lines · Andi Kleen
Quoted 2 lines
Has anybody tested this before? Or have found kdump working when fatal MCEs have actually occurred?
Ying did some testing. mce-test has test cases for kdump.
We'd be glad to hear about any successful testcases with recent kernels. My manual testing was quite similar to what the LTP kdump testcase would do i.e. configure kdump service, trigger crash through /proc/sysrq-trigger and watchout for kdump....but as you could see in the logs, that did not happen.
Quoted 4 lines
My guess is you injected the error into some area used by the kexec code or boot up path of the kexec kernel.
-Andi
The logs did not suggest that the second kernel was booted into. The "Rebooting in ... seconds" message appeared from the first kernel. I tried the kdump testcase in atleast two dissimilar machines but with the same results, so it is not clear if the kexec code was affected by the MCE injection in both the cases.
Thanks,
K.Prasad
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hi, Prasad,
Quoted 16 lines · K.Prasad
Quoted 5 lines · Andi Kleen
Quoted 2 lines
Has anybody tested this before? Or have found kdump working when fatal MCEs have actually occurred?
Ying did some testing. mce-test has test cases for kdump.
We'd be glad to hear about any successful testcases with recent kernels. My manual testing was quite similar to what the LTP kdump testcase would do i.e. configure kdump service, trigger crash through
/proc/sysrq-triggerand watchout for kdump....but as you could see in the logs, that did not happen.Quoted 3 lines
My guess is you injected the error into some area used by the kexec code or boot up path of the kexec kernel.
Quoted 7 lines
Quoted 1 line
-Andi
The logs did not suggest that the second kernel was booted into. The "Rebooting in ... seconds" message appeared from the first kernel. I tried the kdump testcase in atleast two dissimilar machines but with the same results, so it is not clear if the kexec code was affected by the MCE injection in both the cases.
From your panic logs, it seems that panic is triggered for MCE on one CPU, when crash_kexec is executing, another panic is triggered on another CPU for timeout mechanism in MCE. We have seen something like that in mce-test developing. Please try following command line for mce injecting.
mce-inject --no-random /home/prasadkr/mce/mce-test/cases/soft-inj/panic_ucr/data/srar_over
Which is used by kdump test driver of mce-test too.
Best Regards,
Huang Ying
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 13 lines · K.Prasad
Hi All,
I wanted to test the behaviour of kdump when panic is triggered
due to MCE on x86 and found that kdump is not captured.While the kdump service is configured and running and non-MCE panics (such as those triggered through to
/proc/sysrq-trigger) successfully capture a kdump, any fatal MCE error injected through the mce-inject tool causes a reboot of the machine.The code has been traced (using
early_serial_putc()) to enter the kexec path i.e. panic()->crash_kexec()->machine_kexec()->relocate_kernel() but is untraceable further.
Quoted 5 lines
Kdump works fine when the same the similar test is carried out inside a KVM guest.
Has anybody tested this before? Or have found kdump working when fatal MCEs have actually occurred?
Prasad,
I have never tried taking dump in MCE situation. Does kdump work on this machine with normal panic()?
Use --debug and --serial option in kexec-tools to print some debug message and look for "I am in purgatory". This will tell you whether you hanged in first kernel or second kernel.
Then put "outb()" messages in the kernel to trace what happened.
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hello,
Quoted 12 lines · Vivek Goyal
Prasad,
I have never tried taking dump in MCE situation. Does kdump work on this machine with normal
panic()?Use --debug and --serial option in kexec-tools to print some debug message and look for "I am in purgatory". This will tell you whether you hanged in first kernel or second kernel.
Then put "
outb()" messages in the kernel to trace what happened.
Quoted 7 lines
Thanks
Vivek_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
I have seen numerous occurrences of MCE triggered kernel panics on both RHEL & SLES environment used on IA32 architecture. Both in contexts where kexec/kdump was being used.
Matter of fact, MCE triggered panic are part of the reason that pushed
me to work on crashdc : only one crash command is required to get the MCE trace out of the kernel ring buffer. This avoids transfering massive amount of vmcore file over the net.
crashdc does well on those, mcelog can be applied on the data gathered.
HTH,
...Louis
Signature · 10 lines
Louis Bouchard Server Support Analyst Canonical Ltd Ubuntu support: http://landscape.canonical.com _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Quoted 26 lines · Bouchard Louis
Hello,
Quoted 12 lines · Vivek Goyal
Prasad,
I have never tried taking dump in MCE situation. Does kdump work on this machine with normal
panic()?Use --debug and --serial option in kexec-tools to print some debug message and look for "I am in purgatory". This will tell you whether you hanged in first kernel or second kernel.
Then put "
outb()" messages in the kernel to trace what happened.Quoted 7 lines
Thanks
Vivek_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexecI have seen numerous occurrences of MCE triggered kernel panics on both RHEL & SLES environment used on IA32 architecture. Both in contexts where kexec/kdump was being used.
Quoted 7 lines
Matter of fact, MCE triggered panic are part of the reason that pushed
me to work on crashdc : only one crash command is required to get the MCE trace out of the kernel ring buffer. This avoids transfering massive amount of vmcore file over the net.crashdc does well on those, mcelog can be applied on the data gathered.
Louis, is this "crashdc" an independent executable. Where is it packaged? What is the significance of name "crashdc" (Crash data collecor ?) Will it make sense to merge it with makedumpfile which does kernel filtering already and this could be one of the additional features.
Thanks
Vivek
Quoted 9 lines
HTH,
...Louis
Signature · 4 lines
Louis Bouchard Server Support Analyst Canonical Ltd Ubuntu support: http://landscape.canonical.com
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hello,
Quoted 7 lines · Vivek Goyal
Louis, is this "crashdc" an independent executable. Where is it packaged? What is the significance of name "crashdc" (Crash data collecor ?) Will it make sense to merge it with makedumpfile which does kernel filtering already and this could be one of the additional features.
Thanks
Vivek
crashdc is hosted no SF. You can have a look at it here :
http://crashdc.svn.sourceforge.net/viewvc/crashdc/trunk/and
Most of the work happens in /usr/bin/crashdc and the run-crashdc-{distro}.sh scripts.
It does indeed means crash data collector. It basically is a set of shell script that send a set of predefined commands to the crash utility and gather the result in a text file that can be easily sent over email.
In one implementation, it makes use of the KDUMP_POST mechanism to run. Another way to use it is through an init script upon reboot.
As for merging it with makedumpfile, it could be an interesting choice. But that might add requirements to makedumpfile that you might not want to have (dependencies on crash & the kernel-debuginfo packages which are needed).
I have recently changed jobs, so I had to suspend my work on crashdc a bit. I'm hoping to find time to work on it and adapt it to debian based distro in the near future.
Kind regards,
...Louis
Signature · 10 lines
Louis Bouchard Server Support Analyst Canonical Ltd Ubuntu support: http://landscape.canonical.com _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Quoted 18 lines · Bouchard Louis
Hello,
Quoted 15 lines · Vivek Goyal
Prasad,
I have never tried taking dump in MCE situation. Does kdump work on this machine with normal
panic()?Use --debug and --serial option in kexec-tools to print some debug message and look for "I am in purgatory". This will tell you whether you hanged in first kernel or second kernel.
Then put "
outb()" messages in the kernel to trace what happened.Thanks
Vivek
Quoted 8 lines
Quoted 4 lines
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexecI have seen numerous occurrences of MCE triggered kernel panics on both RHEL & SLES environment used on IA32 architecture. Both in contexts where kexec/kdump was being used.
That's interesting! Assuming that these are not software induced MCEs but panic() calls invoked due to unrecoverable memory errors in a physical machine, did you experience any situation where the kdump kernel hung/rebooted due to a second MCE (triggered while reading the faulty memory location belonging to the first kernel)?
Quoted 5 lines
Matter of fact, MCE triggered panic are part of the reason that pushed
me to work on crashdc : only one crash command is required to get the MCE trace out of the kernel ring buffer. This avoids transfering massive amount of vmcore file over the net.
What is the data that is contained in the faulty memory location (whose I/O triggered an MCE in the first place)? Basically we'd like to understand what a 'read' operation on the corrupted memory location would result in.
Quoted 2 lines
crashdc does well on those, mcelog can be applied on the data gathered.
We're contemplating a solution on the similar lines (refer the description of 'slim' kdump at https://lkml.org/lkml/2011/5/4/396) to create a 'crash tool readable coredump containing a message that indicates the cause of the crash as MCE (and not any data from the old memory).
I'll take a look at the crashdc code and see if there are ideas that we can borrow from there.
Thanks,
K.Prasad
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hello,
Quoted 10 lines · K.Prasad
Quoted 4 lines · Bouchard Louis
I have seen numerous occurrences of MCE triggered kernel panics on both RHEL & SLES environment used on IA32 architecture. Both in contexts where kexec/kdump was being used.
That's interesting! Assuming that these are not software induced MCEs but
panic()calls invoked due to unrecoverable memory errors in a physical machine, did you experience any situation where the kdump kernel hung/rebooted due to a second MCE (triggered while reading the faulty memory location belonging to the first kernel)?
What I'm relating is issues I have seen in the past, while I was employed somewhere else. Unfortunately, I no longer have access to any of those dumps.
Kind regards,
...Louis
Signature · 10 lines
Louis Bouchard Server Support Analyst Canonical Ltd Ubuntu support: http://landscape.canonical.com _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Quoted 5 lines · K.Prasad
That's interesting! Assuming that these are not software induced MCEs but
panic()calls invoked due to unrecoverable memory errors in a physical machine, did you experience any situation where the kdump kernel hung/rebooted due to a second MCE (triggered while reading the faulty memory location belonging to the first kernel)?
Someone discussed this with me inside RH, is disabling MCE checking in the second kernel an acceptable solution?
Quoted 6 lines
We're contemplating a solution on the similar lines (refer the description of 'slim' kdump at https://lkml.org/lkml/2011/5/4/396) to create a 'crash tool readable coredump containing a message that indicates the cause of the crash as MCE (and not any data from the old memory).
You might want to try use other way instead of memory to store the log of the first kernel, for example, mtdoops.
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 17 lines · Vivek Goyal
Quoted 16 lines · K.Prasad
Hi All,
I wanted to test the behaviour of kdump when panic is triggered
due to MCE on x86 and found that kdump is not captured.While the kdump service is configured and running and non-MCE panics (such as those triggered through to
/proc/sysrq-trigger) successfully capture a kdump, any fatal MCE error injected through the mce-inject tool causes a reboot of the machine.The code has been traced (using
early_serial_putc()) to enter the kexec path i.e. panic()->crash_kexec()->machine_kexec()->relocate_kernel() but is untraceable further.Kdump works fine when the same the similar test is carried out inside a KVM guest.
Quoted 8 lines
Quoted 2 lines
Has anybody tested this before? Or have found kdump working when fatal MCEs have actually occurred?
Prasad,
I have never tried taking dump in MCE situation. Does kdump work on this machine with normal
panic()?
Hi Vivek,
kdump worked fine on this machine for non-MCE triggered panic
calls (the /proc/sysrq-trigger initiated crashes got the kdump fine).
Quoted 4 lines
Use --debug and --serial option in kexec-tools to print some debug message and look for "I am in purgatory". This will tell you whether you hanged in first kernel or second kernel.
There were no boot logs from the second kernel while the "Rebooting in X seconds..." message had appeared before the system rebooted, suggesting that the second kernel did not boot at all.
Quoted 2 lines
Then put "
outb()" messages in the kernel to trace what happened.
The outb logs showed that the system entered machine_kexec function (traceable upto relocate_kernel) but then rebooted from inside the panic() function.
Thanks,
K.Prasad
Quoted 40 lines · K.Prasad
Quoted 17 lines · Vivek Goyal
Quoted 16 lines · K.Prasad
Hi All,
I wanted to test the behaviour of kdump when panic is triggered
due to MCE on x86 and found that kdump is not captured.While the kdump service is configured and running and non-MCE panics (such as those triggered through to
/proc/sysrq-trigger) successfully capture a kdump, any fatal MCE error injected through the mce-inject tool causes a reboot of the machine.The code has been traced (using
early_serial_putc()) to enter the kexec path i.e. panic()->crash_kexec()->machine_kexec()->relocate_kernel() but is untraceable further.Kdump works fine when the same the similar test is carried out inside a KVM guest.
Quoted 8 lines
Quoted 2 lines
Has anybody tested this before? Or have found kdump working when fatal MCEs have actually occurred?
Prasad,
I have never tried taking dump in MCE situation. Does kdump work on this machine with normal
panic()?Hi Vivek,
kdump worked fine on this machine for non-MCE triggered panic
calls (the/proc/sysrq-triggerinitiated crashes got the kdump fine).Quoted 4 lines
Use --debug and --serial option in kexec-tools to print some debug message and look for "I am in purgatory". This will tell you whether you hanged in first kernel or second kernel.
There were no boot logs from the second kernel while the "Rebooting in X seconds..." message had appeared before the system rebooted, suggesting that the second kernel did not boot at all.
Quoted 5 lines
Quoted 2 lines
Then put "
outb()" messages in the kernel to trace what happened.The outb logs showed that the system entered
machine_kexecfunction (traceable uptorelocate_kernel) but then rebooted from inside thepanic()function.
Ok, that means that we returned from crash_kexec() function instead of transitioning into second kernel. This is strange. machine_kexec() is not supposed to return until and unless it finds that there is no crash kernel loaded. As per your mail you can trace it to relocate_kernel() being entered. So only thing I can suggest is debug relocate_kernel() code now to see why it is returning.
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec