Kdump and memory error handling

Any thoughts/suggestions?

Quoted 1 line

Any thoughts/suggestions?

My old attempts to solve this are

Don't dump on MCE:

http://git.kernel.org/?p=linux/kernel/git/ak/linux-mce-2.6.git;a=shortlog;h=refs/heads/mce/xpanic

Handle dumps of corrupted memory regresions:

http://git.kernel.org/?p=linux/kernel/git/ak/linux-mce-2.6.git;a=shortlog;h=refs/heads/mce/crashdump

Handle dumps of corrupted memory regresions:

http://git.kernel.org/?p=linux/kernel/git/ak/linux-mce-2.6.git;a=shortlog;h=refs/heads/mce/crashdump

Quoted 1 line

Any thoughts/suggestions?

My old attempts to solve this are

Don't dump on MCE:

http://git.kernel.org/?p=linux/kernel/git/ak/linux-mce-2.6.git;a=shortlog;h=refs/heads/mce/xpanic

Handle dumps of corrupted memory regresions:

http://git.kernel.org/?p=linux/kernel/git/ak/linux-mce-2.6.git;a=shortlog;h=refs/heads/mce/crashdump

IMHO these patches are still the right solutions for this.

Quoted 8 lines · Andi Kleen

Quoted 1 line

Any thoughts/suggestions?

My old attempts to solve this are

Don't dump on MCE:

http://git.kernel.org/?p=linux/kernel/git/ak/linux-mce-2.6.git;a=shortlog;h=refs/heads/mce/xpanic

The problem we seen in avoiding a panic->crash_kexec->[coredump capture] is that the user may not have a means to know the reason for crash, unless the serial console is connected to capture and store the panic string.

Alternatively a 'slim' kdump (as described here:
https://lkml.org/lkml/2011/5/4/396) would not contain meaningless data from the old memory, but inform the user about the cause of the crash. I'm intending to post some patches with a quick implementation of it soon.

Quoted 4 lines

Handle dumps of corrupted memory regresions:

http://git.kernel.org/?p=linux/kernel/git/ak/linux-mce-2.6.git;a=shortlog;h=refs/heads/mce/crashdump

Quoted 2 lines

IMHO these patches are still the right solutions for this.

Like Vatsa had raised, the processor's behaviour upon reading (or any I/O operation) the faulty memory location isn't clearly defined (to the extent I read through System Programming Guide Part 1, Volume 3A, Chapter 15). In such a scenario, disabling MCE for the kdump kernel (which can potentially read the faulty memory) is making things hazy.

Hi All,
We've been trying to study and improve the kdump behaviour when
a panic is triggered due to an unrecoverable memory error causing a machine check exception (MCE) followed by a kernel panic.

In this context we foresee a few issues in capturing kdump and would like to receive comments about the ways to handle them.

Probable Issues when capturing coredump through kdump following a memory error
---------------------------
- First, a coredump of the memory from the crashing kernel isn't really helpful in debugging the crash that was caused due to a faulty memory. Collecting the same has some of the problems illustrated below. It should therefore suffice to let the user know the reason of the crash rather than provide a complete dump of the memory.

For this, a 'slim' yet crash-tool readable coredump containing:
- message about the cause (such as crash due to unrecoverable memory error) in the coredump's elf-note section.
- and no data from the memory of the 'crashing' kernel (their elf sections can be reduced to zero length).
may be suitable.

- Alternatively, if the kdump kernel decides to capture the coredump, its attempts to read the faulty memory location may lead to subsequent faults in the context of kdump kernel with fatal consequences. This may either be avoided by:

a) Pass the address of the corrupt memory location to the kdump kernel and skip reading that location while creating the vmcore. This needs an instance of 'struct mce' (from the 'crashing' kernel), which already contains the faulty memory address (in the physical address form, which should be confirmed using the IA32_MCi_MISC[8:6] bits stored in 'misc' field of 'struct mce') to be populated inside the elf (-notes?) section.

b) Use modified copy applications (such as a modified 'cp' command) that can map the /dev/oldmem into user-space and then initiate the creation of vmcore. In this method, the user-space process performing the copy will receive a SIGBUS while consuming the faulty memory (through INT18 -> do_machine_check) but it must be modified to be resilient to the signal, while intelligently skipping to the subsequent memory location for further copying. Meanwhile the data for the faulty memory location can be represented using 'zero-ed' data and the vmcore enhanced to indicate the cause of the crash as one resulting from a fatal MCE.

Any thoughts/suggestions?

Quoted 47 lines · K.Prasad

Hi All,
We've been trying to study and improve the kdump behaviour when
a panic is triggered due to an unrecoverable memory error causing a machine check exception (MCE) followed by a kernel panic.

In this context we foresee a few issues in capturing kdump and would like to receive comments about the ways to handle them.

Probable Issues when capturing coredump through kdump following a memory error
---------------------------
- First, a coredump of the memory from the crashing kernel isn't really helpful in debugging the crash that was caused due to a faulty memory. Collecting the same has some of the problems illustrated below. It should therefore suffice to let the user know the reason of the crash rather than provide a complete dump of the memory.

For this, a 'slim' yet crash-tool readable coredump containing:
- message about the cause (such as crash due to unrecoverable memory error) in the coredump's elf-note section.
- and no data from the memory of the 'crashing' kernel (their elf sections can be reduced to zero length).
may be suitable.

- Alternatively, if the kdump kernel decides to capture the coredump, its attempts to read the faulty memory location may lead to subsequent faults in the context of kdump kernel with fatal consequences. This may either be avoided by:

a) Pass the address of the corrupt memory location to the kdump kernel and skip reading that location while creating the vmcore. This needs an instance of 'struct mce' (from the 'crashing' kernel), which already contains the faulty memory address (in the physical address form, which should be confirmed using the IA32_MCi_MISC[8:6] bits stored in 'misc' field of 'struct mce') to be populated inside the elf (-notes?) section.

b) Use modified copy applications (such as a modified 'cp' command) that can map the /dev/oldmem into user-space and then initiate the creation of vmcore. In this method, the user-space process performing the copy will receive a SIGBUS while consuming the faulty memory (through INT18 -> do_machine_check) but it must be modified to be resilient to the signal, while intelligently skipping to the subsequent memory location for further copying. Meanwhile the data for the faulty memory location can be represented using 'zero-ed' data and the vmcore enhanced to indicate the cause of the crash as one resulting from a fatal MCE.

Quoted 1 line

Any thoughts/suggestions?

In practice this all works for me.

I have received several crash dumps where there was an mce error.

I admit I have my userspace configured to just grab the dmesg from the kernel log and not do a full crash dump. So in that sense I am already a slim crash dump.

But in practice with real hardware errors it is working today without kernel changes.