Slimdump framework using NT_NOCOREDUMP elf-note

There are certain types of crashes induced by faulty hardware in which capturing crashing kernel's memory (through kdump) makes no sense (or sometimes dangerous).

A case in point, is unrecoverable memory errors (resulting in fatal machine check exceptions) in which reading from the faulty memory location from the kexec'ed kernel will cause double fault and system reset (leaving no information for the user).

Quoted 4 lines · K.Prasad

There are certain types of crashes induced by faulty hardware in which capturing crashing kernel's memory (through kdump) makes no sense (or sometimes dangerous).

Quoted 4 lines

A case in point, is unrecoverable memory errors (resulting in fatal machine check exceptions) in which reading from the faulty memory location from the kexec'ed kernel will cause double fault and system reset (leaving no information for the user).

It does make plenty of sense, and I capture the all of the time. It totally doesn't make sense to do this in the kernel when we can filter this from userspace just fine.

Nacked-by "Eric W. Biederman" <[email protected]>

I thought we already had this discussion. Why is this silliness coming back?

I especially dislike the notion of hardcoding policy in the kernel like this.

It's interesting...according to Intel's Software Developer Manual (quoting from Volume 3A, Chapter 15), the MCIP bit in IA32_MCG_STATUS MSR behaves as described below.

"MCIP (machine check in progress) flag, bit 2 Indicates (when set) that a machine-check exception was generated. Software can set or clear this flag. The occurrence of a second Machine-Check Event while MCIP is set will cause the processor to enter a shutdown state."

While in do_machine_check function, we enter the panic path (for unrecoverable errors) much before the IA32_MCG_STATUS MSR is reset and this is likely to dangerous.

911 void do_machine_check(struct pt_regs *regs, long error_code) 912 {
.............
................
1055 if (no_way_out && tolerant < 3)
1056 mce_panic("Fatal machine check on current CPU", final, msg);
.............
................
1073 mce_wrmsrl(MSR_IA32_MCG_STATUS, 0);
1074 out:

It'd be interesting to know the type of memory error (as classified by the processor) for which you're able to capture the memory dump. Maybe a dump of the various MCE status registers (and struct mce) would help us understand the behaviour on your system better.

Quoted 24 lines · K.Prasad

It's interesting...according to Intel's Software Developer Manual (quoting from Volume 3A, Chapter 15), the MCIP bit in IA32_MCG_STATUS MSR behaves as described below.

"MCIP (machine check in progress) flag, bit 2 Indicates (when set) that a machine-check exception was generated. Software can set or clear this flag. The occurrence of a second Machine-Check Event while MCIP is set will cause the processor to enter a shutdown state."

While in do_machine_check function, we enter the panic path (for unrecoverable errors) much before the IA32_MCG_STATUS MSR is reset and this is likely to dangerous.

911 void do_machine_check(struct pt_regs *regs, long error_code) 912 {
.............
................
1055 if (no_way_out && tolerant < 3)
1056 mce_panic("Fatal machine check on current CPU", final, msg);
.............
................
1073 mce_wrmsrl(MSR_IA32_MCG_STATUS, 0);
1074 out:

Quoted 4 lines

It'd be interesting to know the type of memory error (as classified by the processor) for which you're able to capture the memory dump. Maybe a dump of the various MCE status registers (and struct mce) would help us understand the behaviour on your system better.

Well, there are MCE types for which we need to panic but we don't necessarily corrupt memory. Your approach is to unconditionally avoid dumping core whenever we panic while you should look at the MCE signature and decide then whether to capture crashed kernel memory or not.

For example, if the MCE signature says UC DRAM error, then you can be pretty sure that there is a landmine somewhere in the DRAM region mapping the crashed kernel. If it is, say, a UC when doing data fills from L2 to L1, that doesn't necessarily mean that DRAM is corrupted. But even in the first case, you can evaluate the MCi_ADDR reported with the UC DRAM error and simply skip that particular cacheline when dumping the core instead of not capturing anything at all.

Btw, the doublefault example you give above - is this something you experience on real hardware or just a theoretical thing?

Quoted 6 lines

Well, there are MCE types for which we need to panic but we don't necessarily corrupt memory. Your approach is to unconditionally avoid dumping core whenever we panic while you should look at the MCE signature and decide then whether to capture crashed kernel memory or not.

Quoted 8 lines

For example, if the MCE signature says UC DRAM error, then you can be pretty sure that there is a landmine somewhere in the DRAM region mapping the crashed kernel. If it is, say, a UC when doing data fills from L2 to L1, that doesn't necessarily mean that DRAM is corrupted. But even in the first case, you can evaluate the MCi_ADDR reported with the UC DRAM error and simply skip that particular cacheline when dumping the core instead of not capturing anything at all.

True. Like stated by me earlier, there could be two possible outcomes from capturing memory dump in such cases - they're either dangerous or doesn't make sense.

It is best to avoid a normal kdump in both cases, although the elf-note doesn't distinguish between the two.

NT_NOCOREDUMP, in my opinion, is just the first step towards introducing a framework where different code paths that lead to panic() can 'opt-out' from kdump by adding an elf-note.

We can modify this to add more fine-grained messages using different elf-note types (or use the elf-note name under the NT_NOCOREDUMP type) to indicate the cause/type of crash.

I'd like to hear further from you and the rest of the community to see if there's a need felt for such a change.

Quoted 3 lines

Btw, the doublefault example you give above - is this something you experience on real hardware or just a theoretical thing?

Unfortunately, I still haven't been able to try injecting memory errors and study the behaviour (trying to get access to machine with appropriate firmware). I'll have a reply to this after some experiments with memory error injection.

Quoted 15 lines · K.Prasad

Quoted 14 lines

Well, there are MCE types for which we need to panic but we don't necessarily corrupt memory. Your approach is to unconditionally avoid dumping core whenever we panic while you should look at the MCE signature and decide then whether to capture crashed kernel memory or not.

For example, if the MCE signature says UC DRAM error, then you can be pretty sure that there is a landmine somewhere in the DRAM region mapping the crashed kernel. If it is, say, a UC when doing data fills from L2 to L1, that doesn't necessarily mean that DRAM is corrupted. But even in the first case, you can evaluate the MCi_ADDR reported with the UC DRAM error and simply skip that particular cacheline when dumping the core instead of not capturing anything at all.

Quoted 3 lines

True. Like stated by me earlier, there could be two possible outcomes from capturing memory dump in such cases - they're either dangerous or doesn't make sense.

Why, in the second example the only corruption is to the L2 cache so your memory image is intact. Why wouldn't you want to capture a memory dump then? It is business as usual in that case.

Quoted 11 lines

It is best to avoid a normal kdump in both cases, although the elf-note doesn't distinguish between the two.

NT_NOCOREDUMP, in my opinion, is just the first step towards introducing a framework where different code paths that lead to panic() can 'opt-out' from kdump by adding an elf-note.

We can modify this to add more fine-grained messages using different elf-note types (or use the elf-note name under the NT_NOCOREDUMP type) to indicate the cause/type of crash.

Quoted 2 lines

I'd like to hear further from you and the rest of the community to see if there's a need felt for such a change.

I'd make this conditional on whether you have had memory corruption or not by evaluating MCE signatures and acting accordingly.

We don't want to capture memory dump when the machine crashes due to faulty cache, because the end-user derives no benefit by receiving a bulky vmcore and running crash analysis tools over them. Instead a 'slimdump' that contains a meaningful message about the origin of crash (and which can be understood by his analysis tools) would be better, or so I thought.

There are possibly several hardware errors which cause system crash and the kdump would capture full vmcore, although it doesn't make sense (I wouldn't have cared about the second example, you cited, if they did not generate MCE, but a different exception). In an ideal situation, each of these error paths would 'subscribe' to slimdump and add a meaningful message in the NT_NOCOREDUMP note instead of letting the user-space copy the old kernel memory.

Fine with me. I see that the various IA32_MCi_Status registers will hold information about the error and use that to classify MCEs.

I think the best way to go about is to retain NT_NOCOREDUMP for non-DRAM errors also, but use the note-name field in the elf-note and distinguish the various types of errors...say, by using names such as "PANIC_MCE_DRAM", "PANIC_MCE_CACHE", etc (similar to the error codes described in the Intel manual). The upstream tools like 'makedumpfile' and 'crash' will have to be taught to parse the elf-note name and act accordingly.

Fine with me. I see that the various IA32_MCi_Status registers will hold information about the error and use that to classify MCEs.

I think the best way to go about is to retain NT_NOCOREDUMP for non-DRAM errors also, but use the note-name field in the elf-note and distinguish the various types of errors...say, by using names such as "PANIC_MCE_DRAM", "PANIC_MCE_CACHE", etc (similar to the error codes described in the Intel manual). The upstream tools like 'makedumpfile' and 'crash' will have to be taught to parse the elf-note name and act accordingly.

I am assuming that basic MCE error messages are available in kernel log.

Quoted 1 line · Vivek Goyal

I am assuming that basic MCE error messages are available in kernel log.

They're not in the kernel log if it's an MCE that causes the kernel to declare a panic. There's some MCE's that you can retry the operation and continue, and some that you can get away with poisoning a page, killing the process, and rest of the system is OK. But some you really need to roll over and die because you can't guarantee kernel integrity anymore.

And at that point, those messages are never gonna make it to syslogd and onto disk.

Quoted 4 lines · On Wed, Oct 05, 2011 at 12:00:38PM -0400, [email protected] wrote:

Quoted 1 line · Vivek Goyal

I am assuming that basic MCE error messages are available in kernel log.

Quoted 8 lines

They're not in the kernel log if it's an MCE that causes the kernel to declare a panic. There's some MCE's that you can retry the operation and continue, and some that you can get away with poisoning a page, killing the process, and rest of the system is OK. But some you really need to roll over and die because you can't guarantee kernel integrity anymore.

And at that point, those messages are never gonna make it to syslogd and onto disk.

AFAICT, and for the sake of getting the MCE info, arguably one could look for log_buf in the vmcore of the old kernel and try to find the last lines in the kernel log ringbuffer. They should be the MCE error information from the do_machine_check MCE handler.

This all assuming of course we've managed to dump vmcore successfully by sidestepping the landmines :-).

Quoted 1 line · Vivek Goyal

I am assuming that basic MCE error messages are available in kernel log.

They're not in the kernel log if it's an MCE that causes the kernel to declare a panic. There's some MCE's that you can retry the operation and continue, and some that you can get away with poisoning a page, killing the process, and rest of the system is OK. But some you really need to roll over and die because you can't guarantee kernel integrity anymore.

And at that point, those messages are never gonna make it to syslogd and onto disk.

Quoted 3 lines · K.Prasad

True. Like stated by me earlier, there could be two possible outcomes from capturing memory dump in such cases - they're either dangerous or doesn't make sense.

Why, in the second example the only corruption is to the L2 cache so your memory image is intact. Why wouldn't you want to capture a memory dump then? It is business as usual in that case.

Quoted 6 lines · Borislav Petkov

Quoted 3 lines · K.Prasad

True. Like stated by me earlier, there could be two possible outcomes from capturing memory dump in such cases - they're either dangerous or doesn't make sense.

Quoted 3 lines

Why, in the second example the only corruption is to the L2 cache so your memory image is intact. Why wouldn't you want to capture a memory dump then? It is business as usual in that case.

I'll bite. What's the use case for bothering to capture a memory dump when you're looking at an MCE that indicates L2 cache corruption? What additional useful information could you possibly get from the dump?

Quoted 6 lines

Well, there are MCE types for which we need to panic but we don't necessarily corrupt memory. Your approach is to unconditionally avoid dumping core whenever we panic while you should look at the MCE signature and decide then whether to capture crashed kernel memory or not.

Quoted 8 lines

For example, if the MCE signature says UC DRAM error, then you can be pretty sure that there is a landmine somewhere in the DRAM region mapping the crashed kernel. If it is, say, a UC when doing data fills from L2 to L1, that doesn't necessarily mean that DRAM is corrupted. But even in the first case, you can evaluate the MCi_ADDR reported with the UC DRAM error and simply skip that particular cacheline when dumping the core instead of not capturing anything at all.

True. Like stated by me earlier, there could be two possible outcomes from capturing memory dump in such cases - they're either dangerous or doesn't make sense. It is best to avoid a normal kdump in both cases, although the elf-note doesn't distinguish between the two.

Quoted 36 lines · Borislav Petkov

Quoted 24 lines · K.Prasad

It's interesting...according to Intel's Software Developer Manual (quoting from Volume 3A, Chapter 15), the MCIP bit in IA32_MCG_STATUS MSR behaves as described below.

"MCIP (machine check in progress) flag, bit 2 Indicates (when set) that a machine-check exception was generated. Software can set or clear this flag. The occurrence of a second Machine-Check Event while MCIP is set will cause the processor to enter a shutdown state."

While in do_machine_check function, we enter the panic path (for unrecoverable errors) much before the IA32_MCG_STATUS MSR is reset and this is likely to dangerous.

911 void do_machine_check(struct pt_regs *regs, long error_code) 912 {
.............
................
1055 if (no_way_out && tolerant < 3)
1056 mce_panic("Fatal machine check on current CPU", final, msg);
.............
................
1073 mce_wrmsrl(MSR_IA32_MCG_STATUS, 0);
1074 out:

Quoted 4 lines

It'd be interesting to know the type of memory error (as classified by the processor) for which you're able to capture the memory dump. Maybe a dump of the various MCE status registers (and struct mce) would help us understand the behaviour on your system better.

Well, there are MCE types for which we need to panic but we don't necessarily corrupt memory. Your approach is to unconditionally avoid dumping core whenever we panic while you should look at the MCE signature and decide then whether to capture crashed kernel memory or not.

Quoted 8 lines

For example, if the MCE signature says UC DRAM error, then you can be pretty sure that there is a landmine somewhere in the DRAM region mapping the crashed kernel. If it is, say, a UC when doing data fills from L2 to L1, that doesn't necessarily mean that DRAM is corrupted. But even in the first case, you can evaluate the MCi_ADDR reported with the UC DRAM error and simply skip that particular cacheline when dumping the core instead of not capturing anything at all.

True. Like stated by me earlier, there could be two possible outcomes from capturing memory dump in such cases - they're either dangerous or doesn't make sense. It is best to avoid a normal kdump in both cases, although the elf-note doesn't distinguish between the two.

NT_NOCOREDUMP, in my opinion, is just the first step towards introducing a framework where different code paths that lead to panic() can 'opt-out' from kdump by adding an elf-note.

We can modify this to add more fine-grained messages using different elf-note types (or use the elf-note name under the NT_NOCOREDUMP type) to indicate the cause/type of crash.

I'd like to hear further from you and the rest of the community to see if there's a need felt for such a change.

It totally doesn't make sense to do this in the kernel when we can filter this from userspace just fine.

There are certain types of crashes induced by faulty hardware in which capturing crashing kernel's memory (through kdump) makes no sense (or sometimes dangerous).

A case in point, is unrecoverable memory errors (resulting in fatal machine check exceptions) in which reading from the faulty memory location from the kexec'ed kernel will cause double fault and system reset (leaving no information for the user).

Quoted 8 lines · K.Prasad

There are certain types of crashes induced by faulty hardware in which capturing crashing kernel's memory (through kdump) makes no sense (or sometimes dangerous).

A case in point, is unrecoverable memory errors (resulting in fatal machine check exceptions) in which reading from the faulty memory location from the kexec'ed kernel will cause double fault and system reset (leaving no information for the user).

Prasad,

I am just trying to remember what was wrong with Andi's approach of disable MCE while copying the dump?

Quoted 10 lines · Vivek Goyal

Quoted 8 lines · K.Prasad

There are certain types of crashes induced by faulty hardware in which capturing crashing kernel's memory (through kdump) makes no sense (or sometimes dangerous).

A case in point, is unrecoverable memory errors (resulting in fatal machine check exceptions) in which reading from the faulty memory location from the kexec'ed kernel will cause double fault and system reset (leaving no information for the user).

Quoted 5 lines

Prasad,

I am just trying to remember what was wrong with Andi's approach of disable MCE while copying the dump?

Hi Vivek,
The behaviour upon a read operation on an UC memory location is
undefined and so we want to avoid it (previously discussed here:
http://article.gmane.org/gmane.linux.kernel/1146799). When we disable MCE and copy the dump, we will invariably read the faulty memory location.

Quoted 17 lines · K.Prasad

Quoted 10 lines · Vivek Goyal

Quoted 8 lines · K.Prasad

There are certain types of crashes induced by faulty hardware in which capturing crashing kernel's memory (through kdump) makes no sense (or sometimes dangerous).

A case in point, is unrecoverable memory errors (resulting in fatal machine check exceptions) in which reading from the faulty memory location from the kexec'ed kernel will cause double fault and system reset (leaving no information for the user).

Quoted 5 lines

Prasad,

I am just trying to remember what was wrong with Andi's approach of disable MCE while copying the dump?

Quoted 6 lines

Hi Vivek,
The behaviour upon a read operation on an UC memory location is
undefined and so we want to avoid it (previously discussed here:
http://article.gmane.org/gmane.linux.kernel/1146799). When we disable MCE and copy the dump, we will invariably read the faulty memory location.

Right, from the message above:

"- To disable MCE exceptions as done by the patches cited above. However
the result of a read operation on corrupted memory is unknown and the system behaviour is undefined. We're unsure if this is a safe thing to do."

Can you elaborate more on that? Are we talking poisoned memory here or undetected and uncorrectable memory errors?

Quoted 10 lines · Vivek Goyal

Quoted 8 lines · K.Prasad

There are certain types of crashes induced by faulty hardware in which capturing crashing kernel's memory (through kdump) makes no sense (or sometimes dangerous).

A case in point, is unrecoverable memory errors (resulting in fatal machine check exceptions) in which reading from the faulty memory location from the kexec'ed kernel will cause double fault and system reset (leaving no information for the user).

Quoted 5 lines

Prasad,

I am just trying to remember what was wrong with Andi's approach of disable MCE while copying the dump?

Hi Vivek,
The behaviour upon a read operation on an UC memory location is
undefined and so we want to avoid it (previously discussed here:
http://article.gmane.org/gmane.linux.kernel/1146799).

When we disable MCE and copy the dump, we will invariably read the faulty memory location.

Quoted 21 lines · K.Prasad

Quoted 10 lines · Vivek Goyal

Quoted 8 lines · K.Prasad

There are certain types of crashes induced by faulty hardware in which capturing crashing kernel's memory (through kdump) makes no sense (or sometimes dangerous).

A case in point, is unrecoverable memory errors (resulting in fatal machine check exceptions) in which reading from the faulty memory location from the kexec'ed kernel will cause double fault and system reset (leaving no information for the user).

Quoted 5 lines

Prasad,

I am just trying to remember what was wrong with Andi's approach of disable MCE while copying the dump?

Hi Vivek,
The behaviour upon a read operation on an UC memory location is
undefined and so we want to avoid it (previously discussed here:
http://article.gmane.org/gmane.linux.kernel/1146799).

Quoted 2 lines

When we disable MCE and copy the dump, we will invariably read the faulty memory location.

And how that is worse then not capturing a dump at all? Anyway, you said that in case of MCE vmcore is of no use and we don't want to capture it.

The problem, as pointed out by Borislav Petkov in a different mail, is that we might end up capturing a vmcore containing corrupted data when the same is not required for analysing the cause of the crash.

Of course, all this is assuming that reading the faulty memory with MCE disabled is harmless. However, the effect of a read operation in this case is undefined.

Quoted 7 lines · K.Prasad

The problem, as pointed out by Borislav Petkov in a different mail, is that we might end up capturing a vmcore containing corrupted data when the same is not required for analysing the cause of the crash.

Of course, all this is assuming that reading the faulty memory with MCE disabled is harmless. However, the effect of a read operation in this case is undefined.

Frankly, I don't think that it is undefined - you basically should be able to read DRAM albeit with the corrupted data in it. However, you probably best disable the whole DRAM error detection first by clearing a couple of bits in MC4_CTL_MASK (at least on AMD that should work, I dunno how Intel does that).

But, regardless, according to Vivek, the "makedumpfile" tool should be able to jump over poisoned pages and you don't need all the hoopla above at all, right?

So, in any case we may not be able to disable machine-check exceptions (MCEs) only within the context of kexec'ed kernel. Let me know if I've missed something here.

So, in any case we may not be able to disable machine-check exceptions (MCEs) only within the context of kexec'ed kernel. Let me know if I've missed something here.

The MC4_CTL_MASK doesn't appear to be defined in the kernel. Looking at
http://support.amd.com/us/Processor_TechDocs/26094.PDF, Page 196, it states that "This register is typically programmed by BIOS and not by the Kernel software".

So, in any case we may not be able to disable machine-check exceptions (MCEs) only within the context of kexec'ed kernel. Let me know if I've missed something here.

Quoted 4 lines

But, regardless, according to Vivek, the "makedumpfile" tool should be able to jump over poisoned pages and you don't need all the hoopla above at all, right?

In short, the answer is yes. We could add a new string, say "CRASH_REASON=PANIC_MCE" to VMCOREINFO elf-note which can be parsed by 'makedumpfile' and get away without adding the new NT_NOCOREDUMP elf-note. Parsing through the log_buf to lookout for panic string from inside 'makedumpfile' appears to be a clumsy solution though.

i) Scenario1: System crashes because of a fatal MCE

Proposed Solution: Add a new string in the VMCOREINFO elf-note from within the MCE panic path to indicate cause of crash. 'makedumpfile' recognises this string to collect a slimdump instead of the normal dump.

ii) Scenario2: System with PG_hwpoison (or landmine!) pages crashes because of a software bug. In this case, kexec kernel would normally reboot because of reading the PG_poison page. I'll soon get a new version of the patchset implementing this.

Solution: Maintain a linked list of PFNs when the corresponding 'struct page' has been marked PG_hwpoison. We could export/put this list to use in quite a few ways.

Quoted 10 lines

Quoted 4 lines

But, regardless, according to Vivek, the "makedumpfile" tool should be able to jump over poisoned pages and you don't need all the hoopla above at all, right?

In short, the answer is yes. We could add a new string, say "CRASH_REASON=PANIC_MCE" to VMCOREINFO elf-note which can be parsed by 'makedumpfile' and get away without adding the new NT_NOCOREDUMP elf-note. Parsing through the log_buf to lookout for panic string from inside 'makedumpfile' appears to be a clumsy solution though.

Why, 'makedumpfile' reportedly supports some dmesg parsing already - why would you need additional functionality when it can be done with in-house means already. Maybe Vivek should comment on whether this makes sense but I'm basically reiterating what he said.

Quoted 9 lines · Borislav Petkov

Quoted 7 lines · K.Prasad

The problem, as pointed out by Borislav Petkov in a different mail, is that we might end up capturing a vmcore containing corrupted data when the same is not required for analysing the cause of the crash.

Of course, all this is assuming that reading the faulty memory with MCE disabled is harmless. However, the effect of a read operation in this case is undefined.

Quoted 6 lines

Frankly, I don't think that it is undefined - you basically should be able to read DRAM albeit with the corrupted data in it. However, you probably best disable the whole DRAM error detection first by clearing a couple of bits in MC4_CTL_MASK (at least on AMD that should work, I dunno how Intel does that).

The MC4_CTL_MASK doesn't appear to be defined in the kernel. Looking at
http://support.amd.com/us/Processor_TechDocs/26094.PDF, Page 196, it states that "This register is typically programmed by BIOS and not by the Kernel software".

So, in any case we may not be able to disable machine-check exceptions (MCEs) only within the context of kexec'ed kernel. Let me know if I've missed something here.

Quoted 4 lines

But, regardless, according to Vivek, the "makedumpfile" tool should be able to jump over poisoned pages and you don't need all the hoopla above at all, right?

In short, the answer is yes. We could add a new string, say "CRASH_REASON=PANIC_MCE" to VMCOREINFO elf-note which can be parsed by 'makedumpfile' and get away without adding the new NT_NOCOREDUMP elf-note. Parsing through the log_buf to lookout for panic string from inside 'makedumpfile' appears to be a clumsy solution though.

The suggestion to make NT_NOCOREDUMP to contain more fine-granular information can be met by using meaningful strings for VMCOREINFO.

---

In this context, I wish to quickly recollect the issues we've discussed thus far, their proposed solutions and re-evaluate the need for new elf-note.

i) Scenario1: System crashes because of a fatal MCE

Proposed Solution: Add a new string in the VMCOREINFO elf-note from within the MCE panic path to indicate cause of crash. 'makedumpfile' recognises this string to collect a slimdump instead of the normal dump.

ii) Scenario2: System with PG_hwpoison (or landmine!) pages crashes because of a software bug. In this case, kexec kernel would normally reboot because of reading the PG_poison page. I'll soon get a new version of the patchset implementing this.

Solution: Maintain a linked list of PFNs when the corresponding 'struct page' has been marked PG_hwpoison. We could export/put this list to use in quite a few ways.

Quoted 2 lines · K.Prasad

Quoted 1 line · Borislav Petkov

On Fri, Oct 07, 2011 at 09:42:19PM +0530, K.Prasad wrote:

Quoted 6 lines

ii) Scenario2: System with PG_hwpoison (or landmine!) pages crashes because of a software bug. In this case, kexec kernel would normally reboot because of reading the PG_poison page. I'll soon get a new version of the patchset implementing this.

Quoted 3 lines

Solution: Maintain a linked list of PFNs when the corresponding 'struct page' has been marked PG_hwpoison. We could export/put this list to use in quite a few ways.

What's the need of a list and why do we have to export anything. Can't makedumpfile look at the struct page and then just not dump that page if hwpoison flag is set.

Quoted 3 lines · Vivek Goyal

Quoted 2 lines · K.Prasad

Quoted 1 line · Borislav Petkov

On Fri, Oct 07, 2011 at 09:42:19PM +0530, K.Prasad wrote:

[snipped]

Quoted 6 lines

Quoted 6 lines

ii) Scenario2: System with PG_hwpoison (or landmine!) pages crashes because of a software bug. In this case, kexec kernel would normally reboot because of reading the PG_poison page. I'll soon get a new version of the patchset implementing this.

Quoted 8 lines

Quoted 3 lines

Solution: Maintain a linked list of PFNs when the corresponding 'struct page' has been marked PG_hwpoison. We could export/put this list to use in quite a few ways.

What's the need of a list and why do we have to export anything. Can't makedumpfile look at the struct page and then just not dump that page if hwpoison flag is set.

I'll respond to just this part of the comment for now, since I have a few conflicting thoughts crossing my mind regarding the above suggestion and thought I'll put it across to the community to get that clarified.

Using makedumpfile to actually identify and sidestep PG_hwpoison sounds a bit dangerous. Let's for a moment that makedumpfile has this capability, which is implemented as under.

- The list of nodes (pg_data_t) and all struct page's (through node_mem_map) are sent to makedumpfile using VMCOREINFO_SYMBOL().

- makedumpfile would use this information to go to the old kernel's memory, look at pg_data_t and then into each element of node_mem_map to then lookout for PG_hwpoison inside 'struct page'->flags. (Well, this method works for !SPARSEMEM. I'd like to know if I've overlooked any other better method. pfn_to_page() wouldn't work either, as it will give a 'struct page' of a PFN as seen by the kexec'd kernel and not the crashed kernel).

- If PG_hwpoison flag for the corresponding page is clear, then it will allow the copy operation.

- The problem comes when we actually land on a page with PG_hwpoison while carrying out the above 3 steps. For instance, if the page containing the pg_data_t and node_mem_map data structures themselves are marked hw-poisoned.

Quoted 42 lines · K.Prasad

Quoted 3 lines · Vivek Goyal

Quoted 2 lines · K.Prasad

Quoted 1 line · Borislav Petkov

On Fri, Oct 07, 2011 at 09:42:19PM +0530, K.Prasad wrote:

[snipped]

Quoted 6 lines

Quoted 6 lines

ii) Scenario2: System with PG_hwpoison (or landmine!) pages crashes because of a software bug. In this case, kexec kernel would normally reboot because of reading the PG_poison page. I'll soon get a new version of the patchset implementing this.

Quoted 8 lines

Quoted 3 lines

Solution: Maintain a linked list of PFNs when the corresponding 'struct page' has been marked PG_hwpoison. We could export/put this list to use in quite a few ways.

What's the need of a list and why do we have to export anything. Can't makedumpfile look at the struct page and then just not dump that page if hwpoison flag is set.

I'll respond to just this part of the comment for now, since I have a few conflicting thoughts crossing my mind regarding the above suggestion and thought I'll put it across to the community to get that clarified.

Using makedumpfile to actually identify and sidestep PG_hwpoison sounds a bit dangerous. Let's for a moment that makedumpfile has this capability, which is implemented as under.

- The list of nodes (pg_data_t) and all struct page's (through node_mem_map) are sent to makedumpfile using VMCOREINFO_SYMBOL().

- makedumpfile would use this information to go to the old kernel's memory, look at pg_data_t and then into each element of node_mem_map to then lookout for PG_hwpoison inside 'struct page'->flags. (Well, this method works for !SPARSEMEM. I'd like to know if I've overlooked any other better method. pfn_to_page() wouldn't work either, as it will give a 'struct page' of a PFN as seen by the kexec'd kernel and not the crashed kernel).

- If PG_hwpoison flag for the corresponding page is clear, then it will allow the copy operation.

Quoted 4 lines

- The problem comes when we actually land on a page with PG_hwpoison while carrying out the above 3 steps. For instance, if the page containing the pg_data_t and node_mem_map data structures themselves are marked hw-poisoned.

I think it can happen and in that case we don't capture the dump.

is similar to possibility of running into a accessing a poisoned page while you are trying to same the final note which will contain the MCE info or list of poisoned pages.

Even if you export the list successfuly and you find pd_data_t pages are poisoned, what would you do? Not do filtering and save tera bytes of dump.

I think you are just trying to solve every corner case which might not even be required in practice. Kdump is our best effort to capture the dump and there are so many corner cases where it will not work.

So I would suggest that lets us not make the whole thing too complicated now. If the scenario you are describing becomes common enough that it start bothering, we can look into exporting the poisoned pages list.

Frankly, I don't think that it is undefined - you basically should be able to read DRAM albeit with the corrupted data in it. However, you probably best disable the whole DRAM error detection first by clearing a couple of bits in MC4_CTL_MASK (at least on AMD that should work, I dunno how Intel does that).

There are certain types of crashes induced by faulty hardware in which capturing crashing kernel's memory (through kdump) makes no sense (or sometimes dangerous).

A case in point, is unrecoverable memory errors (resulting in fatal machine check exceptions) in which reading from the faulty memory location from the kexec'ed kernel will cause double fault and system reset (leaving no information for the user).

This patch introduces a framework called 'slimdump' enabled through a new elf-note NT_NOCOREDUMP. Any error whose cause cannot be attributed to a software error and cannot be detected by analysing the kernel memory may decide to add this elf-note to the vmcore and indicate the futility of such an exercise. Tools such as 'kexec', 'makedumpfile' and 'crash' are also modified in tandem to recognise this new elf-note and capture 'slimdump'.

The physical address and size of the NT_NOCOREDUMP are made available to the user-space through a "/sys/kernel/nt_nocoredump" sysfs file (just like other kexec related files).

Signed-off-by K.Prasad <[email protected]>

---
 arch/x86/kernel/cpu/mcheck/mce.c |   28 ++++++++++++++++++++++++++++
 include/linux/elf.h              |   18 ++++++++++++++++++
 include/linux/kexec.h            |    1 +
 kernel/kexec.c                   |   11 +++++++++++
 kernel/ksysfs.c                  |   10 ++++++++++
 5 files changed, 68 insertions(+), 0 deletions(-)

arch/x86/kernel/cpu/mcheck/mce.c +0 -0

index 08363b0..483b2fc 100644

Quoted 17 lines · K.Prasad

There are certain types of crashes induced by faulty hardware in which capturing crashing kernel's memory (through kdump) makes no sense (or sometimes dangerous).

A case in point, is unrecoverable memory errors (resulting in fatal machine check exceptions) in which reading from the faulty memory location from the kexec'ed kernel will cause double fault and system reset (leaving no information for the user).

This patch introduces a framework called 'slimdump' enabled through a new elf-note NT_NOCOREDUMP. Any error whose cause cannot be attributed to a software error and cannot be detected by analysing the kernel memory may decide to add this elf-note to the vmcore and indicate the futility of such an exercise. Tools such as 'kexec', 'makedumpfile' and 'crash' are also modified in tandem to recognise this new elf-note and capture 'slimdump'.

Quoted 3 lines

The physical address and size of the NT_NOCOREDUMP are made available to the user-space through a "/sys/kernel/nt_nocoredump" sysfs file (just like other kexec related files).

Even if kernel has to signal to user space the reason for crash, why not add this info to existing vmcoreinfo note. Something like another filed. PANIC_MCE=1.

Secondly, the note name NT_NOCOREDUMP itself sounds binding. Kernel can export the reason of panic and then it is up to user space what do they want to do with it.

So to me,

Quoted 18 lines

Signed-off-by K.Prasad <[email protected]>

---
 arch/x86/kernel/cpu/mcheck/mce.c |   28 ++++++++++++++++++++++++++++
 include/linux/elf.h              |   18 ++++++++++++++++++
 include/linux/kexec.h            |    1 +
 kernel/kexec.c                   |   11 +++++++++++
 kernel/ksysfs.c                  |   10 ++++++++++
 5 files changed, 68 insertions(+), 0 deletions(-)

arch/x86/kernel/cpu/mcheck/mce.c +0 -0

index 08363b0..483b2fc 100644
@@ -238,6 +238,34 @@ static atomic_t mce_paniced;
 static int fake_panic;
 static atomic_t mce_fake_paniced;
 

arch/x86/kernel/cpu/mcheck/mce.c static atomic_t mce_paniced; · line 247

Quoted 7 lines

arch/x86/kernel/cpu/mcheck/mce.c +7 -0

+void arch_add_nocoredump_note(u32 *buf)
+{
+	struct elf_note note;
+	const char note_name[] = "PANIC_MCE";
+	const char desc_msg[] = "Crash induced due to a fatal machine "
+				"check error";
+

Again, note_name and desc_msg seem to be only two exports. Frankly desc string seems pretty obivious and we should be able to ignore it. So just exporting PANIC_MCE=true or something like that in case of MCE.

Quoted 4 lines

Again, note_name and desc_msg seem to be only two exports. Frankly desc string seems pretty obivious and we should be able to ignore it. So just exporting PANIC_MCE=true or something like that in case of MCE.

Yes, adding a new field to the VMCOREINFO note would have been much simpler but there's a second part to the kdump + fatal MCE problem which will need a new elf-note to solve.

On a system containing 'poisoned' pages (generated as a result of detecting UC errors which haven't been 'consumed'), if a software bug results in crashing the machine, the ensuing kdump operation will read from the faulty memory location. This will trigger a new crash within the context of the kexec'ed kernel and we want to avoid this.

The plan is to pass-down the list of poisoned memory pages to the second kernel using an elf-note so that these pages are left untouched during dump capture. I'm working on an implementation of the same and should have patches soon.

Quoted 4 lines

The plan is to pass-down the list of poisoned memory pages to the second kernel using an elf-note so that these pages are left untouched during dump capture. I'm working on an implementation of the same and should have patches soon.

I would say let us first figure out what happens while reading a poisoned page and is this a problem before working on a solution.

Quoted 7 lines

Quoted 4 lines

The plan is to pass-down the list of poisoned memory pages to the second kernel using an elf-note so that these pages are left untouched during dump capture. I'm working on an implementation of the same and should have patches soon.

I would say let us first figure out what happens while reading a poisoned page and is this a problem before working on a solution.

If the page is poisoned because of a real uncorrectable error in memory (reported as SRAO machine check today, or by SRAR real-soon-now). Then accessing the page from the processor while taking a memory dump will result in a machine check.

Note that a large memory system that had been running for a long time may have built up a small stash of these land-mine pages - and we need to worry about them even in the case where the panic is not machine check related (in fact especially in this case ... we are in a case where we actually do want the dump to diagnose the cause of the panic, and we don't want to risk losing the crash dump because we aborted when touching a page that the OS had safely avoided for days/weeks/months).

So passing a list of poisoned pages from the old kernel to the new kernel is a good idea - and is independent of the cause of the crash (except that in the fatal machine check case due to memory error the list is guaranteed to be non-empty).

Passing some crash signature data - so the new kernel/dump-tools can make a choice whether to even try to take a full dump is also interesting (but independent from the bad page list).

Quoted 7 lines

Quoted 4 lines

The plan is to pass-down the list of poisoned memory pages to the second kernel using an elf-note so that these pages are left untouched during dump capture. I'm working on an implementation of the same and should have patches soon.

I would say let us first figure out what happens while reading a poisoned page and is this a problem before working on a solution.

If the page is poisoned because of a real uncorrectable error in memory (reported as SRAO machine check today, or by SRAR real-soon-now). Then accessing the page from the processor while taking a memory dump will result in a machine check.

Note that a large memory system that had been running for a long time may have built up a small stash of these land-mine pages - and we need to worry about them even in the case where the panic is not machine check related (in fact especially in this case ... we are in a case where we actually do want the dump to diagnose the cause of the panic, and we don't want to risk losing the crash dump because we aborted when touching a page that the OS had safely avoided for days/weeks/months).

So passing a list of poisoned pages from the old kernel to the new kernel is a good idea - and is independent of the cause of the crash (except that in the fatal machine check case due to memory error the list is guaranteed to be non-empty).

Quoted 21 lines · Tony

Quoted 7 lines

Quoted 4 lines

The plan is to pass-down the list of poisoned memory pages to the second kernel using an elf-note so that these pages are left untouched during dump capture. I'm working on an implementation of the same and should have patches soon.

I would say let us first figure out what happens while reading a poisoned page and is this a problem before working on a solution.

If the page is poisoned because of a real uncorrectable error in memory (reported as SRAO machine check today, or by SRAR real-soon-now). Then accessing the page from the processor while taking a memory dump will result in a machine check.

Note that a large memory system that had been running for a long time may have built up a small stash of these land-mine pages - and we need to worry about them even in the case where the panic is not machine check related (in fact especially in this case ... we are in a case where we actually do want the dump to diagnose the cause of the panic, and we don't want to risk losing the crash dump because we aborted when touching a page that the OS had safely avoided for days/weeks/months).

Quoted 4 lines

So passing a list of poisoned pages from the old kernel to the new kernel is a good idea - and is independent of the cause of the crash (except that in the fatal machine check case due to memory error the list is guaranteed to be non-empty).

Whre is this poisoned page info stored? In struct page? If yes, then user space can walk through it and make sure not to touch poisoned pages. Anyway user space filtering utility "makedumpfile" walks through struct pages to filter out the pages. It should be able to filter out poisoned pages unconditionally. So there should be no need for kernel to export a list of these pages.

Quoted 27 lines · Vivek Goyal

Quoted 21 lines · Tony

Quoted 7 lines

Quoted 4 lines

The plan is to pass-down the list of poisoned memory pages to the second kernel using an elf-note so that these pages are left untouched during dump capture. I'm working on an implementation of the same and should have patches soon.

I would say let us first figure out what happens while reading a poisoned page and is this a problem before working on a solution.

If the page is poisoned because of a real uncorrectable error in memory (reported as SRAO machine check today, or by SRAR real-soon-now). Then accessing the page from the processor while taking a memory dump will result in a machine check.

Note that a large memory system that had been running for a long time may have built up a small stash of these land-mine pages - and we need to worry about them even in the case where the panic is not machine check related (in fact especially in this case ... we are in a case where we actually do want the dump to diagnose the cause of the panic, and we don't want to risk losing the crash dump because we aborted when touching a page that the OS had safely avoided for days/weeks/months).

Quoted 4 lines

So passing a list of poisoned pages from the old kernel to the new kernel is a good idea - and is independent of the cause of the crash (except that in the fatal machine check case due to memory error the list is guaranteed to be non-empty).

Quoted 6 lines

Whre is this poisoned page info stored? In struct page? If yes, then user space can walk through it and make sure not to touch poisoned pages. Anyway user space filtering utility "makedumpfile" walks through struct pages to filter out the pages. It should be able to filter out poisoned pages unconditionally. So there should be no need for kernel to export a list of these pages.

Does this utility work on a vmcore dump? If so, Tony refers to the creation of the vmcore itself from the memory used by the first kernel.

If there are poisoned pages, merely accessing that portion of DRAM containing the poisoned data would cause further MCEs in the freshly booted kernel so you won't be able to finish creating the dump.

As long as you can get to your struct page arrays, one should be able to filter out poisoned pages without saving the whole dump.

Quoted 52 lines · Vivek Goyal

Quoted 44 lines · Tony

Quoted 35 lines

Quoted 11 lines

Quoted 7 lines

The plan is to pass-down the list of poisoned memory pages to the second
kernel using an elf-note so that these pages are left untouched during
dump capture. I'm working on an implementation of the same and should
have patches soon.

I would say let us first figure out what happens while reading a poisoned
page and is this a problem before working on a solution.

If the page is poisoned because of a real uncorrectable error in memory
(reported as SRAO machine check today, or by SRAR real-soon-now). Then
accessing the page from the processor while taking a memory dump will
result in a machine check.

Note that a large memory system that had been running for a long time
may have built up a small stash of these land-mine pages - and we need
to worry about them even in the case where the panic is not machine
check related (in fact especially in this case ... we are in a case
where we actually do want the dump to diagnose the cause of the panic,
and we don't want to risk losing the crash dump because we aborted when
touching a page that the OS had safely avoided for days/weeks/months).

Quoted 7 lines

So passing a list of poisoned pages from the old kernel to the new kernel
is a good idea - and is independent of the cause of the crash (except that
in the fatal machine check case due to memory error the list is guaranteed
to be non-empty).

Quoted 6 lines

Whre is this poisoned page info stored? In struct page? If yes, then user space can walk through it and make sure not to touch poisoned pages. Anyway user space filtering utility "makedumpfile" walks through struct pages to filter out the pages. It should be able to filter out poisoned pages unconditionally. So there should be no need for kernel to export a list of these pages.

Quoted 3 lines

Does this utility work on a vmcore dump? If so, Tony refers to the creation of the vmcore itself from the memory used by the first kernel.

No, this utitlity can directly work on /proc/vmcore where first kernel's image is still in memory and not on disk.

Quoted 3 lines

If there are poisoned pages, merely accessing that portion of DRAM containing the poisoned data would cause further MCEs in the freshly booted kernel so you won't be able to finish creating the dump.

As long as you can get to your struct page arrays, one should be able to filter out poisoned pages without saving the whole dump.

----- Original Message -----

Quoted 64 lines · Borislav Petkov

Quoted 52 lines · Vivek Goyal

Quoted 44 lines · Tony

Quoted 35 lines

Quoted 11 lines

Quoted 7 lines

The plan is to pass-down the list of poisoned memory pages to the second
kernel using an elf-note so that these pages are left untouched during
dump capture. I'm working on an implementation of the same and should
have patches soon.

I would say let us first figure out what happens while reading a poisoned
page and is this a problem before working on a solution.

If the page is poisoned because of a real uncorrectable error in memory
(reported as SRAO machine check today, or by SRAR real-soon-now). Then
accessing the page from the processor while taking a memory dump will
result in a machine check.

Note that a large memory system that had been running for a long time
may have built up a small stash of these land-mine pages - and we need
to worry about them even in the case where the panic is not machine
check related (in fact especially in this case ... we are in a case
where we actually do want the dump to diagnose the cause of the panic,
and we don't want to risk losing the crash dump because we aborted when
touching a page that the OS had safely avoided for days/weeks/months).

Quoted 7 lines

So passing a list of poisoned pages from the old kernel to the new kernel
is a good idea - and is independent of the cause of the crash (except that
in the fatal machine check case due to memory error the list is guaranteed
to be non-empty).

Quoted 6 lines

Whre is this poisoned page info stored? In struct page? If yes, then user space can walk through it and make sure not to touch poisoned pages. Anyway user space filtering utility "makedumpfile" walks through struct pages to filter out the pages. It should be able to filter out poisoned pages unconditionally. So there should be no need for kernel to export a list of these pages.

Quoted 3 lines

Does this utility work on a vmcore dump? If so, Tony refers to the creation of the vmcore itself from the memory used by the first kernel.

No, this utitlity can directly work on /proc/vmcore where first kernel's image is still in memory and not on disk.

Quoted 3 lines

If there are poisoned pages, merely accessing that portion of DRAM containing the poisoned data would cause further MCEs in the freshly booted kernel so you won't be able to finish creating the dump.

Quoted 2 lines

As long as you can get to your struct page arrays, one should be able to filter out poisoned pages without saving the whole dump.

It's still going to require a minimal kernel change because the PG_hwpoison flag's bit number differs depending upon the kernel configuration, if it exists at all. An additional vmcoreinfo item probably...

arch/x86/kernel/cpu/mcheck/mce.c static atomic_t mce_paniced; · line 244

Quoted 12 lines · K.Prasad

arch/x86/kernel/cpu/mcheck/mce.c +4 -0

index 08363b0..483b2fc 100644
@@ -238,6 +238,34 @@ static atomic_t mce_paniced;
 static int fake_panic;
 static atomic_t mce_fake_paniced;
 
+void arch_add_nocoredump_note(u32 *buf)
+{
+	struct elf_note note;
+	const char note_name[] = "PANIC_MCE";

static const ...

arch/x86/kernel/cpu/mcheck/mce.c static atomic_t mce_paniced; · line 246

Quoted 2 lines

arch/x86/kernel/cpu/mcheck/mce.c +2 -0

+	const char desc_msg[] = "Crash induced due to a fatal machine "
+				"check error";

likewise.

Please don't break string literals onto multiple lines like this, since it prevents users from finding the string using "grep".

Quoted 5 lines · Nick Bowler

arch/x86/kernel/cpu/mcheck/mce.c static atomic_t mce_paniced; · line 244

Quoted 1 line · K.Prasad

arch/x86/kernel/cpu/mcheck/mce.c +1 -0

+	const char note_name[] = "PANIC_MCE";

static const ...

Quoted 4 lines

arch/x86/kernel/cpu/mcheck/mce.c static atomic_t mce_paniced; · line 246

Quoted 2 lines

arch/x86/kernel/cpu/mcheck/mce.c +2 -0

+	const char desc_msg[] = "Crash induced due to a fatal machine "
+				"check error";

likewise.

Not sure why you wanted them to be defined as static. These strings are going to be copied into respective elf-note buffers just a few lines below their definitions.

Quoted 2 lines

Please don't break string literals onto multiple lines like this, since it prevents users from finding the string using "grep".

We could do so, hoping that checkpatch.pl doesn't complain if we crossed the 80-column width :-)