Query about kdump_msg hook into crash_kexec()

Anyway, if an image is loaded and we have setup to capture dump also why do we need to save kmsg with the help of an helper. I am assuming this is more of a debugging aid if we have no other way to capture the kernel log buffer. So if somebody has setup kdump to capture the vmcore, why to call another handler which tries to save part of the vmcore (kmsg) separately.

Quoted 8 lines · Vivek Goyal

Anyway, if an image is loaded and we have setup to capture dump also why do we need to save kmsg with the help of an helper. I am assuming this is more of a debugging aid if we have no other way to capture the kernel log buffer. So if somebody has setup kdump to capture the vmcore, why to call another handler which tries to save part of the vmcore (kmsg) separately.

Well, kmsg dumper is not only for panic/oops, it can also save kernel messages when the system is reboot/halt.

Yeah, it is true that vmcore contains log_buf, but I think the users of ramoops/mtdoops are mainly those who don't have kdump configured in their kernels, they are cheaper than kdump, but less powerful

Quoted 10 lines · Américo Wang

Quoted 8 lines · Vivek Goyal

Anyway, if an image is loaded and we have setup to capture dump also why do we need to save kmsg with the help of an helper. I am assuming this is more of a debugging aid if we have no other way to capture the kernel log buffer. So if somebody has setup kdump to capture the vmcore, why to call another handler which tries to save part of the vmcore (kmsg) separately.

Quoted 6 lines

Well, kmsg dumper is not only for panic/oops, it can also save kernel messages when the system is reboot/halt.

Yeah, it is true that vmcore contains log_buf, but I think the users of ramoops/mtdoops are mainly those who don't have kdump configured in their kernels, they are cheaper than kdump, but less powerful

The issue is the inane call inside crash_kexec.

It requires both a kexec kernel to be loaded and it requires you to be crashing. Given that when I audited the kmsg_dump handlers they really weren't safe in a crash dump scenario we should just remove it.

Quoted 6 lines · Eric W. Biederman

The issue is the inane call inside crash_kexec.

It requires both a kexec kernel to be loaded and it requires you to be crashing. Given that when I audited the kmsg_dump handlers they really weren't safe in a crash dump scenario we should just remove it.

Probably, I think we need to get rid of KMSG_DUMP_KEXEC.

Quoted 10 lines · Américo Wang

Quoted 6 lines · Eric W. Biederman

The issue is the inane call inside crash_kexec.

It requires both a kexec kernel to be loaded and it requires you to be crashing. Given that when I audited the kmsg_dump handlers they really weren't safe in a crash dump scenario we should just remove it.

Probably, I think we need to get rid of KMSG_DUMP_KEXEC.

Here we go.

--------->

KMSG_DUMP_KEXEC is useless because we already save kernel messages inside /proc/vmcore, and it is unsafe to allow modules to do other stuffs in a crash dump scenario.

Reported-by Vivek Goyal <[email protected]>

Cc "Eric W. Biederman" <[email protected]>

Signed-off-by WANG Cong <[email protected]>

---

drivers/char/ramoops.c +1 -2

index 1a9f5f6..8653ba4 100644
@@ -69,8 +69,7 @@ static void ramoops_do_dump(struct kmsg_dumper *dumper,
 	struct timeval timestamp;
 
 	if (reason != KMSG_DUMP_OOPS &&
-	    reason != KMSG_DUMP_PANIC &&
-	    reason != KMSG_DUMP_KEXEC)
+	    reason != KMSG_DUMP_PANIC)
 		return;
 
 	/* Only dump oopses if dump_oops is set */

drivers/mtd/mtdoops.c +1 -2

index e3e40f4..56eac4e 100644
@@ -308,8 +308,7 @@ static void mtdoops_do_dump(struct kmsg_dumper *dumper,
 	char *dst;
 
 	if (reason != KMSG_DUMP_OOPS &&
-	    reason != KMSG_DUMP_PANIC &&
-	    reason != KMSG_DUMP_KEXEC)
+	    reason != KMSG_DUMP_PANIC)
 		return;
 
 	/* Only dump oopses if dump_oops is set */

include/linux/kmsg_dump.h +0 -1

index 2a0d7d6..05fa2a9 100644
@@ -17,7 +17,6 @@
 enum kmsg_dump_reason {
 	KMSG_DUMP_OOPS,
 	KMSG_DUMP_PANIC,
-	KMSG_DUMP_KEXEC,
 	KMSG_DUMP_RESTART,
 	KMSG_DUMP_HALT,
 	KMSG_DUMP_POWEROFF,

kernel/kexec.c +0 -3

index ec19b92..3ac0218 100644
@@ -32,7 +32,6 @@
 #include <linux/console.h>
 #include <linux/vmalloc.h>
 #include <linux/swap.h>
-#include <linux/kmsg_dump.h>
 
 #include <asm/page.h>
 #include <asm/uaccess.h>

@@ -1078,8 +1077,6 @@ void crash_kexec(struct pt_regs *regs)
 		if (kexec_crash_image) {
 			struct pt_regs fixed_regs;
 
-			kmsg_dump(KMSG_DUMP_KEXEC);
-
 			crash_setup_regs(&fixed_regs, regs);
 			crash_save_vmcoreinfo();
 			machine_crash_shutdown(&fixed_regs);

Quoted 12 lines · Américo Wang

Quoted 10 lines · Américo Wang

Quoted 6 lines · Eric W. Biederman

The issue is the inane call inside crash_kexec.

It requires both a kexec kernel to be loaded and it requires you to be crashing. Given that when I audited the kmsg_dump handlers they really weren't safe in a crash dump scenario we should just remove it.

Probably, I think we need to get rid of KMSG_DUMP_KEXEC.

Quoted 8 lines

Here we go.

--------->

KMSG_DUMP_KEXEC is useless because we already save kernel messages inside /proc/vmcore, and it is unsafe to allow modules to do other stuffs in a crash dump scenario.

I think this is right thing to do. First of all kmsg_dump() call inside crash_kexec() does not make any sense. Secondly, if kdump kernel is loaded, we anyway are going to capture log buffers in vmcore and there should not be any need to try to save messages twice.

Now one can argue that what if kdump does not capture the dump. I think then right solution (though painful one) is to try to make kdump as reliable as possible instead of trying to come up with backup mechanisms to save logs in case kdump fails.

Quoted 10 lines · Eric W. Biederman

Quoted 6 lines

The issue is the inane call inside crash_kexec.

It requires both a kexec kernel to be loaded and it requires you to be crashing. Given that when I audited the kmsg_dump handlers they really weren't safe in a crash dump scenario we should just remove it.

Probably, I think we need to get rid of KMSG_DUMP_KEXEC.

Here we go.

--------->

KMSG_DUMP_KEXEC is useless because we already save kernel messages inside /proc/vmcore, and it is unsafe to allow modules to do other stuffs in a crash dump scenario.

Reported-by Vivek Goyal <[email protected]>

Cc "Eric W. Biederman" <[email protected]>

Signed-off-by WANG Cong <[email protected]>

Quoted 20 lines · Américo Wang

Quoted 10 lines · Eric W. Biederman

Quoted 6 lines

The issue is the inane call inside crash_kexec.

It requires both a kexec kernel to be loaded and it requires you to be crashing. Given that when I audited the kmsg_dump handlers they really weren't safe in a crash dump scenario we should just remove it.

Probably, I think we need to get rid of KMSG_DUMP_KEXEC.

Here we go.

--------->

KMSG_DUMP_KEXEC is useless because we already save kernel messages inside /proc/vmcore, and it is unsafe to allow modules to do other stuffs in a crash dump scenario.

Quoted 3 lines

Reported-by Vivek Goyal <[email protected]>

Cc "Eric W. Biederman" <[email protected]>

Signed-off-by WANG Cong <[email protected]>

I wrote why this is no good idea by another mail. Please see it. Anyway you have a right to don't use this feature.

Nack.

Quoted 3 lines

I wrote why this is no good idea by another mail. Please see it. Anyway you have a right to don't use this feature.

But you have not explained that why do you need to hook into crash_kexec() and you have also not explained why do you need to send out kdump_msg() notification if kdump is configured.

Some detailed explanation here would help.

Quoted 8 lines

Quoted 3 lines

I wrote why this is no good idea by another mail. Please see it. Anyway you have a right to don't use this feature.

But you have not explained that why do you need to hook into crash_kexec() and you have also not explained why do you need to send out kdump_msg() notification if kdump is configured.

Quoted 1 line

Some detailed explanation here would help.

Hi,

I send it you now :)

Quoted 10 lines · On Thu, 3 Feb 2011 13:53:01 +0900 (JST) KOSAKI Motohiro…

Quoted 9 lines

Quoted 3 lines

I wrote why this is no good idea by another mail. Please see it. Anyway you have a right to don't use this feature.

But you have not explained that why do you need to hook into crash_kexec() and you have also not explained why do you need to send out kdump_msg() notification if kdump is configured.

Some detailed explanation here would help.

Quoted 4 lines

Hi,

I send it you now :)

What happened with this? kexec-remove-kmsg_dump_kexec.patch has two acks and one unexplained nack :(

Quoted 13 lines · Andrew Morton

Quoted 10 lines · On Thu,  3 Feb 2011 13:53:01 +0900 (JST) KOSAKI Motohiro…

Quoted 9 lines

Quoted 3 lines

I wrote why this is no good idea by another mail. Please see it. Anyway you have a right to don't use this feature.

But you have not explained that why do you need to hook into crash_kexec() and you have also not explained why do you need to send out kdump_msg() notification if kdump is configured.

Some detailed explanation here would help.

Quoted 7 lines

Quoted 4 lines

Hi,

I send it you now :)

What happened with this?  kexec-remove-kmsg_dump_kexec.patch has two acks and one unexplained nack :(

As I recall the nack was based on a theoretical use case, and a cleanup of kmsg_dump to make it more robust which I don't believe has happened, instead of something real.

My feel is that we should remove kmsg_dump and if a real use case comes up reconsider it at that time.

I don't think anyone cares too strongly at the moment because the features are not expected to be used in conjunction with each other, nor even expected to be compiled into the same kernel.  However given that they are not used to be used in conjunction with each other a call into kmsg_dump from crash_kexec is really just cluttering the code for no benefit to anyone.

Quoted 10 lines · On Thu, 3 Feb 2011 13:53:01 +0900 (JST) KOSAKI Motohiro…

Quoted 9 lines

Quoted 3 lines

I wrote why this is no good idea by another mail. Please see it. Anyway you have a right to don't use this feature.

But you have not explained that why do you need to hook into crash_kexec() and you have also not explained why do you need to send out kdump_msg() notification if kdump is configured.

Some detailed explanation here would help.

Quoted 4 lines

Hi,

I send it you now :)

What happened with this? kexec-remove-kmsg_dump_kexec.patch has two acks and one unexplained nack :(

I'm sorry I've missed this mail long time.

Quoted 11 lines

Quoted 11 lines

---------------------------------------------------------------------

(quoted diff) +1 -0

@@ -74,6 +75,7 @@ NORET_TYPE void panic(const char * fmt, ...) 
        dump_stack(); 
 #endif 
+       kmsg_dump(KMSG_DUMP_PANIC); 
        /* 
         * If we have crashed and we have a crash kernel loaded let it handle 
         * everything else. 
         * Do we want to call this before we try to display a message? 

         */ 
        crash_kexec(NULL); 

Quoted 8 lines

Quoted 1 line

---------------------------------------------------------------------

And I think to compensate for that somebody introduced additional kmsg_dump(KEXEC) call inside crash_kexec() and put it under CONFIG option so that one can change the behavior based on config options. I think this makes the logic somewhat twisted and an unnecessary call inside crash_kexec(). So until and unless there is a strong reason we can get rid of KEXEC event and move kmsg_dump call before crash_kexec() for now and see how does it go, IMHO.

I think I can agree your proposal. But could you please explain why do you think kmsg _before_ kdump and kmsg _in_ kdump are so different? I think it is only C level difference. CPU don't care C function and anyway the kernel call kmsg_dump() because invoke second kernel even if you proposal applied.
It is only curious. I'm not against your proposal.
Thanks.

Quoted 18 lines · Andrew Morton

Quoted 14 lines · On Thu, 3 Feb 2011 13:53:01 +0900 (JST) KOSAKI Motohiro…

Quoted 9 lines

Quoted 3 lines

I wrote why this is no good idea by another mail. Please see it. Anyway you have a right to don't use this feature.

But you have not explained that why do you need to hook into crash_kexec() and you have also not explained why do you need to send out kdump_msg() notification if kdump is configured.

Some detailed explanation here would help.

Hi,

I send it you now :)

Quoted 2 lines

What happened with this? kexec-remove-kmsg_dump_kexec.patch has two acks and one unexplained nack :(

http://groups.google.com/group/linux.kernel/browse_thread/thread/1084f406573d76ac/ee19e34b45f83536?lnk=raot&pli=1

At last mail, Vivek proposed move kms_dump() instead remove. and I asked following question and I've got no response. I'm still waiting his.

Quoted 30 lines

I'm sorry I've missed this mail long time.

Quoted 11 lines

Quoted 11 lines

---------------------------------------------------------------------

(quoted diff) +1 -0

@@ -74,6 +75,7 @@ NORET_TYPE void panic(const char * fmt, ...) 
        dump_stack(); 
 #endif 
+       kmsg_dump(KMSG_DUMP_PANIC); 
        /* 
         * If we have crashed and we have a crash kernel loaded let it handle 
         * everything else. 
         * Do we want to call this before we try to display a message? 

         */ 
        crash_kexec(NULL); 

Quoted 8 lines

Quoted 1 line

---------------------------------------------------------------------

And I think to compensate for that somebody introduced additional kmsg_dump(KEXEC) call inside crash_kexec() and put it under CONFIG option so that one can change the behavior based on config options. I think this makes the logic somewhat twisted and an unnecessary call inside crash_kexec(). So until and unless there is a strong reason we can get rid of KEXEC event and move kmsg_dump call before crash_kexec() for now and see how does it go, IMHO.

I think I can agree your proposal. But could you please explain why do you think kmsg _before_ kdump and kmsg _in_ kdump are so different? I think it is only C level difference. CPU don't care C function and anyway the kernel call kmsg_dump() because invoke second kernel even if you proposal applied.
It is only curious. I'm not against your proposal.
Thanks.

Quoted 7 lines

Quoted 7 lines

I think I can agree your proposal. But could you please explain why do you think kmsg _before_ kdump and kmsg _in_ kdump are so different? I think it is only C level difference. CPU don't care C function and anyway the kernel call kmsg_dump() because invoke second kernel even if you proposal applied.
It is only curious. I'm not against your proposal.
Thanks.

Few reasons.

- There is no correlation between crash_kexec() and kdump_msg(). What you are creating is equivalent of panic notifiers and calling those notifiers before dump happened. So calling it inside of crash_kexec() does not make much sense from code point of view.

- Why does somebody need to keep track of event KMSG_DUMP_KEXEC?

- There is one kernel CONFIG option introduce which looks completely superfluous.

My general take on the whole issue.

- In general I think exporting a hook to module so that they can do anything before crash is a bad idea. Now this can be overloaded to do things like sending crash notifications in clustered environement where we recommend doing it in second kernel.

- Even if we really have to do it, there seemed to be two concern areas.

- Reliability of kdump_msg() generic infrastructure and its capability in terms of handling races with other cpus and

	  NMIs.

	- Reliability of module which is getting the callback from
	  kdump_msg().

I think in one of the mails I was discussing that common infrastructure between kdump and kmsg_dump() can be put in a separate function, like stopping all cpus etc to avoid races in generic infrastrucutre and then first we can all kmsg_dump() and then crash_kexec().

But this still does not provide us any protection against modules getting control after crash and possiblly worsen the situation.

Hi

Sorry for the delay. I had got stuck LinuxCon Japan and piled up plenty paper works.

Quoted 8 lines

Quoted 7 lines

Quoted 7 lines

I think I can agree your proposal. But could you please explain why do you think kmsg _before_ kdump and kmsg _in_ kdump are so different? I think it is only C level difference. CPU don't care C function and anyway the kernel call kmsg_dump() because invoke second kernel even if you proposal applied.
It is only curious. I'm not against your proposal.
Thanks.

Quoted 6 lines

Few reasons.

- There is no correlation between crash_kexec() and kdump_msg(). What you are creating is equivalent of panic notifiers and calling those notifiers before dump happened. So calling it inside of crash_kexec() does not make much sense from code point of view.

Thank you for the replay. I got you _think_ no makes sense, but I haven't explain what you talk about the code of "code point of view". If you read the code, you can understand kdump_msg() and panic_notifiers are not same point.

Quoted 1 line

- Why does somebody need to keep track of event KMSG_DUMP_KEXEC?

I believe I answered already at last thread.

http://groups.google.com/group/linux.kernel/browse_thread/thread/1084f406573d76ac/daa1a08804385089?q=kexec%3A+remove+KMSG_DUMP_KEXEC&lnk=ol&

Quoted 3 lines

- There is one kernel CONFIG option introduce which looks completely superfluous.

What you mean "superfluous"? We already have billion kernel CONFIG. Is it also bad?

Quoted 7 lines

My general take on the whole issue.

- In general I think exporting a hook to module so that they can do anything before crash is a bad idea. Now this can be overloaded to do things like sending crash notifications in clustered environement where we recommend doing it in second kernel.

??
It's not my issue and I haven't talked about such thing. I guess you confuse I and Aguch-san or someone else.

Quoted 11 lines

- Even if we really have to do it, there seemed to be two concern areas.

- Reliability of kdump_msg() generic infrastructure and its capability in terms of handling races with other cpus and

	  NMIs.

	- Reliability of module which is getting the callback from
	  kdump_msg().

Indeed. I thought Aguch-san said he promised he work on improve them. However it doesn't happen yet. Okay, I

Quoted 4 lines

I think in one of the mails I was discussing that common infrastructure between kdump and kmsg_dump() can be put in a separate function, like stopping all cpus etc to avoid races in generic infrastrucutre and then first we can all kmsg_dump() and then crash_kexec().

Nice idea! Yes. I didn't think enterprise folks start to use this feature and it now happen.
If nobody are working on this, I'll do it.

Quoted 4 lines

But this still does not provide us any protection against modules getting control after crash and possiblly worsen the situation.

I think this doesn't big matter. If module author hope to get hook, they can use kprobe in nowadays. I don't think we can make perfect kprobe protection. I think I wrote this at last thread too.

Probably most reliability stupid module detect way is, watching lkml and revewing kmsg_dump() user conteniously. However, if you strongly worry about this issue, I can agree we make tiny foolish module protection. (but I don't have concrete idea yet)

Hi,

As Americo said, sometimes kdump doesn't work in the real world.

So, I would like to keep discussing implementation of kmsg_dump(KMSG_DUMP_KEXEC) rather than just removing it.

What are you using kmsg_dump() for? Using mtdoops, ramoops or something else? Is it working reliably for you?

Hi,

Quoted 2 lines

What are you using kmsg_dump() for? Using mtdoops, ramoops or something else? Is it working reliably for you?

I plan to use kmsg_dump() for set_variable service of UEFI. I proposed a prototype patch this month and will improve it. (kmsg_dump is used inside pstore.)

https://lkml.org/lkml/2011/5/10/340

Quoted 8 lines

Quoted 2 lines

What are you using kmsg_dump() for? Using mtdoops, ramoops or something else? Is it working reliably for you?

I plan to use kmsg_dump() for set_variable service of UEFI. I proposed a prototype patch this month and will improve it. (kmsg_dump is used inside pstore.)

https://lkml.org/lkml/2011/5/10/340

Shudder. Firmware calls in the crash path.

If that is the use, we need to remove the kmsg_dump(KMSG_DUMP_KEXEC) hook from crash_kexec yesterday. It is leading to some really ludicrous suggestions that are on the way from making kexec on panic unreliable and useless.

There will always be EFI implementations where that will not work and there will be no way we can fix those.

There is a long history of people trying to do things in a crashing kernel, things that simply do not work when the system is in a bad state. kmsg_dump() when I reviewed the code had significant implementation problems for being called from interrupt handlers and the like.

To introduce a different solution for capturing information when a kernel crashes we need to see numbers that in a large number of situations that the mechanism you are proposing is more reliable and/or more maintainable than the current kexec on panic implementation.

The best work I know of on the reliability of the current situation
is "Evaluating Linux Kernel Crash Dumping Mechanisms", by Fernando Luis Vazquez Cao.
http://www.linuxsymposium.org/archives/OLS/Reprints-2006/cao-reprint.pdf

Now it does happen to be a fact that our efi support in linux is so buggy kexec does not work let alone kexec on panic (if the target
kernel has any efi support). But our efi support being buggy is not
a reason to add more ways to fail when we have a kernel with efi support. It is an argument to remove our excessive use of EFI calls.

So let's just remove the ridiculous kmsg_dump(KMSG_DUMP_KEXEC) hook from crash_kexec and remove any temptation for abuses like wanting to use kmsg_dump() on anything but a deeply embedded system where there simply is not enough memory for 2 kernels.

Quoted 9 lines · Seiji Aguchi

Hi,

Quoted 2 lines

What are you using kmsg_dump() for? Using mtdoops, ramoops or something else? Is it working reliably for you?

I plan to use kmsg_dump() for set_variable service of UEFI. I proposed a prototype patch this month and will improve it. (kmsg_dump is used inside pstore.)

Quoted 1 line

https://lkml.org/lkml/2011/5/10/340

Shudder. Firmware calls in the crash path.

If that is the use, we need to remove the kmsg_dump(KMSG_DUMP_KEXEC) hook from crash_kexec yesterday. It is leading to some really ludicrous suggestions that are on the way from making kexec on panic unreliable and useless.

There will always be EFI implementations where that will not work and there will be no way we can fix those.

There is a long history of people trying to do things in a crashing kernel, things that simply do not work when the system is in a bad state. kmsg_dump() when I reviewed the code had significant implementation problems for being called from interrupt handlers and the like.

To introduce a different solution for capturing information when a kernel crashes we need to see numbers that in a large number of situations that the mechanism you are proposing is more reliable and/or more maintainable than the current kexec on panic implementation.

The best work I know of on the reliability of the current situation
is "Evaluating Linux Kernel Crash Dumping Mechanisms", by Fernando Luis Vazquez Cao.
http://www.linuxsymposium.org/archives/OLS/Reprints-2006/cao-reprint.pdf

Now it does happen to be a fact that our efi support in linux is so buggy kexec does not work let alone kexec on panic (if the target
kernel has any efi support). But our efi support being buggy is not
a reason to add more ways to fail when we have a kernel with efi support. It is an argument to remove our excessive use of EFI calls.

So let's just remove the ridiculous kmsg_dump(KMSG_DUMP_KEXEC) hook from crash_kexec and remove any temptation for abuses like wanting to use kmsg_dump() on anything but a deeply embedded system where there simply is not enough memory for 2 kernels.

Hi,

I noticed following commit which hooks into crash_kexec() for calling kmsg_dump().

I think it is not a very good idea to pass control to modules after crash_kexec() has been called. Because modules can try to take locks or try to do some other operations which we really should not be doing now and fail kdump also. The whole design of kdump is built on the fact that in crashing kernel we do minimal thing and try to make transition to second kernel robust. Now with this hook, kmsg dumper breaks that assumption.

Anyway, if an image is loaded and we have setup to capture dump also why do we need to save kmsg with the help of an helper. I am assuming this is more of a debugging aid if we have no other way to capture the kernel log buffer. So if somebody has setup kdump to capture the vmcore, why to call another handler which tries to save part of the vmcore (kmsg) separately.

Hi

Quoted 5 lines

Hi,

I noticed following commit which hooks into crash_kexec() for calling kmsg_dump().

Quoted 7 lines

I think it is not a very good idea to pass control to modules after crash_kexec() has been called. Because modules can try to take locks or try to do some other operations which we really should not be doing now and fail kdump also. The whole design of kdump is built on the fact that in crashing kernel we do minimal thing and try to make transition to second kernel robust. Now with this hook, kmsg dumper breaks that assumption.

I guess you talked about some foolish can shoot their own foot. if so, Yes. Any kernel module can make kernel panic or more disaster result.

Quoted 8 lines

Anyway, if an image is loaded and we have setup to capture dump also why do we need to save kmsg with the help of an helper. I am assuming this is more of a debugging aid if we have no other way to capture the kernel log buffer. So if somebody has setup kdump to capture the vmcore, why to call another handler which tries to save part of the vmcore (kmsg) separately.

No.

kmsg_dump() is desingned for embedded.

kexec for non dumping purpose. (Have you seen your embedded devices show "Now storing dump image.." message?)

Anyway, you can feel free to avoid to use ksmg_dump().

Quoted 15 lines

Hi

Quoted 5 lines

Hi,

I noticed following commit which hooks into crash_kexec() for calling kmsg_dump().

Quoted 7 lines

I think it is not a very good idea to pass control to modules after crash_kexec() has been called. Because modules can try to take locks or try to do some other operations which we really should not be doing now and fail kdump also. The whole design of kdump is built on the fact that in crashing kernel we do minimal thing and try to make transition to second kernel robust. Now with this hook, kmsg dumper breaks that assumption.

Quoted 2 lines

I guess you talked about some foolish can shoot their own foot. if so, Yes. Any kernel module can make kernel panic or more disaster result.

Yes, the difference is that once a fool shoots his foot, kernel tries to take a meaningful action to figure out what went wrong. Like displayig an oops backtrace or like dumping a core (if kdump is configured) so that one can figure out who was the fool and what did who do.

Now think give the control to two fools. First fool shoots his foot and then kernel transfers the control to another fool which completely screws up the situation and one can not save the core.

Quoted 7 lines

Quoted 6 lines

Anyway, if an image is loaded and we have setup to capture dump also why do we need to save kmsg with the help of an helper. I am assuming this is more of a debugging aid if we have no other way to capture the kernel log buffer. So if somebody has setup kdump to capture the vmcore, why to call another handler which tries to save part of the vmcore (kmsg) separately.

Quoted 3 lines

No.

kmsg_dump() is desingned for embedded.

Great. And I like the idea of trying to save some useful information to non volatile RAM or flash or something like that.

Quoted 3 lines

kexec for non dumping purpose. (Have you seen your embedded devices show "Now storing dump image.." message?)

No I have not seen. Can you explain a bit more that apart from kernel dump, what are the other purposes of kdump.

Quoted 2 lines

Anyway, you can feel free to avoid to use ksmg_dump().

Yes, that is one more way but this information is not even exported to user space to figure out if there are any registerd users of kmsg_dump.

Seconly there are two more important things.

- Why do you need a notification from inside crash_kexec(). IOW, what is the usage of KMSG_DUMP_KEXEC.

- One can anyway call kmsg_dump() outside crash_kexec() before it so that kmsg_dump notification will go out before kdump gets the control. What I am arguing here is that it is not necessarily a very good idea because external modules can try to do any amount of unsafe actions once we export the hook.

Doing this is still fine if kdump is not configured as anyway syste would have rebooted. But if kdump is configured, then we are just reducing the reliability of the operation by passing the control in the hands of unaudited code and trusting it when kernel data structures are corrupt.

So to me, sending out kmsg_dump notifications are perfectly fine when kdump is not configured. But if it is configured, then it probably is not a good idea. Anyway, if you have configured the system to capture the full dump, why do you also need kmsg_dump. And if you are happy with kmsg_dump() then you don't need kdump. So these both seem to be mutually exclusive anyway.

AFAIK, kexec is used sneak rebooting way when the system face unexpected scenario on some devices. (Some embedded is running very long time, then it can't avoid memory bit corruption. all of reset is a last resort. and a vendor gather logs at periodically checkback).

The main purpose of to introduce KMSG_DUMP_KEXEC is to be separate it from KMSG_DUMP_PANIC. At kmsg_dump() initial patch, KMSG_DUMP_PANIC is always called both kdump is configured or not. But it's no good idea the same log is to be appeared when both kexec was successed and failured. Moreover someone don't want any log at kexec phase. They only want logs when real panic (ie kexec failure) route. Then, I've separated it to two. Two separated argument can solve above both requreiment.

Quoted 5 lines

AFAIK, kexec is used sneak rebooting way when the system face unexpected scenario on some devices. (Some embedded is running very long time, then it can't avoid memory bit corruption. all of reset is a last resort. and a vendor gather logs at periodically checkback).

Quoted 7 lines

The main purpose of to introduce KMSG_DUMP_KEXEC is to be separate it from KMSG_DUMP_PANIC. At kmsg_dump() initial patch, KMSG_DUMP_PANIC is always called both kdump is configured or not. But it's no good idea the same log is to be appeared when both kexec was successed and failured. Moreover someone don't want any log at kexec phase. They only want logs when real panic (ie kexec failure) route. Then, I've separated it to two. Two separated argument can solve above both requreiment.

A bit additional explanation, An original patch have kmsg_dump(KMSG_DUMP_PANIC) callsite at following point. I didn't think it makes either embedded or kdump folks happiness. Thus I moved it after crash_kexec().

---------------------------------------------------------------------

(quoted diff) +1 -0

@@ -74,6 +75,7 @@ NORET_TYPE void panic(const char * fmt, ...)
        dump_stack();
 #endif
 
+       kmsg_dump(KMSG_DUMP_PANIC);
        /*
         * If we have crashed and we have a crash kernel loaded let it handle
         * everything else.

         * Do we want to call this before we try to display a message?
         */
        crash_kexec(NULL);

---------------------------------------------------------------------

Quoted 12 lines

---------------------------------------------------------------------

(quoted diff) +1 -0

@@ -74,6 +75,7 @@ NORET_TYPE void panic(const char * fmt, ...)
        dump_stack();
 #endif
 
+       kmsg_dump(KMSG_DUMP_PANIC);
        /*
         * If we have crashed and we have a crash kernel loaded let it handle
         * everything else.

         * Do we want to call this before we try to display a message?
         */
        crash_kexec(NULL);

Quoted 1 line

---------------------------------------------------------------------

And I think to compensate for that somebody introduced additional kmsg_dump(KEXEC) call inside crash_kexec() and put it under CONFIG option so that one can change the behavior based on config options.

I think this makes the logic somewhat twisted and an unnecessary call inside crash_kexec(). So until and unless there is a strong reason we can get rid of KEXEC event and move kmsg_dump call before crash_kexec() for now and see how does it go, IMHO.

Quoted 4 lines

Seconly there are two more important things.

- Why do you need a notification from inside crash_kexec(). IOW, what is the usage of KMSG_DUMP_KEXEC.

AFAIK, kexec is used sneak rebooting way when the system face unexpected scenario on some devices. (Some embedded is running very long time, then it can't avoid memory bit corruption. all of reset is a last resort. and a vendor gather logs at periodically checkback).

The main purpose of to introduce KMSG_DUMP_KEXEC is to be separate it from KMSG_DUMP_PANIC. At kmsg_dump() initial patch, KMSG_DUMP_PANIC is always called both kdump is configured or not. But it's no good idea the same log is to be appeared when both kexec was successed and failured. Moreover someone don't want any log at kexec phase. They only want logs when real panic (ie kexec failure) route. Then, I've separated it to two. Two separated argument can solve above both requreiment.

Quoted 7 lines

- One can anyway call kmsg_dump() outside crash_kexec() before it so that kmsg_dump notification will go out before kdump gets the control. What I am arguing here is that it is not necessarily a very good idea because external modules can try to do any amount of unsafe actions once we export the hook.

I wrote why I don't think I added new risk. (shortly, It can be a lot of another way)
Can you please tell me your view of my point? I'm afraid I haven't understand your worry. So, I hope to understand it before making alternative fixing patch.

Quoted 6 lines

Doing this is still fine if kdump is not configured as anyway syste would have rebooted. But if kdump is configured, then we are just reducing the reliability of the operation by passing the control in the hands of unaudited code and trusting it when kernel data structures are corrupt.

At minimum, I'm fully agree we need reliable kdump. I only put a doubtness this removing is a real solution or not.

Quoted 6 lines

So to me, sending out kmsg_dump notifications are perfectly fine when kdump is not configured. But if it is configured, then it probably is not a good idea. Anyway, if you have configured the system to capture the full dump, why do you also need kmsg_dump. And if you are happy with kmsg_dump() then you don't need kdump. So these both seem to be mutually exclusive anyway.

Honestly, I haven't heared anyone are using both at the same time. But I can guess some reason. 1) If the system is very big box, kdump is really slooooooow operation. example Some stock exchange system have half terabytes memory and it mean dump delivery need to hald days at worse. But market should open just 9:00 at next day. So, summry information (eg log and trace information) spoiling is important thing. 2) Two sequence crash (ie crash kdump reboot-start next-crash-before-finish-reboot) can override former dump image. Usually admin _guess_ the reason of two are same and report boss so. But unfortunatelly customers at high end area often refuse a _guess_ report.

Or, it's for business competition reason. As far as I heared, IBM and HP UNI*X system can save the logs both dump and special flash device.

PS: FWIW, Hitach folks have usage idea for their enterprise purpose, but
unfortunately I don't know its detail. I hope anyone tell us it.

PS: FWIW, Hitach folks have usage idea for their enterprise purpose, but
unfortunately I don't know its detail. I hope anyone tell us it.

- Why do you need a notification from inside crash_kexec(). IOW, what is the usage of KMSG_DUMP_KEXEC.

- One can anyway call kmsg_dump() outside crash_kexec() before it so that kmsg_dump notification will go out before kdump gets the control. What I am arguing here is that it is not necessarily a very good idea because external modules can try to do any amount of unsafe actions once we export the hook.

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Vivek Goyal
Sent: Wednesday, February 02, 2011 9:05 PM
To: KOSAKI Motohiro
Cc: Eric W. Biederman; linux kernel mailing list; Jarod Wilson
Subject: Re: Query about kdump_msg hook into crash_kexec()

Quoted 15 lines · KOSAKI Motohiro

Hi

Quoted 5 lines

Hi,

I noticed following commit which hooks into crash_kexec() for calling kmsg_dump().

Quoted 7 lines

I think it is not a very good idea to pass control to modules after crash_kexec() has been called. Because modules can try to take locks or try to do some other operations which we really should not be doing now and fail kdump also. The whole design of kdump is built on the fact that in crashing kernel we do minimal thing and try to make transition to second kernel robust. Now with this hook, kmsg dumper breaks that assumption.

Quoted 2 lines

I guess you talked about some foolish can shoot their own foot. if so, Yes. Any kernel module can make kernel panic or more disaster result.

Yes, the difference is that once a fool shoots his foot, kernel tries to take a meaningful action to figure out what went wrong. Like displayig an oops backtrace or like dumping a core (if kdump is configured) so that one can figure out who was the fool and what did who do.

Now think give the control to two fools. First fool shoots his foot and then kernel transfers the control to another fool which completely screws up the situation and one can not save the core.

Quoted 9 lines

Quoted 8 lines

Anyway, if an image is loaded and we have setup to capture dump also why do we need to save kmsg with the help of an helper. I am assuming this is more of a debugging aid if we have no other way to capture the kernel log buffer. So if somebody has setup kdump to capture the vmcore, why to call another handler which tries to save part of the vmcore (kmsg) separately.

Quoted 3 lines

No.

kmsg_dump() is desingned for embedded.

Great. And I like the idea of trying to save some useful information to non volatile RAM or flash or something like that.

Quoted 2 lines

kexec for non dumping purpose. (Have you seen your embedded devices show "Now storing dump image.." message?)

No I have not seen. Can you explain a bit more that apart from kernel dump, what are the other purposes of kdump.

Quoted 2 lines

Anyway, you can feel free to avoid to use ksmg_dump().

Yes, that is one more way but this information is not even exported to user space to figure out if there are any registerd users of kmsg_dump.

Seconly there are two more important things.

- Why do you need a notification from inside crash_kexec(). IOW, what is the usage of KMSG_DUMP_KEXEC.

- One can anyway call kmsg_dump() outside crash_kexec() before it so that kmsg_dump notification will go out before kdump gets the control. What I am arguing here is that it is not necessarily a very good idea because external modules can try to do any amount of unsafe actions once we export the hook.

Doing this is still fine if kdump is not configured as anyway syste would have rebooted. But if kdump is configured, then we are just reducing the reliability of the operation by passing the control in the hands of unaudited code and trusting it when kernel data structures are corrupt.

So to me, sending out kmsg_dump notifications are perfectly fine when kdump is not configured. But if it is configured, then it probably is not a good idea. Anyway, if you have configured the system to capture the full dump, why do you also need kmsg_dump. And if you are happy with kmsg_dump() then you don't need kdump. So these both seem to be mutually exclusive anyway.

Thanks
Vivek
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Hi,

Quoted 2 lines

PS: FWIW, Hitach folks have usage idea for their enterprise purpose, but
unfortunately I don't know its detail. I hope anyone tell us it.

I explain the usage of kmsg_dump(KMSG_DUMP_KEXEC) in enterprise area.

[Background]
In our support service experience, we always need to detect root cause of OS panic.
So, customers in enterprise area never forgive us if kdump fails and we can't detect the root cause of panic due to lack of materials for investigation.

Quoted 2 lines

- Why do you need a notification from inside crash_kexec(). IOW, what is the usage of KMSG_DUMP_KEXEC.

The usage of kdump(KMSG_DUMP_KEXEC) in enterprise area is getting useful information for investigating kernel crash in case kdump kernel doesn't boot.

Kdump kernel may not start booting because there is a sha256 checksum verified over the kdump kernel before it starts booting. This means kdump kernel may fail even if there is no bug in kdump and we can't get any information for detecting root cause of kernel crash

As I mentioned in [Background], We must avoid lack of materials for investigation.
So, kdump(KMSG_DUMP_KEXEC) is very important feature in enterprise area.

-----Original Message-----
From: Eric W. Biederman [mailto:[email protected]]
Sent: Thursday, February 03, 2011 4:13 PM
To: Seiji Aguchi
Cc: Vivek Goyal; KOSAKI Motohiro; linux kernel mailing list; Jarod Wilson
Subject: Re: Query about kdump_msg hook into crash_kexec()

Quoted 18 lines · Seiji Aguchi

Hi,

Quoted 2 lines

PS: FWIW, Hitach folks have usage idea for their enterprise purpose, but
unfortunately I don't know its detail. I hope anyone tell us it.

I explain the usage of kmsg_dump(KMSG_DUMP_KEXEC) in enterprise area.

[Background]
In our support service experience, we always need to detect root cause of OS panic.
So, customers in enterprise area never forgive us if kdump fails and we can't detect the root cause of panic due to lack of materials for investigation.

Quoted 2 lines

- Why do you need a notification from inside crash_kexec(). IOW, what is the usage of KMSG_DUMP_KEXEC.

Quoted 8 lines

The usage of kdump(KMSG_DUMP_KEXEC) in enterprise area is getting useful information for investigating kernel crash in case kdump kernel doesn't boot.

Kdump kernel may not start booting because there is a sha256 checksum verified over the kdump kernel before it starts booting. This means kdump kernel may fail even if there is no bug in kdump and we can't get any information for detecting root cause of kernel crash

Sure it is theoretically possible that the sha256 checksum gets corrupted (I have never seen it happen or heard reports of it happening). It is a feature that if someone has corrupted your code the code doesn't try and run anyway and corrupt anything else.

That you are arguing against have such a feature in the code you use to write to persistent storage is scary.

Quoted 4 lines

As I mentioned in [Background], We must avoid lack of materials for investigation.
So, kdump(KMSG_DUMP_KEXEC) is very important feature in enterprise area.

That sounds wonderful, but it doesn't jive with the
code. kmsg_dump(KMSG_DUMP_KEXEC) when I read through it was simply not written to be robust when most of the kernel is suspect. Making it in appropriate for use on the crash_kexec path. I do not believe kmsg_dump has seen any testing in kernel failure scenarios.

There is this huge assumption that kmsg_dump is more reliable than crash_kexec, from my review of the code kmsg_dump is simply not safe in the context of a broken kernel. The kmsg_dump code last I looked code won't work if called with interrupts disabled.

Furthermore kmsg_dump(KMSG_DUMP_KEXEC) is only useful for debugging crash_kexec. Which has it backwards as it is kmsg_dump that needs the debugging.

You just argued that it is better to corrupt the target of your kmsg_dump in the event of a kernel failure instead of to fail silently.

I don't want that unreliable code that wants to corrupt my jffs partition anywhere near my machines.

Eric

Hi Eric,

Thank you for your prompt reply.

I would like to consider "Needs in enterprise area" and "Implementation of kmsg_dump()" separately.

(1) Needs in enterprise area
In case of kdump failure, we would like to store kernel buffer to NVRAM/flush memory for detecting root cause of kernel crash.

(2) Implementation of kmsg_dump
You suggest to review/test cording of kmsg_dump() more.

What do you think about (1)?
Is it acceptable for you?

Quoted 13 lines · Seiji Aguchi

Hi Eric,

Thank you for your prompt reply.

I would like to consider "Needs in enterprise area" and "Implementation of kmsg_dump()" separately.

(1) Needs in enterprise area
In case of kdump failure, we would like to store kernel buffer to NVRAM/flush memory for detecting root cause of kernel crash.

(2) Implementation of kmsg_dump
You suggest to review/test cording of kmsg_dump() more.

Quoted 3 lines

What do you think about (1)?
Is it acceptable for you?

For "in case of kdump fails", we can move KMSG_DUMP_OOPS/KMSG_DUMP_PANIC before calling crash_kexec(), so you can capture the log before kdump happens.

Quoted 18 lines · Américo Wang

Quoted 16 lines · Seiji Aguchi

Hi Eric,

Thank you for your prompt reply.

I would like to consider "Needs in enterprise area" and "Implementation of kmsg_dump()" separately.

(1) Needs in enterprise area
In case of kdump failure, we would like to store kernel buffer to NVRAM/flush memory for detecting root cause of kernel crash.

(2) Implementation of kmsg_dump
You suggest to review/test cording of kmsg_dump() more.

What do you think about (1)?
Is it acceptable for you?

Quoted 3 lines

For "in case of kdump fails", we can move KMSG_DUMP_OOPS/KMSG_DUMP_PANIC before calling crash_kexec(), so you can capture the log before kdump happens.

True. If the idea is to capture the logs before kdump starts saving core, then kdump_msg() call, can be put before crash_kexec() call and there is no need to hook into crash_kexec().

Secondly now the question of whether kdump_msg() call should be before crash_kexec() or not because modules might want to do lots of unreliable things, I am now split on that. Especially because of the fact that if somebody wants it probably can use kprobe to hook into crash_kexec() or panic() or something like that to execute its code before everything else.

The other reason I am little split on this because in the past I have seen kdump fail many a times either because of chipset issues or because of driver issues etc. So though I don't like it and I want drivers to be fixed to take care of booting in kdump environment, things not necessarily worked as well as we expected them to and it is not hard to meet a kdump failure.

Hence though I do not prefer it but I will not stand in the way of kdump_msg() being called before crash_kexec() gets the control.

Quoted 6 lines

Secondly now the question of whether kdump_msg() call should be before crash_kexec() or not because modules might want to do lots of unreliable things, I am now split on that. Especially because of the fact that if somebody wants it probably can use kprobe to hook into crash_kexec() or panic() or something like that to execute its code before everything else.

If kprobe is the reason, then probably we can remove lots of other kernel API's like kmsg dumper.

Hi Eric,

Thank you for your prompt reply.

I would like to consider "Needs in enterprise area" and "Implementation of kmsg_dump()" separately.

(1) Needs in enterprise area
In case of kdump failure, we would like to store kernel buffer to NVRAM/flush memory for detecting root cause of kernel crash.

(2) Implementation of kmsg_dump
You suggest to review/test cording of kmsg_dump() more.

What do you think about (1)?
Is it acceptable for you?

Seiji

Quoted 71 lines

-----Original Message-----
From: Eric W. Biederman [mailto:[email protected]]
Sent: Thursday, February 03, 2011 4:13 PM
To: Seiji Aguchi
Cc: Vivek Goyal; KOSAKI Motohiro; linux kernel mailing list; Jarod Wilson
Subject: Re: Query about kdump_msg hook into crash_kexec()

Quoted 18 lines · Seiji Aguchi

Hi,

Quoted 2 lines

PS: FWIW, Hitach folks have usage idea for their enterprise purpose, but
unfortunately I don't know its detail. I hope anyone tell us it.

I explain the usage of kmsg_dump(KMSG_DUMP_KEXEC) in enterprise area.

[Background]
In our support service experience, we always need to detect root cause of OS panic.
So, customers in enterprise area never forgive us if kdump fails and we can't detect the root cause of panic due to lack of materials for investigation.

Quoted 2 lines

- Why do you need a notification from inside crash_kexec(). IOW, what is the usage of KMSG_DUMP_KEXEC.

Quoted 8 lines

The usage of kdump(KMSG_DUMP_KEXEC) in enterprise area is getting useful information for investigating kernel crash in case kdump kernel doesn't boot.

Kdump kernel may not start booting because there is a sha256 checksum verified over the kdump kernel before it starts booting. This means kdump kernel may fail even if there is no bug in kdump and we can't get any information for detecting root cause of kernel crash

Sure it is theoretically possible that the sha256 checksum gets corrupted (I have never seen it happen or heard reports of it happening). It is a feature that if someone has corrupted your code the code doesn't try and run anyway and corrupt anything else.

That you are arguing against have such a feature in the code you use to write to persistent storage is scary.

Quoted 4 lines

As I mentioned in [Background], We must avoid lack of materials for investigation.
So, kdump(KMSG_DUMP_KEXEC) is very important feature in enterprise area.

That sounds wonderful, but it doesn't jive with the
code. kmsg_dump(KMSG_DUMP_KEXEC) when I read through it was simply not written to be robust when most of the kernel is suspect. Making it in appropriate for use on the crash_kexec path. I do not believe kmsg_dump has seen any testing in kernel failure scenarios.

There is this huge assumption that kmsg_dump is more reliable than crash_kexec, from my review of the code kmsg_dump is simply not safe in the context of a broken kernel. The kmsg_dump code last I looked code won't work if called with interrupts disabled.

Furthermore kmsg_dump(KMSG_DUMP_KEXEC) is only useful for debugging crash_kexec. Which has it backwards as it is kmsg_dump that needs the debugging.

You just argued that it is better to corrupt the target of your kmsg_dump in the event of a kernel failure instead of to fail silently.

I don't want that unreliable code that wants to corrupt my jffs partition anywhere near my machines.

Eric

Quoted 10 lines · Seiji Aguchi

Hi Eric,

Thank you for your prompt reply.

I would like to consider "Needs in enterprise area" and "Implementation of kmsg_dump()" separately.

(1) Needs in enterprise area
In case of kdump failure, we would like to store kernel buffer to NVRAM/flush memory for detecting root cause of kernel crash.

Quoted 5 lines

(2) Implementation of kmsg_dump
You suggest to review/test cording of kmsg_dump() more.

What do you think about (1)?
Is it acceptable for you?

Ok, I am just trying to think loud about this problem and see if something fruitful comes out which paves the way forward.

- So ideally we would like kdump_msg() to be called after crash_kexec() so that any unaudited (third party modules), unreliable calls do not compromise the realiability of kdump operation.

But hitachi folks seems to be wanting to save atleast kernel buffers somwhere in the NVRAM etc because they think that kdump can be unreliable and we might not capture any information after the crash. So they kind of want two mechanisms in place. One is light weight which tries to save kernel buffers in NVRAM and then one heavy weight one which tries to save the entire/filtered kernel core.

Personally I am not too excited about the idea but I guess I can live with it. We can try to audit atleast in kernel module and for external modules we don't have much control and live with the fact that if modules screw up, we don't capture the dump.

Those who don't want this behavior can do three things.

- Disable kdump_msg() at compile time.
- Do not load any module which registers for kdump_msg()
- Implement a /proc tunable which allows controlling this behavior.

- Ok, having said why do we want it, comes the question of how to do it so that it works reasonably well.

- There seems to be on common requirement of kmsg_dump() and kdump() and that is stop other cpus reliably (use nmi if possible). Can we try to share this code between kmsg_dump and crash_kexec(). So something like as follows.

- panic happens
- Do all the activities related to printing panic string and stack dump.
- Stop other cpus.
- This can be probably be done with the equivalent of machine_crash_shutdown() function. In fact this function can probably be broken down in two parts. First part does shutdown_prepare() where all other cpus are shot down and second part can do the actual disabling of LAPIC/IOAPIC and saving cpu registers etc.

if (mutex_trylock(some_shutdown_mutex)) {
/* setp regs, fix vmcoreinfo etc */ crash_kexec_prepare(); machine_shutdown_prepare(); kdump_msg(); crash_kexec_execute()
/* Also call panic_notifier_list here ? */
}

crash_kexec_prepare () {
		crash_setup_regs(&fixed_regs, regs);
		crash_save_vmcoreinfo();
}

crash_kexec_execute() {
			/* Shutdown lapic/ioapic, save this cpu register etc */
			machine_shutdown();
			machine_kexec()
}

So basically we break down machine_shutdown() function in two parts and start sharing common part between kdump_msg(), crash_kexec and possibly panic_notifiers.

If kdump is not configured, then after executing kdump_msg() and panic notifiers, we should either be sitting in tight loop with interrupt enabled for somebody to press Ctrl-boot or reboot system upon lapse of panic_timeout().

Eric, does it make sense to you?

Quoted 86 lines · Vivek Goyal

Quoted 10 lines · Seiji Aguchi

Hi Eric,

Thank you for your prompt reply.

I would like to consider "Needs in enterprise area" and "Implementation of kmsg_dump()" separately.

(1) Needs in enterprise area
In case of kdump failure, we would like to store kernel buffer to NVRAM/flush memory for detecting root cause of kernel crash.

Quoted 5 lines

(2) Implementation of kmsg_dump
You suggest to review/test cording of kmsg_dump() more.

What do you think about (1)?
Is it acceptable for you?

Ok, I am just trying to think loud about this problem and see if something fruitful comes out which paves the way forward.

- So ideally we would like kdump_msg() to be called after crash_kexec() so that any unaudited (third party modules), unreliable calls do not compromise the realiability of kdump operation.

But hitachi folks seems to be wanting to save atleast kernel buffers somwhere in the NVRAM etc because they think that kdump can be unreliable and we might not capture any information after the crash. So they kind of want two mechanisms in place. One is light weight which tries to save kernel buffers in NVRAM and then one heavy weight one which tries to save the entire/filtered kernel core.

Personally I am not too excited about the idea but I guess I can live with it. We can try to audit atleast in kernel module and for external modules we don't have much control and live with the fact that if modules screw up, we don't capture the dump.

Those who don't want this behavior can do three things.

- Disable kdump_msg() at compile time.
- Do not load any module which registers for kdump_msg()
- Implement a /proc tunable which allows controlling this behavior.

- Ok, having said why do we want it, comes the question of how to do it so that it works reasonably well.

- There seems to be on common requirement of kmsg_dump() and kdump() and that is stop other cpus reliably (use nmi if possible). Can we try to share this code between kmsg_dump and crash_kexec(). So something like as follows.

- panic happens
- Do all the activities related to printing panic string and stack dump.
- Stop other cpus.
- This can be probably be done with the equivalent of machine_crash_shutdown() function. In fact this function can probably be broken down in two parts. First part does shutdown_prepare() where all other cpus are shot down and second part can do the actual disabling of LAPIC/IOAPIC and saving cpu registers etc.

if (mutex_trylock(some_shutdown_mutex)) {
/* setp regs, fix vmcoreinfo etc */ crash_kexec_prepare(); machine_shutdown_prepare(); kdump_msg(); crash_kexec_execute()
/* Also call panic_notifier_list here ? */
}

crash_kexec_prepare () {
		crash_setup_regs(&fixed_regs, regs);
		crash_save_vmcoreinfo();
}

crash_kexec_execute() {
			/* Shutdown lapic/ioapic, save this cpu register etc */
			machine_shutdown();
			machine_kexec()
}

So basically we break down machine_shutdown() function in two parts and start sharing common part between kdump_msg(), crash_kexec and possibly panic_notifiers.

Quoted 6 lines

If kdump is not configured, then after executing kdump_msg() and panic notifiers, we should either be sitting in tight loop with interrupt enabled for somebody to press Ctrl-boot or reboot system upon lapse of panic_timeout().

Eric, does it make sense to you?

kexec on panic doesn't strictly require that we stop other cpus.

What makes sense to me at this point is for someone on the kmsg_dump side to make a strong case that the code actually works in a crash dump scenario. We have lots of experience over the years that says a design like kmsg_dump is attractive but turns out to be a unreliable piece of junk that fails just when you need it. Because developers only test the case when the kernel is happy and because people share code with the regular path drivers, and that code assumes things are happy.

I forget exactly why but last I looked.
local_irq_disable()
kmsg_dump()
local_irq_enalbe()

Was a recipe for disaster, and you have be at least that good to even have a chance of working in a crash dump scenario.

In part I am puzzled why the kmsg dump doesn't just use the printk interface. Strangely enough printk works in the event of a crash and has been shown to be reliable over the years.

Quoted 88 lines · Eric W. Biederman

Quoted 86 lines · Vivek Goyal

Quoted 10 lines · Seiji Aguchi

Hi Eric,

Thank you for your prompt reply.

I would like to consider "Needs in enterprise area" and "Implementation of kmsg_dump()" separately.

(1) Needs in enterprise area
In case of kdump failure, we would like to store kernel buffer to NVRAM/flush memory for detecting root cause of kernel crash.

Quoted 5 lines

(2) Implementation of kmsg_dump
You suggest to review/test cording of kmsg_dump() more.

What do you think about (1)?
Is it acceptable for you?

Ok, I am just trying to think loud about this problem and see if something fruitful comes out which paves the way forward.

- So ideally we would like kdump_msg() to be called after crash_kexec() so that any unaudited (third party modules), unreliable calls do not compromise the realiability of kdump operation.

But hitachi folks seems to be wanting to save atleast kernel buffers somwhere in the NVRAM etc because they think that kdump can be unreliable and we might not capture any information after the crash. So they kind of want two mechanisms in place. One is light weight which tries to save kernel buffers in NVRAM and then one heavy weight one which tries to save the entire/filtered kernel core.

Personally I am not too excited about the idea but I guess I can live with it. We can try to audit atleast in kernel module and for external modules we don't have much control and live with the fact that if modules screw up, we don't capture the dump.

Those who don't want this behavior can do three things.

- Disable kdump_msg() at compile time.
- Do not load any module which registers for kdump_msg()
- Implement a /proc tunable which allows controlling this behavior.

- Ok, having said why do we want it, comes the question of how to do it so that it works reasonably well.

- There seems to be on common requirement of kmsg_dump() and kdump() and that is stop other cpus reliably (use nmi if possible). Can we try to share this code between kmsg_dump and crash_kexec(). So something like as follows.

- panic happens
- Do all the activities related to printing panic string and stack dump.
- Stop other cpus.
- This can be probably be done with the equivalent of machine_crash_shutdown() function. In fact this function can probably be broken down in two parts. First part does shutdown_prepare() where all other cpus are shot down and second part can do the actual disabling of LAPIC/IOAPIC and saving cpu registers etc.

if (mutex_trylock(some_shutdown_mutex)) {
/* setp regs, fix vmcoreinfo etc */ crash_kexec_prepare(); machine_shutdown_prepare(); kdump_msg(); crash_kexec_execute()
/* Also call panic_notifier_list here ? */
}

crash_kexec_prepare () {
		crash_setup_regs(&fixed_regs, regs);
		crash_save_vmcoreinfo();
}

crash_kexec_execute() {
			/* Shutdown lapic/ioapic, save this cpu register etc */
			machine_shutdown();
			machine_kexec()
}

So basically we break down machine_shutdown() function in two parts and start sharing common part between kdump_msg(), crash_kexec and possibly panic_notifiers.

Quoted 8 lines

Quoted 6 lines

If kdump is not configured, then after executing kdump_msg() and panic notifiers, we should either be sitting in tight loop with interrupt enabled for somebody to press Ctrl-boot or reboot system upon lapse of panic_timeout().

Eric, does it make sense to you?

kexec on panic doesn't strictly require that we stop other cpus.

Yes but it is desirable.

- We don't want cpus to be scribbling on old memory and possibly on new kernel's memory also in case of corrupted pointer and crash the freshly booted kernel (New kerenl's memory is mapped in old kernel)

- We don't want other cpus to complete panic() and jump to BIOS or lead to some kind of triple fault and reset the system etc.

So that would suggest to be robust, stopping other cpus is required.

On a side note, kdump_msg() hook is present in emergency_reboot() too. So if these paths are not properly designed, then system might not even reboot automatically even if panic_timeout() has been specified.

Quoted 8 lines

What makes sense to me at this point is for someone on the kmsg_dump side to make a strong case that the code actually works in a crash dump scenario. We have lots of experience over the years that says a design like kmsg_dump is attractive but turns out to be a unreliable piece of junk that fails just when you need it. Because developers only test the case when the kernel is happy and because people share code with the regular path drivers, and that code assumes things are happy.

Quoted 4 lines

I forget exactly why but last I looked.
local_irq_disable()
kmsg_dump()
local_irq_enalbe()

I agree that kdump_msg() code should be able to work with interrupts disabled, atleast.

There seem to be two pieces to it. One is generic call which calls all the dumpers and then respective dumpers.

Looking at generic dumping call, I can't think why does it need interrupts to be enabled. There is one spin_lock() and then rcu_read_lock(). That's it.

Regarding mtdoops, it is hard to tell. There is lot of code. But the good thing is that they have introduced a separate write path for panic context. That way atleast one can do special casing in panic path to avoid taking locks and not be dependent on interrupts.

ramoops seems to be simple. It seems to be just memcpy() except do_gettimeofday(). I noticed that in the past you raised concern about usage of do_gettimeofday(), but I am not sure what is the concern here (silly question i guess).

So to me, ramoops seems to be simple memcpy (atleast in principle) and mtdoops has a dedicated path for handling panic context. So atleast it is fixable for possible issues. Generic code seems harmless to me at this point of time.

Quoted 7 lines

Was a recipe for disaster, and you have be at least that good to even have a chance of working in a crash dump scenario.

In part I am puzzled why the kmsg dump doesn't just use the printk interface. Strangely enough printk works in the event of a crash and has been shown to be reliable over the years.

Are you suggesting implementing these things as console driver and register with printk as console? Sounds interesting. I think one of the issues they probably will find that they don't want to log everything. They want to do selective logging. Not sure how would they get this info.

In some cases like emergency_restart(), there are no printk() and they just consider it one of the events to dump the kernel buffer contents.