add new notifier function ,take3
Subsystems (2)
cover add new notifier function ,take3
Hi,
A big thanks to everybody who read and replied to previous version.
changelog take2 -> take3
- Rebased 2.6.25-rc8-mm1
- comment updated
- renamed the notifiner name "tunable_notifier" to "tunable_atomic_notifier"
- fixed typo
- move control files debugfs to /sys/kernel
These patches add new notifier function and implement it to panic_notifier_list. We used the hardcoded notifier chain so far, but it was not flexible. New notifier is very flexible, because user can change a list of order by control files.
Example)
# cd /sys/kernel/notifiers/ # ls panic_notifier_list # cd panic_notifier_list/ # ls ipmi_msghandler ipmi_wdog # insmod notifier_test.ko # ls ipmi_msghandler ipmi_wdog notifier_test1 notifier_test2 # cat */priority 200 150 500 1000 Kernel panic - not syncing: Panic by panic_module. __tunable_atomic_notifier_call_chain enter notifier_test: notifier_test_panic2() is called. notifier_test: notifier_test_panic() is called. msg_handler:panic_event was called. ipmi_wdog:wdog_panic_handler was called.
.....(reboot)
# cd /sys/kernel/notifiers/panic_notifier_list/ # ls ipmi_msghandler ipmi_wdog notifier_test1 notifier_test2 # cat */priority 200 150 500 1000 # echo 10000 > ipmi_msghandler/priority # echo 5000 > ipmi_wdog/priority # echo 3000 > notifier_test1/priority # echo 1500 > notifier_test2/priority # cat */priority 10000 5000 3000 1500 Kernel panic - not syncing: Panic by panic_module. __tunable_atomic_notifier_call_chain enter msg_handler:panic_event was called. ipmi_wdog:wdog_panic_handler was called. notifier_test: notifier_test_panic() is called. notifier_test: notifier_test_panic2() is called.
--
Takenori Nagano <[email protected]>
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 56 lines · On Fri, 11 Apr 2008 16:53:47 +0900 Takenori Nagano <[email protected]>…
Hi,
A big thanks to everybody who read and replied to previous version.
changelog take2 -> take3
- Rebased 2.6.25-rc8-mm1
- comment updated
- renamed the notifiner name "tunable_notifier" to "tunable_atomic_notifier"
- fixed typo
- move control files debugfs to/sys/kernelThese patches add new notifier function and implement it to
panic_notifier_list. We used the hardcoded notifier chain so far, but it was not flexible. New notifier is very flexible, because user can change a list of order by control files.Example)
# cd /sys/kernel/notifiers/ # ls panic_notifier_list # cd panic_notifier_list/ # ls ipmi_msghandler ipmi_wdog # insmod notifier_test.ko # ls ipmi_msghandler ipmi_wdog notifier_test1 notifier_test2 # cat */priority 200 150 500 1000 Kernel panic - not syncing: Panic by panic_module. __tunable_atomic_notifier_call_chain enter notifier_test: notifier_test_panic2() is called. notifier_test: notifier_test_panic() is called. msg_handler:panic_event was called. ipmi_wdog:wdog_panic_handler was called......(reboot)
# cd /sys/kernel/notifiers/panic_notifier_list/ # ls ipmi_msghandler ipmi_wdog notifier_test1 notifier_test2 # cat */priority 200 150 500 1000 # echo 10000 > ipmi_msghandler/priority # echo 5000 > ipmi_wdog/priority # echo 3000 > notifier_test1/priority # echo 1500 > notifier_test2/priority # cat */priority 10000 5000
Quoted 8 lines
3000 1500 Kernel panic - not syncing: Panic by panic_module. __tunable_atomic_notifier_call_chain enter msg_handler:panic_event was called. ipmi_wdog:wdog_panic_handler was called. notifier_test: notifier_test_panic() is called. notifier_test: notifier_test_panic2() is called.
OK. But I don't see anywhere in here the most important piece of
information: why do we need this feature in Linux?
What are the use-cases? What is the value? etc.
Often I can guess (but I like the originator to remove the guesswork). In this case I'm stumped - I can't see any reason why anyone would want this.
Awaiting enlightenment ;)
Thanks.
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
On Fri, Apr 11, 2008 at 09:07:51PM -0700, Andrew Morton wrote:
[..]
Quoted 7 lines
Quoted 6 lines
Kernel panic - not syncing: Panic by panic_module.
__tunable_atomic_notifier_call_chainenter
msg_handler:panic_event was called.
ipmi_wdog:wdog_panic_handler was called.notifier_test:notifier_test_panic()is called.notifier_test:notifier_test_panic2()is called.
Quoted 8 lines
OK. But I don't see anywhere in here the most important piece of
information: why do we need this feature in Linux?What are the use-cases? What is the value? etc.
Often I can guess (but I like the originator to remove the guesswork). In this case I'm stumped - I can't see any reason why anyone would want this.
Hi Andrew,
To begin with, he wants kdb, kgdb etc to co-exist with kdump. He wants to put all the RAS tools (who are interested in panic event) on a list and export it to user space and let user decide in what order do the tool get executed at panic time (based on priority).
This brings in little bit reliability concerns for kdump due to notifier code being run after panic.
I think people want to use this infrastrutucure beyond RAS tools. I remember somebody wanting to send a message to remote node after a panic (before kdump kicks in) so that remote node can initiate failover etc.
Ideally, doing any operation after panic is not safe and one should avoid such things and any action required should be done in next kernel (like sending messages to remote nodes etc). Having said that, it makes the job harder as one needs to pass all the required data to second kernel.
So it will not left to user whether he should execute the code after panic in first kernel or create required bits to execute code in second kernel. Things should be more reliable in second kernel.
I am not very sure how paranoid one should be about this additional bit of notifier code being executed after panic. Probably we can take this in to make user's life easier.
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 26 lines · Vivek Goyal
On Fri, Apr 11, 2008 at 09:07:51PM -0700, Andrew Morton wrote:
[..]
Quoted 7 lines
Quoted 6 lines
Kernel panic - not syncing: Panic by panic_module.
__tunable_atomic_notifier_call_chainenter
msg_handler:panic_event was called.
ipmi_wdog:wdog_panic_handler was called.notifier_test:notifier_test_panic()is called.notifier_test:notifier_test_panic2()is called.Quoted 8 lines
OK. But I don't see anywhere in here the most important piece of
information: why do we need this feature in Linux?What are the use-cases? What is the value? etc.
Often I can guess (but I like the originator to remove the guesswork). In this case I'm stumped - I can't see any reason why anyone would want this.
Hi Andrew,
To begin with, he wants kdb, kgdb etc to co-exist with kdump. He wants to put all the RAS tools (who are interested in panic event) on a list and export it to user space and let user decide in what order do the tool get executed at panic time (based on priority).
Quoted 8 lines
This brings in little bit reliability concerns for kdump due to notifier code being run after panic.
I think people want to use this infrastrutucure beyond RAS tools. I remember somebody wanting to send a message to remote node after a panic (before kdump kicks in) so that remote node can initiate failover etc.
I know it doesn't particularly relate to this patch, but FWIW, for cases like failover, I've inserted infrastrucutre in the userspace part of kdump for Fedora/RHEL to support this sort of thing. We can run arbitrary scripts righte before and after a capture so that notifications can be sent to remote nodes in a much safer fashion than using the notifier chain after a panic. Neil
Signature · 13 lines
/*************************************************** *Neil Horman *Software Engineer *Red Hat, Inc. *[email protected] *gpg keyid: 1024D / 0x92A74FA1 *http://pgp.mit.edu ***************************************************/ _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Quoted 6 lines · Neil Horman
I know it doesn't particularly relate to this patch, but FWIW, for cases like failover, I've inserted infrastrucutre in the userspace part of kdump for Fedora/RHEL to support this sort of thing. We can run arbitrary scripts righte before and after a capture so that notifications can be sent to remote nodes in a much safer fashion than using the notifier chain after a panic.
But that doesn't help if you want to use a debugger (KDB, KGDB).
Bernhard
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 35 lines · Neil Horman
Quoted 26 lines · Vivek Goyal
On Fri, Apr 11, 2008 at 09:07:51PM -0700, Andrew Morton wrote:
[..]
Quoted 7 lines
Quoted 6 lines
Kernel panic - not syncing: Panic by panic_module.
__tunable_atomic_notifier_call_chainenter
msg_handler:panic_event was called.
ipmi_wdog:wdog_panic_handler was called.notifier_test:notifier_test_panic()is called.notifier_test:notifier_test_panic2()is called.Quoted 8 lines
OK. But I don't see anywhere in here the most important piece of
information: why do we need this feature in Linux?What are the use-cases? What is the value? etc.
Often I can guess (but I like the originator to remove the guesswork). In this case I'm stumped - I can't see any reason why anyone would want this.
Hi Andrew,
To begin with, he wants kdb, kgdb etc to co-exist with kdump. He wants to put all the RAS tools (who are interested in panic event) on a list and export it to user space and let user decide in what order do the tool get executed at panic time (based on priority).
Quoted 8 lines
This brings in little bit reliability concerns for kdump due to notifier code being run after panic.
I think people want to use this infrastrutucure beyond RAS tools. I remember somebody wanting to send a message to remote node after a panic (before kdump kicks in) so that remote node can initiate failover etc.
Quoted 7 lines
I know it doesn't particularly relate to this patch, but FWIW, for cases like failover, I've inserted infrastrucutre in the userspace part of kdump for Fedora/RHEL to support this sort of thing. We can run arbitrary scripts righte before and after a capture so that notifications can be sent to remote nodes in a much safer fashion than using the notifier chain after a panic. Neil
That's great. I did not know about these. So user can write custom scripts/binaries which can be packed into kdump initrd and executed either before or after dump capture? Any idea, if somebody has started using it already?
If that's the case then only other serious user at this point of time is kernel debugger (kdb, kgdb), which needs to run before kdump, in case of panic. And Eric suggested for those cases debugger can just insert a break point at panic(), instead of introducing the tunable notifier list infrastructure.
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 44 lines · Vivek Goyal
Quoted 35 lines · Neil Horman
Quoted 26 lines · Vivek Goyal
On Fri, Apr 11, 2008 at 09:07:51PM -0700, Andrew Morton wrote:
[..]
Quoted 7 lines
Quoted 6 lines
Kernel panic - not syncing: Panic by panic_module.
__tunable_atomic_notifier_call_chainenter
msg_handler:panic_event was called.
ipmi_wdog:wdog_panic_handler was called.notifier_test:notifier_test_panic()is called.notifier_test:notifier_test_panic2()is called.Quoted 8 lines
OK. But I don't see anywhere in here the most important piece of
information: why do we need this feature in Linux?What are the use-cases? What is the value? etc.
Often I can guess (but I like the originator to remove the guesswork). In this case I'm stumped - I can't see any reason why anyone would want this.
Hi Andrew,
To begin with, he wants kdb, kgdb etc to co-exist with kdump. He wants to put all the RAS tools (who are interested in panic event) on a list and export it to user space and let user decide in what order do the tool get executed at panic time (based on priority).
Quoted 8 lines
This brings in little bit reliability concerns for kdump due to notifier code being run after panic.
I think people want to use this infrastrutucure beyond RAS tools. I remember somebody wanting to send a message to remote node after a panic (before kdump kicks in) so that remote node can initiate failover etc.
Quoted 7 lines
I know it doesn't particularly relate to this patch, but FWIW, for cases like failover, I've inserted infrastrucutre in the userspace part of kdump for Fedora/RHEL to support this sort of thing. We can run arbitrary scripts righte before and after a capture so that notifications can be sent to remote nodes in a much safer fashion than using the notifier chain after a panic. Neil
Quoted 5 lines
That's great. I did not know about these. So user can write custom
scripts/binarieswhich can be packed into kdump initrd and executed either before or after dump capture? Any idea, if somebody has started using it already?
Thats exactly right. I'm not sure if there is any serious use as of yet, but I've had some interrogatories about it. Specific cases that I recall include:
1) A set of users in japan that are using the pre-dump script to block execution until a scsi controller detects all its drives (it apparently takes up to three minues to scan its bus)
2) I think some people using clustering services were using the pre-script to notify cluster peers of the failure to avoid power fencing while a node completed the crash dump
3) A national lab had an interest in using the pre script to send an email to an administrative address to log the failure in a cluster
Neil
Quoted 8 lines
If that's the case then only other serious user at this point of time is kernel debugger (kdb, kgdb), which needs to run before kdump, in case of panic. And Eric suggested for those cases debugger can just insert a break point at
panic(), instead of introducing the tunable notifier list infrastructure.Thanks
Vivek
Signature · 13 lines
/*************************************************** *Neil Horman *Software Engineer *Red Hat, Inc. *[email protected] *gpg keyid: 1024D / 0x92A74FA1 *http://pgp.mit.edu ***************************************************/ _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Quoted 57 lines · On Mon, 14 Apr 2008 12:01:46 -0400 Neil Horman <[email protected]> wrote:
Quoted 44 lines · Vivek Goyal
Quoted 35 lines · Neil Horman
Quoted 26 lines · Vivek Goyal
On Fri, Apr 11, 2008 at 09:07:51PM -0700, Andrew Morton wrote:
[..]
Quoted 7 lines
Quoted 6 lines
Kernel panic - not syncing: Panic by panic_module.
__tunable_atomic_notifier_call_chainenter
msg_handler:panic_event was called.
ipmi_wdog:wdog_panic_handler was called.notifier_test:notifier_test_panic()is called.notifier_test:notifier_test_panic2()is called.Quoted 8 lines
OK. But I don't see anywhere in here the most important piece of
information: why do we need this feature in Linux?What are the use-cases? What is the value? etc.
Often I can guess (but I like the originator to remove the guesswork). In this case I'm stumped - I can't see any reason why anyone would want this.
Hi Andrew,
To begin with, he wants kdb, kgdb etc to co-exist with kdump. He wants to put all the RAS tools (who are interested in panic event) on a list and export it to user space and let user decide in what order do the tool get executed at panic time (based on priority).
Quoted 8 lines
This brings in little bit reliability concerns for kdump due to notifier code being run after panic.
I think people want to use this infrastrutucure beyond RAS tools. I remember somebody wanting to send a message to remote node after a panic (before kdump kicks in) so that remote node can initiate failover etc.
Quoted 7 lines
I know it doesn't particularly relate to this patch, but FWIW, for cases like failover, I've inserted infrastrucutre in the userspace part of kdump for Fedora/RHEL to support this sort of thing. We can run arbitrary scripts righte before and after a capture so that notifications can be sent to remote nodes in a much safer fashion than using the notifier chain after a panic. Neil
Quoted 5 lines
That's great. I did not know about these. So user can write custom
scripts/binarieswhich can be packed into kdump initrd and executed either before or after dump capture? Any idea, if somebody has started using it already?Thats exactly right. I'm not sure if there is any serious use as of yet, but I've had some interrogatories about it. Specific cases that I recall include:
1) A set of users in japan that are using the pre-dump script to block execution until a scsi controller detects all its drives (it apparently takes up to three minues to scan its bus)
Quoted 7 lines
2) I think some people using clustering services were using the pre-script to notify cluster peers of the failure to avoid power fencing while a node completed the crash dump
3) A national lab had an interest in using the pre script to send an email to an administrative address to log the failure in a cluster
OK, thanks.
I think I'll duck the patch for now as it seems that a littlee more thought and coordination is neeed.
Plus it appears that the only users of this infrastructure are provided via presently-out-of-tree patches, so people who are already patching and building their own kernels can easily add this other patch as well, for now.
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 65 lines · Andrew Morton
Quoted 54 lines · On Mon, 14 Apr 2008 12:01:46 -0400 Neil Horman <[email protected]> wrote:
Quoted 41 lines · Vivek Goyal
Quoted 33 lines · Neil Horman
Quoted 24 lines · Vivek Goyal
On Fri, Apr 11, 2008 at 09:07:51PM -0700, Andrew Morton wrote:
[..]
Quoted 6 lines
Quoted 6 lines
Kernel panic - not syncing: Panic by panic_module.
__tunable_atomic_notifier_call_chainenter
msg_handler:panic_event was called.
ipmi_wdog:wdog_panic_handler was called.notifier_test:notifier_test_panic()is called.notifier_test:notifier_test_panic2()is called.Quoted 8 lines
OK. But I don't see anywhere in here the most important piece of
information: why do we need this feature in Linux?What are the use-cases? What is the value? etc.
Often I can guess (but I like the originator to remove the guesswork). In this case I'm stumped - I can't see any reason why anyone would want this.
Hi Andrew,
To begin with, he wants kdb, kgdb etc to co-exist with kdump. He wants to put all the RAS tools (who are interested in panic event) on a list and export it to user space and let user decide in what order do the tool get executed at panic time (based on priority).
Quoted 8 lines
This brings in little bit reliability concerns for kdump due to notifier code being run after panic.
I think people want to use this infrastrutucure beyond RAS tools. I remember somebody wanting to send a message to remote node after a panic (before kdump kicks in) so that remote node can initiate failover etc.
Quoted 7 lines
I know it doesn't particularly relate to this patch, but FWIW, for cases like failover, I've inserted infrastrucutre in the userspace part of kdump for Fedora/RHEL to support this sort of thing. We can run arbitrary scripts righte before and after a capture so that notifications can be sent to remote nodes in a much safer fashion than using the notifier chain after a panic. Neil
Quoted 5 lines
That's great. I did not know about these. So user can write custom
scripts/binarieswhich can be packed into kdump initrd and executed either before or after dump capture? Any idea, if somebody has started using it already?Thats exactly right. I'm not sure if there is any serious use as of yet, but I've had some interrogatories about it. Specific cases that I recall include:
1) A set of users in japan that are using the pre-dump script to block execution until a scsi controller detects all its drives (it apparently takes up to three minues to scan its bus)
Quoted 7 lines
2) I think some people using clustering services were using the pre-script to notify cluster peers of the failure to avoid power fencing while a node completed the crash dump
3) A national lab had an interest in using the pre script to send an email to an administrative address to log the failure in a cluster
Quoted 10 lines
OK, thanks.
I think I'll duck the patch for now as it seems that a littlee more thought and coordination is neeed.
Plus it appears that the only users of this infrastructure are provided via presently-out-of-tree patches, so people who are already patching and building their own kernels can easily add this other patch as well, for now.
Hi,
The one of the reason why I want this functionality is managing RAS tool behavior for postmotem actions, initially from kdb invocation. (I used kdb for debugging and crash analysis very useful in lkcd days, but it is "want" and it is not "must" today ;-))
The other postmotem action is disabling hardware watchdog. Watch dog handler would stop keepalive heartbeat when system panics and we must disable hardware watchdog as soon as possible, since 2nd kernel startup takes some time (10 or 100? secs) and there may be miss-firing window. But currently we have no chance to do anything before crash_exec().
And thinking about a clustering software. If the system encounter the panic, system must notify standby node. But... :-(
I am interested in pre-dump scripts Neil mentioned. I think it can resolve some of our requirements. I will try it.
For quick invocation of kdump, I partially agree with the idea of "kdump should be invoked as soon as system panic, since we can not trust broken kernels", but we would like to have some choise what to do on panic (and if notifier is controllable by my patch, you can still call kdump first)
Anyway, completely broken kernel can not call kdump or any other mechanism ;-P and I feel it is somewhat matter of degree.
Thanks,
Takenori
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 6 lines · Takenori Nagano
Hi,
The one of the reason why I want this functionality is managing RAS tool behavior for postmotem actions, initially from kdb invocation. (I used kdb for debugging and crash analysis very useful in lkcd days, but it is "want" and it is not "must" today ;-))
Ok. I have not heard any reason here why a break point at panic or inside of panic is not useful.
Quoted 6 lines
The other postmotem action is disabling hardware watchdog. Watch dog handler would stop keepalive heartbeat when system panics and we must disable hardware watchdog as soon as possible, since 2nd kernel startup takes some time (10 or 100? secs) and there may be miss-firing window. But currently we have no chance to do anything before
crash_exec().
The transition time from one kernel to the next should be under 1 sec. After that you are talking time for the drivers to initialize. Although sha256 over the kdump kernel and it's ramdisk may slow things down a little more for lots of data.
If the concern is of petting a watchdog to keep the system from rebooting getting the kernel to initialize the watchdogs quickly appears to be the correct answer.
Quoted 2 lines
And thinking about a clustering software. If the system encounter the panic, system must notify standby node. But... :-(
If the concern is to notify another system of the crash quickly. I see no reason why very early in the second kernel or perhaps even in the purgatory code in kexec we can not do this. If the code is to hairy to do there then the code is likely to be too hairy to do reliably when the system panics.
Quoted 11 lines
I am interested in pre-dump scripts Neil mentioned. I think it can resolve some of our requirements. I will try it.
For quick invocation of kdump, I partially agree with the idea of "kdump should be invoked as soon as system panic, since we can not trust broken kernels", but we would like to have some choise what to do on panic (and if notifier is controllable by my patch, you can still call kdump first)
Anyway, completely broken kernel can not call kdump or any other mechanism ;-P and I feel it is somewhat matter of degree.
Yes. Completely broken kernels may not recognize they have a problem.
It is the design goal of the kexec on panic path to work with as much of kernel broken as possible.
Additionally reviewing and testing that code is extremely difficult, because it is the one piece of code in the kernel where debugging tools are not available. Putting random tunable code on that path hugely reduces it's maintainability.
Further for all of the cases I have seen there is only one correct action to take, things do not need to be tunable. So a generally tunable interface appears to be a design mistake.
Eric
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec