subsystem crashes reboot system?
Hi,
I have a feature request, I'm willing to hack away at it myself, but I want to know if there's any way of doing what I want to, or if there's a good technical reason why it would be impossible.
As I mentioned earlier, we had an ext3 subsystem crash, which a helpful person was nice enough to tell me that upgrading the kernel would fix. All well and good. But this crash left the system in a semi-functional state. The networking stack was up and running, the kernel was running, but the filesystem was not functional and because of this the kernel was in a nearly unusable state. Because the system was pingable, most tcp-stack level detectors would not have been able to tell that something serious was wrong. The machine (our main production machine that serves millions of hits a week) was down for three hours.
Since this was an assertion that failed, one would think that bringing the system down automatically in an orderly - then, if that fails, disorderly - fashion would be possible. In particular, I would like for it to behave similar as with the panic sysctl. If a subsystem crashes, reboot the machine, because the system is essentially worthless in that state. I realize that this behavior isn't required for everyone, so a sysctl (panic_on_subsys_crash maybe) would be sufficient.
Since the machine was in a semi-usable state, one might ask why we just didn't have an automated process in place. Two reasons: a subsystem crashing happens rarely enough that I didn't see any reason to put the effort into it until now, and when the system is in a state like that it is impossible to tell what will work and what will not. For example, when we did the three finger salute, the system would not go down all the way because one of the user space programs made an io call to the crashed filesystem.
In order of helpfulness, please tell me (only one of the following is more than enough):
- whether I can do this using the existing sysctl mechanism
- whether there is a patch available (or coming available) to do this
- whether there is a technical reason for me not to do this
- what would be a good place in the code to begin applying a patch.
Please CC me with any replies as I am not on the list.
Thanks.
--Russell
Quoted 7 lines
good. But this crash left the system in a semi-functional state. The networking stack was up and running, the kernel was running, but the filesystem was not functional and because of this the kernel was in a nearly unusable state. Because the system was pingable, most tcp-stack level detectors would not have been able to tell that something serious was wrong. The machine (our main production machine that serves millions of hits a week) was down for three hours.
Isn't this what watchdog is for? I think even the software watchdog would catch this, then you can panic and reboot.
M
Quoted 2 lines · Mitch Adair
Isn't this what watchdog is for? I think even the software watchdog would catch this, then you can panic and reboot.
hm, I don't think, that watchdog will catch this, because the userspace-watchdog daemon will still be running properly in a crash case
(or did I understand something wrong?)
Regards Michael Buesch.
Signature · 3 lines
------------- My homepage: http://www.8ung.at/tuxsoft fighting for peace is like fu**ing for virginity
Quoted 6 lines
Quoted 2 lines
Isn't this what watchdog is for? I think even the software watchdog would catch this, then you can panic and reboot.
hm, I don't think, that watchdog will catch this, because the userspace-watchdog daemon will still be running properly in a crash case
(or did I understand something wrong?)
But it wouldn't be able to write to the filesystem so it would trigger if I believe.
M
Quoted 8 lines · Michael Buesch
Quoted 2 lines · Mitch Adair
Isn't this what watchdog is for? I think even the software watchdog would catch this, then you can panic and reboot.
hm, I don't think, that watchdog will catch this, because the userspace-watchdog daemon will still be running properly in a crash case (or did I understand something wrong?)
Unless you configure it to stat your filesystems, like in:
watchdog-device = /dev/misc/watchdog realtime = yes priority = 99 admin = file = / file = /var file = /usr
...
Phil.
Quoted 6 lines · you
Quoted 3 lines
hm, I don't think, that watchdog will catch this, because the userspace-watchdog daemon will still be running properly in a crash case (or did I understand something wrong?)
Unless you configure it to stat your filesystems, like in:
Quoted 8 lines
watchdog-device = /dev/misc/watchdog realtime = yes priority = 99 admin = file = / file = /var file = /usr...
Yes that's true. I didn't remember this option. With this, watchdog would be a solution of russel's problem, without writing some kernel-error-handling for it.
Regards Michael Buesch.
Signature · 3 lines
------------- My homepage: http://www.8ung.at/tuxsoft fighting for peace is like fu**ing for virginity
Quoted 4 lines · Russell Miller
Since this was an assertion that failed, one would think that bringing the system down automatically in an orderly - then, if that fails, disorderly - fashion would be possible.
The way to handle this is to make arch/i386/kernel/traps.c:die() optionally call panic() rather than do_exit().
It makes sense. It does mean that we now have zero chance of the diagnostic info making it to the system logs.
Any chance of making the dying thread sleep just long enough for syslogd to write it out to the file, then panic? Since it's an assertion, we have a little more leeway then in a page fault OOPS, for example.
--Russell
Quoted 10 lines · Andrew Morton
Quoted 3 lines · Russell Miller
Since this was an assertion that failed, one would think that bringing the system down automatically in an orderly - then, if that fails, disorderly - fashion would be possible.
The way to handle this is to make arch/i386/kernel/traps.c:die() optionally call
panic()rather thando_exit().It makes sense. It does mean that we now have zero chance of the diagnostic info making it to the system logs.
Quoted 5 lines · Russell Miller
Any chance of making the dying thread sleep just long enough for syslogd to write it out to the file, then panic? Since it's an assertion, we have a little more leeway then in a page fault OOPS, for example.
Yes, that would probably be OK. It won't make anything worse than it already is.
hm, the kernel used to panic if schedule() was called from in_interrupt(),
but that seems to have been taken out. It's easy enough (and free) to
put back in.
Quoted 6 lines · Andrew Morton
Yes, that would probably be OK. It won't make anything worse than it already is.
hm, the kernel used to panic if
schedule()was called fromin_interrupt(),
but that seems to have been taken out. It's easy enough (and free) to
put back in.
I'll see about writing a patch if you would like. Sounds like a good thing to get my feet wet with.
--Russell