removal of sync in panic
I have a question regarding the sys_sync() call within the panic function. --------snip---------------
printk(KERN_EMERG "Kernel panic: %s\n",buf);
if (in_interrupt())
printk(KERN_EMERG "In interrupt handler - not syncing\n");
else if (!current->pid)
printk(KERN_EMERG "In idle task - not syncing\n");
else
sys_sync(); <--------------------
bust_spinlocks(0);#ifdef CONFIG_SMP
smp_send_stop();#endif
--------------------------
I have seen panic failing two times lately on an SMP system. The box panic'ed but was running happily on the other cpus. The culprit of this failure is the fact, that these panics have been caused by a block device or a filesystem (e.g. using errors=panic). In these cases the likelihood of a failure/hang of sys_sync() is high. This is exactly what happened in both cases I have seen. Meanwhile the other cpus are happily continuing destroying data as the kernel has a severe problem but its not aware of that as smp_send_stop happens after sys_sync.
I can imagine several changes but I am not sure if this is a problem which must be fixed and which fix is the best.
Here are my alternatives:
1. remove sys_sync completely: syslogd and klogd use fsync. No need to help them. Furthermore we have a severe problem which is worth a panic, so we better dont do any I/O.
2. move smp_send_stop before sys_sync. This at least prevents other cpus of doing harm if sys_sync hangs. Here I am not sure if this is really working.
3. Add an
if (doing_io())
printk(KERN_EMERG "In I/O routine - not syncing\n");check like in_interrupt check. Unfortunately I have no clue how this can be achieved and it looks quite ugly.
Thanks for any ideas and clarifications
cheers
Christian
Quoted 7 lines · Christian Borntraeger
I can imagine several changes but I am not sure if this is a problem which must be fixed and which fix is the best.
Here are my alternatives:1. remove
sys_synccompletely: syslogd and klogd use fsync. No need to help them. Furthermore we have a severe problem which is worth a panic, so we better dont do any I/O.
I've seen exactly the behaviour you describe and would be inclined to go for this option too.
Quoted 5 lines
3. Add an
if (doing_io()) printk(KERN_EMERG "In I/O routine - not syncing\n");check like
in_interruptcheck. Unfortunately I have no clue how this can be achieved and it looks quite ugly.
This would also work of course, but as you point out, it's more complex.
Sincerely,
Lars Marowsky-Brée <[email protected]>
Signature · 3 lines
High Availability & Clustering \ ever tried. ever failed. no matter. SUSE Labs, Research and Development | try again. fail again. fail better. SUSE LINUX AG - A Novell company \ -- Samuel Beckett
PATCH 1/1 was: [RFC] removal of sync in panic
Quoted 6 lines · Lars Marowsky-Bree
Quoted 3 lines
1. remove
sys_synccompletely: syslogd and klogd use fsync. No need to help them. Furthermore we have a severe problem which is worth a panic, so we better dont do any I/O.I've seen exactly the behaviour you describe and would be inclined to go for this option too.
As this problem definitely exists, here is a patch.
kernel/panic.c 2004-07-14 19:37:02.000000000 +0200 renamed from kernel/panic.c 2004-06-16 07:20:04.000000000 +0200 +1 -7
@@ -59,13 +59,7 @@ NORET_TYPE void panic(const char * fmt, va_start(args, fmt); vsnprintf(buf, sizeof(buf), fmt, args); va_end(args); - printk(KERN_EMERG "Kernel panic: %s\n",buf); - if (in_interrupt()) - printk(KERN_EMERG "In interrupt handler - not syncing\n"); - else if (!current->pid) - printk(KERN_EMERG "In idle task - not syncing\n"); - else - sys_sync(); + printk(KERN_EMERG "Kernel panic - not syncing: %s\n",buf); bust_spinlocks(0); #ifdef CONFIG_SMP
Re: [PATCH] was: [RFC] removal of sync in panic
Quoted 18 lines · Christian Borntraeger
Quoted 6 lines · Lars Marowsky-Bree
Quoted 3 lines
1. remove
sys_synccompletely: syslogd and klogd use fsync. No need to help them. Furthermore we have a severe problem which is worth a panic, so we better dont do any I/O.I've seen exactly the behaviour you describe and would be inclined to go for this option too.
As this problem definitely exists, here is a patch.
kernel/panic.c 2004-07-14 19:37:02.000000000 +0200 renamed from kernel/panic.c 2004-06-16 07:20:04.000000000 +0200 +0
-1@@ -59,13 +59,7 @@ NORET_TYPE void panic(const char * fmt, va_start(args, fmt); vsnprintf(buf, sizeof(buf), fmt, args); va_end(args); - printk(KERN_EMERG "Kernel panic: %s\n",buf);
kernel/panic.c 2004-07-14 19:37:02.000000000 +0200 NORET_TYPE void panic(const char * fmt, · line 65
Quoted 10 lines
kernel/panic.c 2004-07-14 19:37:02.000000000 +0200 renamed from kernel/panic.c 2004-06-16 07:20:04.000000000 +0200 +1
-6- if (in_interrupt()) - printk(KERN_EMERG "In interrupt handler - not syncing\n"); - else if (!current->pid) - printk(KERN_EMERG "In idle task - not syncing\n"); - else - sys_sync(); + printk(KERN_EMERG "Kernel panic - not syncing: %s\n",buf); bust_spinlocks(0); #ifdef CONFIG_SMP
I agree with the patch in principle, but I'd be interested in what observed problem motivated it?
Re: [PATCH] was: [RFC] removal of sync in panic
Quoted 2 lines · Andrew Morton
I agree with the patch in principle, but I'd be interested in what observed problem motivated it?
see the first posting.
-----------snip--------------
I have seen panic failing two times lately on an SMP system. The box panic'ed but was running happily on the other cpus. The culprit of this failure is the fact, that these panics have been caused by a block device or a filesystem (e.g. using errors=panic). In these cases the likelihood of a failure/hang of sys_sync() is high. This is exactly what happened in both cases I have seen. Meanwhile the other cpus are happily continuing destroying data as the kernel has a severe problem but its not aware of that as smp_send_stop happens after sys_sync.
I can imagine several changes but I am not sure if this is a problem which must be fixed and which fix is the best.
Here are my alternatives:
1. remove sys_sync completely: syslogd and klogd use fsync. No need to help them. Furthermore we have a severe problem which is worth a panic, so we better dont do any I/O.
2. move smp_send_stop before sys_sync. This at least prevents other cpus of doing harm if sys_sync hangs. Here I am not sure if this is really working.
3. Add an
if (doing_io())
printk(KERN_EMERG "In I/O routine - not syncing\n");check like in_interrupt check. Unfortunately I have no clue how this can be achieved and it looks quite ugly.
---------------------
Re: [PATCH] was: [RFC] removal of sync in panic
Quoted 8 lines · Christian Borntraeger
I have seen panic failing two times lately on an SMP system. The box panic'ed but was running happily on the other cpus. The culprit of this failure is the fact, that these panics have been caused by a block device or a filesystem (e.g. using errors=panic). In these cases the likelihood of a failure/hang of
sys_sync()is high. This is exactly what happened in both cases I have seen. Meanwhile the other cpus are happily continuing destroying data as the kernel has a severe problem but its not aware of that assmp_send_stophappens aftersys_sync.
I've seen SMP boxen run interrupt handlers for ages after panicking, but I never thought much of it.
-- wli
Re: [PATCH] was: [RFC] removal of sync in panic
In reply to Christian BorntraegerYes, I've seen this multiple times.
I also agree that it seems a sensible patch. I have one dumb question. Given that we're panicing and we know things are "bad", is there any reason not to call smp_send_stop() as early as possible, rather than as the last thing which we currently do? As you say, the other cpus are happily continuing, potentially destroying data, and it seems that stopping this as quickly as possible would be desirable.
Tim
Quoted 36 lines · Christian Borntraeger
Quoted 2 lines · Andrew Morton
I agree with the patch in principle, but I'd be interested in what observed problem motivated it?
see the first posting.
-----------snip--------------
I have seen panic failing two times lately on an SMP system. The box panic'ed but was running happily on the other cpus. The culprit of this failure is the fact, that these panics have been caused by a block device or a filesystem (e.g. using errors=panic). In these cases the likelihood of a failure/hang ofsys_sync()is high. This is exactly what happened in both cases I have seen. Meanwhile the other cpus are happily continuing destroying data as the kernel has a severe problem but its not aware of that assmp_send_stophappens aftersys_sync.I can imagine several changes but I am not sure if this is a problem which must be fixed and which fix is the best.
Here are my alternatives:1. remove
sys_synccompletely: syslogd and klogd use fsync. No need to help them. Furthermore we have a severe problem which is worth a panic, so we better dont do any I/O.
2. movesmp_send_stopbeforesys_sync. This at least prevents other cpus of doing harm ifsys_synchangs. Here I am not sure if this is really working.
3. Add anif (doing_io()) printk(KERN_EMERG "In I/O routine - not syncing\n");check like
in_interruptcheck. Unfortunately I have no clue how this can be achieved and it looks quite ugly.
---------------------
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 2 lines
Tim Wright <[email protected]> Splhi
Re: [PATCH] was: [RFC] removal of sync in panic
Quoted 7 lines · Tim Wright
Yes, I've seen this multiple times.
I also agree that it seems a sensible patch. I have one dumb question. Given that we're panicing and we know things are "bad", is there any reason not to callsmp_send_stop()as early as possible, rather than as the last thing which we currently do? As you say, the other cpus are happily continuing, potentially destroying data, and it seems that stopping this as quickly as possible would be desirable.
That suggestion was my number 2 is my first mail :-)
On the other hand, if we remove the sync stuff, smp_send_stop is called quite early. Only a printf is called before smp_send_stop().