Skip to contentSkip to conversation

2.6.13.2 crash on shutdown on SMP machine

DISCUSSION Clemens Koller 3 participants 14 messages Sep 28, 2005 – Dec 8, 2005

lore
Share & tools
lore
#0 lore raw

Hi!

Last night, right before thinking about going to bed, my newly installed old SMP machine crashed after a #shutdown -h now as shown below:

linux-2.6.13.2
old Tyan Tomcat Board, Dual Processor, 2xPentium MMX 200MHz SMP enabled, preemption enabled..

[...]
Shutdown: hda
Power down.
Badness in send_IPI_mask_bitmask at arch/i386/kernel/smp.c:168

c010fdd5    send_IPI_mask_bitmask+0x65/0x70
c0110236    smp_send_reschedule+0x16/0x20
c01188d6    __migrate_task+0xb6/0xc0
c01189ad    migration_thread+0xcd/0x120
c01188e0    migration_thread+0x0/0x120
c012ef43    kthread+0x93/0xc0
c012eeb0    kthread+0x0/0x120
c010104d    kernel_thread_helper+0x5/0x18

The board cannot do any acpi and auto-powerdown thing. It should just stop after "Power down."

I can try the latest git or 2.6.14-rc2 tonight and get you some more info (.config) when I am back home... BTW what is IPI? Any ideas? What do you need to track down this issue?

Thanks,

Signature · 11 lines
Clemens Koller
_______________________________
R&D Imaging Devices
Anagramm GmbH
Rupert-Mayer-Str. 45/1
81379 Muenchen
Germany

http://www.anagramm.de
Phone: +49-89-741518-50
Fax: +49-89-741518-19
#1 lore raw
Quoted 8 lines · Clemens Koller

Last night, right before thinking about going to bed, my newly installed old SMP machine crashed after a #shutdown -h now as shown below:

linux-2.6.13.2
old Tyan Tomcat Board, Dual Processor, 2xPentium MMX 200MHz SMP enabled, preemption enabled..

Quoted 6 lines

[...]
Shutdown: hda
Power down.
Badness in send_IPI_mask_bitmask at arch/i386/kernel/smp.c:168
c010fdd5 send_IPI_mask_bitmask+0x65/0x70
c0110236 smp_send_reschedule+0x16/0x20

We've seen this one before, how reproducible is it for you? Could you also please test a 2.6.14-rc -mm kernel?

Thanks,
Zwane

#2 lore raw

Hello, Zwane!

Quoted 10 lines · Zwane Mwaikambo
Quoted 8 lines · Clemens Koller

Last night, right before thinking about going to bed, my newly installed old SMP machine crashed after a #shutdown -h now as shown below:

linux-2.6.13.2
old Tyan Tomcat Board, Dual Processor, 2xPentium MMX 200MHz SMP enabled, preemption enabled..

Quoted 8 lines
Quoted 6 lines

[...]
Shutdown: hda
Power down.
Badness in send_IPI_mask_bitmask at arch/i386/kernel/smp.c:168
c010fdd5 send_IPI_mask_bitmask+0x65/0x70
c0110236 smp_send_reschedule+0x16/0x20

We've seen this one before, how reproducible is it for you? Could you also please test a 2.6.14-rc -mm kernel?

It's reproducable... I got the same thing with a slightly different configured 2.6.13.2-npe (no preemtion, no acpi, no apm) but beside that, I got other very strange crashes (page table something thingys?) as well during a CRUX pkgmk tool to build i.e. samba. So I wasn't able to get the system stable enough for more serious testing yet.
I am about to grab the latest linus' git tree and try that...

This system was running for a long time with linux without any problems in the past. But I had to change the hdd (old one was broken) and installed a new (CRUX) system from scratch... I migrated to 2.6.13.2 and switched over to udev... I was running memtest86 for about half a day. It didn't show any problems. Are there good torture tests to check if a system's hw is stable?

Thanks,

Signature · 11 lines
Clemens Koller
_______________________________
R&D Imaging Devices
Anagramm GmbH
Rupert-Mayer-Str. 45/1
81379 Muenchen
Germany

http://www.anagramm.de
Phone: +49-89-741518-50
Fax: +49-89-741518-19
#3 lore raw
Quoted 10 lines · Clemens Koller

Zwane Mwaikambo wrote:

It's reproducable... I got the same thing with a slightly different configured 2.6.13.2-npe (no preemtion, no acpi, no apm) but beside that, I got other very strange crashes (page table something thingys?) as well during a CRUX pkgmk
tool to build i.e. samba. So I wasn't able to get the system stable enough for more serious testing yet.
I am about to grab the latest linus' git tree and try that...

Quoted 5 lines

This system was running for a long time with linux without any problems in the past. But I had to change the hdd (old one was broken) and installed a new (CRUX) system from scratch... I migrated to 2.6.13.2 and switched over to udev... I was running memtest86 for about half a day. It didn't show any problems. Are there good torture tests to check if a system's hw is stable?

memtest and repeated multijob kernel/gcc builds seems to do a very good job. Let me know how the new kernel goes, i'm going to try and see if any of my systems can trigger it.

Thanks,
Zwane

#4 lore raw

Hello, Zwane!

Quoted 1 line · You

memtest and repeated multijob kernel/gcc builds seems to do a very good job. Let me know how the new kernel goes, i'm going to try and see if any of my systems can trigger it.

Okay, I've tried the latest linux' git tree from friday last week:
2.6.14-rc3-something, but the

c010fdd5 send_IPI_mask_bitmask+0x65/0x70
c0110236 smp_send_reschedule+0x16/0x20

still occurs when I want to do a "shutdown -h now".
A reboot or "shutdown -r now" seems to work without any errors (or at least I cannot see any until the graphcs card get re-initialized)!

All that is on a Tyan Tomcat IIID (S1563D) machine, w/ the latest BIOS (4.02) APM and ACPI are currently disabled...

Unfortunately, I have had a severe two-at-a-time hard-disk crash on another machine, which kept me quite busy the last days. :-(
Hopefully, by the end of this week, I will be able to debug into that more... Can you give me a pointer to some code, where the kernel actually splits up in between rebooting and halting the system?

Thanks,

Signature · 11 lines
Clemens Koller
_______________________________
R&D Imaging Devices
Anagramm GmbH
Rupert-Mayer-Str. 45/1
81379 Muenchen
Germany

http://www.anagramm.de
Phone: +49-89-741518-50
Fax: +49-89-741518-19
#5 lore raw
Quoted 8 lines · Clemens Koller

All that is on a Tyan Tomcat IIID (S1563D) machine, w/ the latest BIOS (4.02) APM and ACPI are currently disabled...

Unfortunately, I have had a severe two-at-a-time hard-disk crash on another machine, which kept me quite busy the last days. :-(
Hopefully, by the end of this week, I will be able to debug into that more... Can you give me a pointer to some code, where the kernel actually splits up in between rebooting and halting the system?

Hello Clemens,
You can start by having a look at sys_reboot in kernel/sys.c

Cheers,
Zwane

#6 lore raw
In reply to Clemens Koller
Quoted 3 lines · Clemens Koller

I can try the latest git or 2.6.14-rc2 tonight and get you some more info (.config) when I am back home... BTW what is IPI? Any ideas? What do you need to track down

Forgot to answer your other question, IPI is Inter Processor Interrupt, the cpu's method of triggering an interrupt on remote processors.

#7 lore raw
In reply to Clemens Koller

Hello, Guys, hello, Jeff!

This issue seems to happen more than once:

Quoted 1 line · Jeff Collins

I experience a panic whenever I shut down a 4 cpu Intel PII Xeon SMP system.

What panic do you get?

Quoted 7 lines

Linux sitka 2.6.14.3 #2 SMP Fri Dec 2 09:01:46 PST 2005 i686 unknown unknown GNU/Linux Base OS: Slackware 10.2
Kernel: 2.6.14.3 from kernel.org

"shutdown -h now" causes the panic

"shutdown -r now" reboots correctly.

I guess it panics, too, but the reboot still works, so you just don't see the panic. (?)

Quoted 1 line

I got the same panic when I substituted the 2.6.13 kernel.

Still the same thing over here. Unfortunately, I am pretty busy with other work, and the affected system isn't really needed. It's an old Tyan Tomcat IIID Mainboard with two Pentium I MMX 200MHz CPU's. Theoretically I would be able to test the latest git snapshots, but currently it's just not possible. :-(
Let me know if you cannot solve this issue - maybe I can spend some time to give some more information for debugging by the end of this week.

Good Luck,

Best greets,

Signature · 11 lines
Clemens Koller
_______________________________
R&D Imaging Devices
Anagramm GmbH
Rupert-Mayer-Str. 45/1
81379 Muenchen
Germany

http://www.anagramm.de
Phone: +49-89-741518-50
Fax: +49-89-741518-19
#8 lore raw
Quoted 31 lines · Clemens Koller

Hello, Guys, hello, Jeff!

This issue seems to happen more than once:

Quoted 1 line · Jeff Collins

I experience a panic whenever I shut down a 4 cpu Intel PII Xeon SMP system.

What panic do you get?

Quoted 8 lines

Linux sitka 2.6.14.3 #2 SMP Fri Dec 2 09:01:46 PST 2005 i686 unknown unknown GNU/Linux
Base OS: Slackware 10.2
Kernel: 2.6.14.3 from kernel.org

"shutdown -h now" causes the panic

"shutdown -r now" reboots correctly.

I guess it panics, too, but the reboot still works, so you just don't see the panic. (?)

Quoted 1 line

I got the same panic when I substituted the 2.6.13 kernel.

Still the same thing over here. Unfortunately, I am pretty busy with other work, and the affected system isn't really needed. It's an old Tyan Tomcat IIID Mainboard with two Pentium I MMX 200MHz CPU's. Theoretically I would be able to test the latest git snapshots, but currently it's just not possible. :-(
Let me know if you cannot solve this issue - maybe I can spend some time to give some more information for debugging by the end of this week.

Quoted 1 line

From what i hear it's this issue;

http://bugzilla.kernel.org/show_bug.cgi?id=5203

Which is being looked at, feel free to chip in though.

#9 lore raw

Hello, Zwane!

Quoted 3 lines
Quoted 1 line

From what i hear it's this issue;

http://bugzilla.kernel.org/show_bug.cgi?id=5203

Yes it seems to be the same issue.
But who is Eric, mentioned in bugzilla? :-]
If it makes sense I can test his patch while/before he is pushing it upstream.

Thanks!

Signature · 11 lines
Clemens Koller
_______________________________
R&D Imaging Devices
Anagramm GmbH
Rupert-Mayer-Str. 45/1
81379 Muenchen
Germany

http://www.anagramm.de
Phone: +49-89-741518-50
Fax: +49-89-741518-19
#10 lore raw
Quoted 10 lines · Clemens Koller

Hello, Zwane!

Quoted 3 lines
Quoted 1 line

From what i hear it's this issue;

http://bugzilla.kernel.org/show_bug.cgi?id=5203

Yes it seems to be the same issue.
But who is Eric, mentioned in bugzilla? :-]
If it makes sense I can test his patch while/before he is pushing it upstream.

Eric is 'Eric Biederman', Jeff tested his patch but there appears to be a failure case when there is no power management callback installed. Could you please test the following patch?

arch/i386/kernel/reboot.c +7 -3
diff -r 3815424104b0 arch/i386/kernel/reboot.c
@@ -359,6 +359,10 @@
 
 	if (pm_power_off)
 		pm_power_off();
-}
-
-
+
+	local_irq_disable();
+	if (cpu_data[0].hlt_works_ok)
+		while (1) halt();
+	while (1);
+}
+
arch/x86_64/kernel/reboot.c +4 -0
diff -r 3815424104b0 arch/x86_64/kernel/reboot.c
@@ -159,5 +159,9 @@
 	}
 	if (pm_power_off)
 		pm_power_off();
+
+	local_irq_disable();
+	while (1)
+		halt();
 }
 
#11 lore raw

With this patch, my PII Xeon 4 cpu system reaches the state of "Powered Down" and stops. (Running Slackware Linux 10.2 with 2.6.14.3 from kernel.org)

At this point, I can power off or hit the reset button to restart.

Thank you for the patch.

Jeff

Quoted 48 lines · Zwane Mwaikambo
Quoted 10 lines · Clemens Koller

Hello, Zwane!

Quoted 3 lines
Quoted 1 line

From what i hear it's this issue;

http://bugzilla.kernel.org/show_bug.cgi?id=5203

Yes it seems to be the same issue.
But who is Eric, mentioned in bugzilla? :-]
If it makes sense I can test his patch while/before he is pushing it upstream.

Eric is 'Eric Biederman', Jeff tested his patch but there appears to be a failure case when there is no power management callback installed. Could you please test the following patch?

diff -r 3815424104b0 arch/i386/kernel/reboot.c

arch/i386/kernel/reboot.c +7 -3
@@ -359,6 +359,10 @@
 
 	if (pm_power_off)
 		pm_power_off();
-}
-
-
+
+	local_irq_disable();
+	if (cpu_data[0].hlt_works_ok)
+		while (1) halt();
+	while (1);
+}
+

diff -r 3815424104b0 arch/x86_64/kernel/reboot.c

arch/x86_64/kernel/reboot.c +4 -0
@@ -159,5 +159,9 @@
 	}
 	if (pm_power_off)
 		pm_power_off();
+
+	local_irq_disable();
+	while (1)
+		halt();
 }
 
#12 lore raw
Quoted 6 lines · Jeff Collins

With this patch, my PII Xeon 4 cpu system reaches the state of "Powered Down" and stops. (Running Slackware Linux 10.2
with 2.6.14.3 from kernel.org)

At this point, I can power off or hit the reset button to restart.

Thanks for confirming Jeff.

#13 lore raw
In reply to Zwane Mwaikambo

Hello, Zwane!

Quoted 11 lines · Zwane Mwaikambo
Quoted 8 lines
Quoted 3 lines
Quoted 1 line

From what i hear it's this issue;

http://bugzilla.kernel.org/show_bug.cgi?id=5203

Yes it seems to be the same issue.
But who is Eric, mentioned in bugzilla? :-]
If it makes sense I can test his patch while/before he is pushing it upstream.

Eric is 'Eric Biederman', Jeff tested his patch but there appears to be a failure case when there is no power management callback installed. Could you please test the following patch?

Thanks again for the support. Yesterday night, I've checked your patch with the latest git snapshot (2.6.15-rc5-ge4f5c82a), but it doesn't fix the bug(s) for.

2.6.15-rc5-ge4f5c82a unpatched showes up with:
[...cut due to out of screen...]
try_to_wake_up
__wake_up_common
__wake_up
__queue_work
call_usermodehelper_keys
__call_usermodehelper
kobject_hotplug
class_device_del
...

2.6.15-rc5-ge4f5c82a with your patch crashes differently:
[...cut due to out of screen...] send_IPI_mask_bitmask
smp_send_reschedule
try_to_wake_up
__wake_up_common
__wake_up_sync
do_notify_parent
....

or: exactly the same patched kernel but at a later time with a different crash:
Badness in send_IPI_mask_bitmask at arch/i386/kernel/smp.c:167 send_IPI_mask_bitmask
smp_send_reschedule
__migrate_task
migration_thread
migration_thread
kthread
kthread
kernel_thread_helper

The attachments are three .gif images of the screen snapshots of the above error messages (just some more details).

Any ideas?

Greets,

Signature · 11 lines
Clemens Koller
_______________________________
R&D Imaging Devices
Anagramm GmbH
Rupert-Mayer-Str. 45/1
81379 Muenchen
Germany

http://www.anagramm.de
Phone: +49-89-741518-50
Fax: +49-89-741518-19