Skip to contentSkip to conversation

2.6.32.21 + dahdi_dummy (dahdi-2.3.0.1) - uptime related crash?

DISCUSSION Nikola Ciprich 2 participants 2 messages Apr 26, 2011 – Apr 27, 2011

lore
Share & tools
lore
#0 lore raw

Hello everybody,
I have just experienced (almost) simultaneous crash of two identical machines running asterisks and using dahdi_dummy. Both machines were running without problems for about 250 days and suddenly almost at same time, both of them crashed. both machines were running 2.6.32.21 (SMP x86_64) and using dahdi_dummy (dahdi-2.3.0.1)

here's the tail of the backtrace:

[<ffffffff81120cc7>] pollwake+0x57/0x60
[<ffffffff81046720>] ? default_wake_function+0x0/0x10
[<ffffffff8103683a>] __wake_up_common+0x5a/0x90
[<ffffffff8103a313>] __wake_up+0x43/0x70
[<ffffffffa0321573>] process_masterspan+0x643/0x670 [dahdi]
[<ffffffffa0326595>] coretimer_func+0x135/0x1d0 [dahdi]
[<ffffffff8105d74d>] run_timer_softirq+0x15d/0x320
[<ffffffffa0326460>] ? coretimer_func+0x0/0x1d0 [dahdi]
[<ffffffff8105690c>] __do_softirq+0xcc/0x220
[<ffffffff8100c40c>] call_softirq+0x1c/0x30
[<ffffffff8100e3ba>] do_softirq+0x4a/0x80
[<ffffffff810567c7>] irq_exit+0x87/0x90
[<ffffffff8100d7b7>] do_IRQ+0x77/0xf0
[<ffffffff8100bc53>] ret_from_intr+0x0/Oxa
<EUI> [<ffffffffa019e556>] ? acpi_idle_enter_bm+0x273/0x2a1 [processor]
[<ffffffffa019e54c>] ? acpi_idle_enter_bm+0x269/0x2a1 [processor]
[<ffffffff81280095>] ? cpuidle_idle_call+0xa5/0x150
[<ffffffff8100a18f>] ? cpu_idle+0x4f/0x90
[<ffffffff81323c95>] ? rest_init+0x75/0x80
[<ffffffff81582d7f>] ? start_kernel+0x2ef/0x390
[<ffffffff81582271>] ? x86_64_start_reservations+0x81/0xc0
[<ffffffff81582386>] ? x86_64_start_kernel+0xd6/0x100

Sorry, it's trimmed a bit :(

I can't find any related bugreport, so I'm not sure whether I've hit some problem already solved. To me, it seems like some counter might have overflowed or the like (that could explain why machines were running for so long and then suddenly both of them crashed..) Of course used dahdi version was quite old, so I should update anyways, but I'd sleep safer if I'd know the problem already got fixed..
Anyone has some idea?
Should more information be needed, I'd be more then glad to help...
Thanks a lot in advance!
with best regards
nik

Signature · 13 lines
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------
#1 lore raw
Quoted 33 lines · Nikola Ciprich

Hello everybody,
I have just experienced (almost) simultaneous crash of two identical machines running asterisks and using dahdi_dummy. Both machines were running without problems for about 250 days and suddenly almost at same time, both of them crashed. both machines were running 2.6.32.21 (SMP x86_64) and using dahdi_dummy (dahdi-2.3.0.1)

here's the tail of the backtrace:

[<ffffffff81120cc7>] pollwake+0x57/0x60
[<ffffffff81046720>] ? default_wake_function+0x0/0x10
[<ffffffff8103683a>] __wake_up_common+0x5a/0x90
[<ffffffff8103a313>] __wake_up+0x43/0x70
[<ffffffffa0321573>] process_masterspan+0x643/0x670 [dahdi]
[<ffffffffa0326595>] coretimer_func+0x135/0x1d0 [dahdi]
[<ffffffff8105d74d>] run_timer_softirq+0x15d/0x320
[<ffffffffa0326460>] ? coretimer_func+0x0/0x1d0 [dahdi]
[<ffffffff8105690c>] __do_softirq+0xcc/0x220
[<ffffffff8100c40c>] call_softirq+0x1c/0x30
[<ffffffff8100e3ba>] do_softirq+0x4a/0x80
[<ffffffff810567c7>] irq_exit+0x87/0x90
[<ffffffff8100d7b7>] do_IRQ+0x77/0xf0
[<ffffffff8100bc53>] ret_from_intr+0x0/Oxa
<EUI> [<ffffffffa019e556>] ? acpi_idle_enter_bm+0x273/0x2a1 [processor]
[<ffffffffa019e54c>] ? acpi_idle_enter_bm+0x269/0x2a1 [processor]
[<ffffffff81280095>] ? cpuidle_idle_call+0xa5/0x150
[<ffffffff8100a18f>] ? cpu_idle+0x4f/0x90
[<ffffffff81323c95>] ? rest_init+0x75/0x80
[<ffffffff81582d7f>] ? start_kernel+0x2ef/0x390
[<ffffffff81582271>] ? x86_64_start_reservations+0x81/0xc0
[<ffffffff81582386>] ? x86_64_start_kernel+0xd6/0x100

Sorry, it's trimmed a bit :(

Quoted 9 lines

I can't find any related bugreport, so I'm not sure whether I've hit some problem already solved. To me, it seems like some counter might have overflowed or the like (that could explain why machines were running for so long and then suddenly both of them crashed..)
Of course used dahdi version was quite old, so I should update anyways, but I'd sleep safer if I'd know the problem already got fixed.. Anyone has some idea?
Should more information be needed, I'd be more then glad to help... Thanks a lot in advance!

Hello Nikola,

Based on what is posted here, I can't think of any reason hwy there would be a problem in the pollwake function that is timer / overflow dependent.

I also didn't see any changes in the 2.6.32.y stable repository from 2.6.32.21 to the 2.6.32.28 release that appeared to be in the code path.

The only change that looks like it would even be in the same area is r9549 [1], which prevents the wait_queues from being reinitialized before the channel is unregistered, but then I've never personally seen a case where someone actually hit that, and I can't be certain you did. Also, that would be a race so the probability of hitting it on two machines at the same time would be extremely low.

[1] http://svn.asterisk.org/view/dahdi?view=revision&revision=9549

So in summary: I'm not aware of any fixes for what you've seen in either DAHDI or the 2.6.32 stable kernel. Sorry.

Cheers,
Shaun

Signature · 4 lines
Shaun Ruffell
Digium, Inc. | Linux Kernel Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at: www.digium.com & www.asterisk.org