Skip to contentSkip to conversation

Strange Panic (Deadlock)

DISCUSSION Badalian Vyacheslav 3 participants 11 messages Dec 24, 2007 – Dec 27, 2007

Subsystems (3)
Networking Network drivers Traffic control (tc)
lore
Share & tools
lore

Badalian Vyacheslav

author
#0 lore raw

Hello all. Some time machine freeze. No information on monitor. No rebooting on sysctl "kernel.panic".
Any idea?

Catched by netconsole:

[91922.085864] ------------[ cut here ]------------
[91922.085975] kernel BUG at kernel/timer.c:606!
[91922.086058] invalid opcode: 0000 [#1]
[91922.086127] SMP
[91922.086201] Modules linked in: netconsole cls_u32 sch_sfq sch_htb
xt_tcpudp iptable_filter ip_tables x_tables i2c_i801 i2c_core
[91922.086386] CPU:    1
[91922.086387] EIP:    0060:[<c0127387>]    Not tainted VLI
[91922.086389] EFLAGS: 00010087   (2.6.23-gentoo-r4-fw #4)
[91922.086600] EIP is at cascade+0x34/0x4f
[91922.086669] eax: c0452200   ebx: f450408c   ecx: 00000022   edx: f3c6e08c
[91922.086740] esi: 00000022   edi: c21ce000   ebp: 00000001   esp: c21c3ef8
[91922.086815] ds: 007b   es: 007b   fs: 00d8  gs: 0000  ss: 0068
[91922.086885] Process swapper (pid: 0, ti=c21c2000 task=c21af000
task.ti=c21c2000)
[91922.086954] Stack: f3c6e08c c21bfb74 00000000 c21ce000 0000000a
c012767a c21af000 00000001
[91922.087119]        c21c3f18 c0106963 c21c3f68 00000001 00000021
c03c0b08 0000000a c0124556
[91922.087285]        00000046 00000000 c21c2008 00000000 c01245ec
c2015120 c0114a11 00000046
[91922.087451] Call Trace:
[91922.087586]  [<c012767a>] run_timer_softirq+0x51/0x154
[91922.087669]  [<c0106963>] profile_pc+0x21/0x46
[91922.087752]  [<c0124556>] __do_softirq+0x5d/0xc1
[91922.087833]  [<c01245ec>] do_softirq+0x32/0x36
[91922.087915]  [<c0114a11>] smp_apic_timer_interrupt+0x74/0x80
[91922.087997]  [<c010484c>] apic_timer_interrupt+0x28/0x30
[91922.088076]  [<c0102255>] mwait_idle_with_hints+0x3b/0x3f
[91922.088162]  [<c0102259>] mwait_idle+0x0/0xa
[91922.088237]  [<c0102398>] cpu_idle+0x91/0xaa
[91922.088319]  =======================
[91922.088390] Code: 08 8d 04 ca 8b 10 89 62 04 89 14 24 8b 50 04 89 22
89 00 89 54 24 04 8b 14 24 89 40 04 8b 1a eb 19 8b 42 14 83 e0 fe 39 f8
74 04 <0f> 0b eb fe 89 f8 e8 d8 fe ff ff 89 da 8b 1b 39 e2 75 e3 59 89
[91922.088864] EIP: [<c0127387>] cascade+0x34/0x4f SS:ESP 0068:c21c3ef8
author
#1 lore raw

Hello again.
Its bug depend to
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=4aae07025265151e3f7041dfbf0f529e122de1d8
?
Its very critical bug to us. This PC must be HA. Server place so far for me to go and reboot server. I go to it 1-3 times in week =( Please help to fix it =)

Quoted 42 lines

Hello all. Some time machine freeze. No information on monitor. No rebooting on sysctl "kernel.panic".
Any idea?

Catched by netconsole:

[91922.085864] ------------[ cut here ]------------
[91922.085975] kernel BUG at kernel/timer.c:606!
[91922.086058] invalid opcode: 0000 [#1]
[91922.086127] SMP
[91922.086201] Modules linked in: netconsole cls_u32 sch_sfq sch_htb
xt_tcpudp iptable_filter ip_tables x_tables i2c_i801 i2c_core
[91922.086386] CPU:    1
[91922.086387] EIP:    0060:[<c0127387>]    Not tainted VLI
[91922.086389] EFLAGS: 00010087   (2.6.23-gentoo-r4-fw #4)
[91922.086600] EIP is at cascade+0x34/0x4f
[91922.086669] eax: c0452200   ebx: f450408c   ecx: 00000022   edx: f3c6e08c
[91922.086740] esi: 00000022   edi: c21ce000   ebp: 00000001   esp: c21c3ef8
[91922.086815] ds: 007b   es: 007b   fs: 00d8  gs: 0000  ss: 0068
[91922.086885] Process swapper (pid: 0, ti=c21c2000 task=c21af000
task.ti=c21c2000)
[91922.086954] Stack: f3c6e08c c21bfb74 00000000 c21ce000 0000000a
c012767a c21af000 00000001
[91922.087119]        c21c3f18 c0106963 c21c3f68 00000001 00000021
c03c0b08 0000000a c0124556
[91922.087285]        00000046 00000000 c21c2008 00000000 c01245ec
c2015120 c0114a11 00000046
[91922.087451] Call Trace:
[91922.087586]  [<c012767a>] run_timer_softirq+0x51/0x154
[91922.087669]  [<c0106963>] profile_pc+0x21/0x46
[91922.087752]  [<c0124556>] __do_softirq+0x5d/0xc1
[91922.087833]  [<c01245ec>] do_softirq+0x32/0x36
[91922.087915]  [<c0114a11>] smp_apic_timer_interrupt+0x74/0x80
[91922.087997]  [<c010484c>] apic_timer_interrupt+0x28/0x30
[91922.088076]  [<c0102255>] mwait_idle_with_hints+0x3b/0x3f
[91922.088162]  [<c0102259>] mwait_idle+0x0/0xa
[91922.088237]  [<c0102398>] cpu_idle+0x91/0xaa
[91922.088319]  =======================
[91922.088390] Code: 08 8d 04 ca 8b 10 89 62 04 89 14 24 8b 50 04 89 22
89 00 89 54 24 04 8b 14 24 89 40 04 8b 1a eb 19 8b 42 14 83 e0 fe 39 f8
74 04 <0f> 0b eb fe 89 f8 e8 d8 fe ff ff 89 da 8b 1b 39 e2 75 e3 59 89
[91922.088864] EIP: [<c0127387>] cascade+0x34/0x4f SS:ESP 0068:c21c3ef8
Quoted 6 lines

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

#2 lore raw

[email protected] wrote, On 12/24/2007 07:18 PM:

Quoted 4 lines

Hello again.
Its bug depend to
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=4aae07025265151e3f7041dfbf0f529e122de1d8
?

Hello Vyacheslav!

I wonder why do you think there is such a dependency, and why do you report timer.c bug to netdev after all? I added some CCs here, but IMHO you would better open a new bug at bugzilla.kernel.org, adding some more details like .config, and reply back to this thread with the bug's number. BTW, if it's patched by Gentoo or otherwise, you should try and report on 'vanilla' one only.

Regards,
Jarek P.

Quoted 63 lines

Its very critical bug to us. This PC must be HA. Server place so far for me to go and reboot server. I go to it 1-3 times in week =( Please help to fix it =)

Quoted 42 lines

Hello all. Some time machine freeze. No information on monitor. No rebooting on sysctl "kernel.panic".
Any idea?

Catched by netconsole:

[91922.085864] ------------[ cut here ]------------
[91922.085975] kernel BUG at kernel/timer.c:606!
[91922.086058] invalid opcode: 0000 [#1]
[91922.086127] SMP
[91922.086201] Modules linked in: netconsole cls_u32 sch_sfq sch_htb
xt_tcpudp iptable_filter ip_tables x_tables i2c_i801 i2c_core
[91922.086386] CPU:    1
[91922.086387] EIP:    0060:[<c0127387>]    Not tainted VLI
[91922.086389] EFLAGS: 00010087   (2.6.23-gentoo-r4-fw #4)
[91922.086600] EIP is at cascade+0x34/0x4f
[91922.086669] eax: c0452200   ebx: f450408c   ecx: 00000022   edx: f3c6e08c
[91922.086740] esi: 00000022   edi: c21ce000   ebp: 00000001   esp: c21c3ef8
[91922.086815] ds: 007b   es: 007b   fs: 00d8  gs: 0000  ss: 0068
[91922.086885] Process swapper (pid: 0, ti=c21c2000 task=c21af000
task.ti=c21c2000)
[91922.086954] Stack: f3c6e08c c21bfb74 00000000 c21ce000 0000000a
c012767a c21af000 00000001
[91922.087119]        c21c3f18 c0106963 c21c3f68 00000001 00000021
c03c0b08 0000000a c0124556
[91922.087285]        00000046 00000000 c21c2008 00000000 c01245ec
c2015120 c0114a11 00000046
[91922.087451] Call Trace:
[91922.087586]  [<c012767a>] run_timer_softirq+0x51/0x154
[91922.087669]  [<c0106963>] profile_pc+0x21/0x46
[91922.087752]  [<c0124556>] __do_softirq+0x5d/0xc1
[91922.087833]  [<c01245ec>] do_softirq+0x32/0x36
[91922.087915]  [<c0114a11>] smp_apic_timer_interrupt+0x74/0x80
[91922.087997]  [<c010484c>] apic_timer_interrupt+0x28/0x30
[91922.088076]  [<c0102255>] mwait_idle_with_hints+0x3b/0x3f
[91922.088162]  [<c0102259>] mwait_idle+0x0/0xa
[91922.088237]  [<c0102398>] cpu_idle+0x91/0xaa
[91922.088319]  =======================
[91922.088390] Code: 08 8d 04 ca 8b 10 89 62 04 89 14 24 8b 50 04 89 22
89 00 89 54 24 04 8b 14 24 89 40 04 8b 1a eb 19 8b 42 14 83 e0 fe 39 f8
74 04 <0f> 0b eb fe 89 f8 e8 d8 fe ff ff 89 da 8b 1b 39 e2 75 e3 59 89
[91922.088864] EIP: [<c0127387>] cascade+0x34/0x4f SS:ESP 0068:c21c3ef8
Quoted 6 lines

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#3 lore raw

Jarek Poplawski пишет:

Quoted 9 lines

[email protected] wrote, On 12/24/2007 07:18 PM:

Quoted 5 lines

Hello again.
Its bug depend to
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=4aae07025265151e3f7041dfbf0f529e122de1d8
?

ok. i will add it to bugtracker, but bug process in gentoo and in vanilla kernel.
I send to netdev mail list becouse i think that bug depend to TC or IPTABLES functional.
I have 4 machine. All platforms different. All machine do 1 time in
hour rebuild TC and IPTABLES rules.
After it do
echo START >> log.txt
iptables-restore < xxx.txt
tc qdisc del dev eth0 root
tc qdisc del dev eth1 root
tc -b new_rules.txt
echo END >> log.txt

and its all that its doing.
Bug always be between START and END
All machines have above 300mbs traffic.
I try turn off rebuilding rules on 1 PC and it work 3 week without reboot!

I think that situation ask that problem depends to network. If its mistake - sorry please.

Slavon

Quoted 81 lines

Hello Vyacheslav!

I wonder why do you think there is such a dependency, and why do you report timer.c bug to netdev after all? I added some CCs here, but IMHO you would better open a new bug at bugzilla.kernel.org, adding some more details like .config, and reply back to this thread with the bug's number. BTW, if it's patched by Gentoo or otherwise, you should try and report on 'vanilla' one only.

Regards,
Jarek P.

Quoted 64 lines

Its very critical bug to us. This PC must be HA. Server place so far for me to go and reboot server. I go to it 1-3 times in week =( Please help to fix it =)

Quoted 42 lines

Hello all. Some time machine freeze. No information on monitor. No rebooting on sysctl "kernel.panic".
Any idea?

Catched by netconsole:

[91922.085864] ------------[ cut here ]------------
[91922.085975] kernel BUG at kernel/timer.c:606!
[91922.086058] invalid opcode: 0000 [#1]
[91922.086127] SMP
[91922.086201] Modules linked in: netconsole cls_u32 sch_sfq sch_htb
xt_tcpudp iptable_filter ip_tables x_tables i2c_i801 i2c_core
[91922.086386] CPU:    1
[91922.086387] EIP:    0060:[<c0127387>]    Not tainted VLI
[91922.086389] EFLAGS: 00010087   (2.6.23-gentoo-r4-fw #4)
[91922.086600] EIP is at cascade+0x34/0x4f
[91922.086669] eax: c0452200   ebx: f450408c   ecx: 00000022   edx: f3c6e08c
[91922.086740] esi: 00000022   edi: c21ce000   ebp: 00000001   esp: c21c3ef8
[91922.086815] ds: 007b   es: 007b   fs: 00d8  gs: 0000  ss: 0068
[91922.086885] Process swapper (pid: 0, ti=c21c2000 task=c21af000
task.ti=c21c2000)
[91922.086954] Stack: f3c6e08c c21bfb74 00000000 c21ce000 0000000a
c012767a c21af000 00000001
[91922.087119]        c21c3f18 c0106963 c21c3f68 00000001 00000021
c03c0b08 0000000a c0124556
[91922.087285]        00000046 00000000 c21c2008 00000000 c01245ec
c2015120 c0114a11 00000046
[91922.087451] Call Trace:
[91922.087586]  [<c012767a>] run_timer_softirq+0x51/0x154
[91922.087669]  [<c0106963>] profile_pc+0x21/0x46
[91922.087752]  [<c0124556>] __do_softirq+0x5d/0xc1
[91922.087833]  [<c01245ec>] do_softirq+0x32/0x36
[91922.087915]  [<c0114a11>] smp_apic_timer_interrupt+0x74/0x80
[91922.087997]  [<c010484c>] apic_timer_interrupt+0x28/0x30
[91922.088076]  [<c0102255>] mwait_idle_with_hints+0x3b/0x3f
[91922.088162]  [<c0102259>] mwait_idle+0x0/0xa
[91922.088237]  [<c0102398>] cpu_idle+0x91/0xaa
[91922.088319]  =======================
[91922.088390] Code: 08 8d 04 ca 8b 10 89 62 04 89 14 24 8b 50 04 89 22
89 00 89 54 24 04 8b 14 24 89 40 04 8b 1a eb 19 8b 42 14 83 e0 fe 39 f8
74 04 <0f> 0b eb fe 89 f8 e8 d8 fe ff ff 89 da 8b 1b 39 e2 75 e3 59 89
[91922.088864] EIP: [<c0127387>] cascade+0x34/0x4f SS:ESP 0068:c21c3ef8
Quoted 7 lines

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#4 lore raw

On Tue, Dec 25, 2007 at 12:11:50PM +0300, Badalian Vyacheslav wrote:
...

Quoted 14 lines

ok. i will add it to bugtracker, but bug process in gentoo and in vanilla kernel.
I send to netdev mail list becouse i think that bug depend to TC or IPTABLES functional.
I have 4 machine. All platforms different. All machine do 1 time in
hour rebuild TC and IPTABLES rules.
After it do
echo START >> log.txt
iptables-restore < xxx.txt
tc qdisc del dev eth0 root
tc qdisc del dev eth1 root
tc -b new_rules.txt
echo END >> log.txt

Quoted 8 lines

and its all that its doing.
Bug always be between START and END
All machines have above 300mbs traffic.
I try turn off rebuilding rules on 1 PC and it work 3 week without reboot!

I think that situation ask that problem depends to network. If its mistake - sorry please.

Yes, this description seems to point at network, but since the bug triggers in timer.c ...we could try to share this work with somebody (or even blame them 100% if they are not clever enough...)?!

I think there were similar things reported especially around HTB, but it seems there were problems with later debugging. I'll try to think about it, but if there are some more logs or details, it should be helpful.

BTW: you've written there is a need to go and reboot this each time:
did you try something like drivers/char/watchdog/softdog.c?

Regards,
Jarek P.

#5 lore raw

Probably also there is TCO watchdog, if it is Intel motherboard. I am using also on unreliable machines nmi_watchdog. And also if it is servers, probably there is IPMI.

Plus it is IMHO important to know lspci -vvv, cat /proc/interrupts, and kernel config.

Vyacheslav, on my experience i am tried also latest rc kernels on production machines, it helps sometimes for me, and also it helps alot to kernel developers. Gentoo kernels not heavily patched in critical parts, but IMHO it will make debugging difficult, if patch exist in "failed" part.

I am not kernel developer, but can help with watchdogs and etc, and probably extended debugging. Contact me via ICQ 17962627 or MSN nuclearcat AT nuclearcat.com, probably i can help somehow to make your setup more reliable.

On Tue, 25 Dec 2007 15:49:40 +0100, Jarek Poplawski wrote

Quoted 37 lines · Badalian Vyacheslav

....

Quoted 14 lines

ok. i will add it to bugtracker, but bug process in gentoo and in vanilla kernel.
I send to netdev mail list becouse i think that bug depend to TC or IPTABLES functional.
I have 4 machine. All platforms different. All machine do 1 time in
hour rebuild TC and IPTABLES rules.
After it do
echo START >> log.txt
iptables-restore < xxx.txt
tc qdisc del dev eth0 root
tc qdisc del dev eth1 root
tc -b new_rules.txt
echo END >> log.txt

Quoted 8 lines

and its all that its doing.
Bug always be between START and END
All machines have above 300mbs traffic.
I try turn off rebuilding rules on 1 PC and it work 3 week without reboot!

I think that situation ask that problem depends to network. If its mistake - sorry please.

Yes, this description seems to point at network, but since the bug triggers in timer.c ...we could try to share this work with somebody (or even blame them 100% if they are not clever enough...)?!

I think there were similar things reported especially around HTB, but it seems there were problems with later debugging. I'll try to think about it, but if there are some more logs or details, it should be helpful.

BTW: you've written there is a need to go and reboot this each time:
did you try something like drivers/char/watchdog/softdog.c?

Quoted 6 lines

Regards,
Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.

#6 lore raw
In reply to Badalian Vyacheslav

On Tue, Dec 25, 2007 at 12:11:50PM +0300, Badalian Vyacheslav wrote:
...

Quoted 10 lines

I have 4 machine. All platforms different. All machine do 1 time in
hour rebuild TC and IPTABLES rules. After it do
echo START >> log.txt iptables-restore < xxx.txt
tc qdisc del dev eth0 root
tc qdisc del dev eth1 root
tc -b new_rules.txt
echo END >> log.txt

Quoted 4 lines

and its all that its doing.
Bug always be between START and END
All machines have above 300mbs traffic.
I try turn off rebuilding rules on 1 PC and it work 3 week without reboot!

Hi Slavon,

After some looking around net schedulers' timers I think you could try these 3 patches (2 in next messages). They should change 3 suspicious (maybe only to me) places, but it's only guessing, to eliminate some most nearby possibilities. These patches are independent, but of course trying all at once should be quicker.

Thanks,
Jarek P.

[PATCH 1/3]
---

net/sched/sch_generic.c +3 -5
diff -Nurp linux-2.6.23.12-/net/sched/sch_generic.c linux-2.6.23.12+/net/sched/sch_generic.c
@@ -251,10 +251,8 @@ static void dev_watchdog_up(struct net_d
 
 static void dev_watchdog_down(struct net_device *dev)
 {
-	netif_tx_lock_bh(dev);
-	if (del_timer(&dev->watchdog_timer))
+	if (del_timer_sync(&dev->watchdog_timer))
 		dev_put(dev);
-	netif_tx_unlock_bh(dev);
 }
 
 void netif_carrier_on(struct net_device *dev)

@@ -560,6 +558,8 @@ void dev_deactivate(struct net_device *d
 	struct Qdisc *qdisc;
 	struct sk_buff *skb;
 
+	dev_watchdog_down(dev);
+
 	spin_lock_bh(&dev->queue_lock);
 	qdisc = dev->qdisc;
 	dev->qdisc = &noop_qdisc;

@@ -572,8 +572,6 @@ void dev_deactivate(struct net_device *d
 
 	kfree_skb(skb);
 
-	dev_watchdog_down(dev);
-
 	/* Wait for outstanding dev_queue_xmit calls. */
 	synchronize_rcu();
#7 lore raw

On Wed, Dec 26, 2007 at 07:54:11PM +0100, Jarek Poplawski wrote:
...

[PATCH 2/3] (for testing only)
---

net/sched/sch_sfq.c +1 -1
diff -Nurp linux-2.6.23.12-/net/sched/sch_sfq.c linux-2.6.23.12+/net/sched/sch_sfq.c
@@ -457,7 +457,7 @@ static int sfq_init(struct Qdisc *sch, s
 static void sfq_destroy(struct Qdisc *sch)
 {
 	struct sfq_sched_data *q = qdisc_priv(sch);
-	del_timer(&q->perturb_timer);
+	del_timer_sync(&q->perturb_timer);
 }
 
 static int sfq_dump(struct Qdisc *sch, struct sk_buff *skb)
#8 lore raw

On Wed, Dec 26, 2007 at 07:56:42PM +0100, Jarek Poplawski wrote:
...

[PATCH 3/3] (for testing only)
---

net/sched/sch_api.c +4 -1
diff -Nurp linux-2.6.23.12-/net/sched/sch_api.c linux-2.6.23.12+/net/sched/sch_api.c
@@ -514,8 +514,11 @@ qdisc_create(struct net_device *dev, u32
 				 * a ops->reset() here? The qdisc was never
 				 * in action so it shouldn't be necessary.
 				 */
-				if (ops->destroy)
+				if (ops->destroy) {
+					qdisc_lock_tree(dev);
 					ops->destroy(sch);
+					qdisc_unlock_tree(dev);
+				}
 				goto err_out3;
 			}
 		}
#9 lore raw
Quoted 1 line · Jarek Poplawski

...

And here is one more: this place needs more advanced debugging, but let's check something simple at the beginning... (and just like before
- could be tested with or without these earlier patches).

Jarek P.

[PATCH 4/3] (for testing only)
---

kernel/timer.c +4 -1
diff -Nurp linux-2.6.23-/kernel/timer.c linux-2.6.23+/kernel/timer.c
@@ -603,7 +603,10 @@ static int cascade(tvec_base_t *base, tv
 	 * don't have to detach them individually.
 	 */
 	list_for_each_entry_safe(timer, tmp, &tv_list, entry) {
-		BUG_ON(tbase_get_base(timer->base) != base);
+		if (tbase_get_base(timer->base) != base) {
+			print_ip_sym((long)timer->function);
+			BUG_ON(1);
+		}
 		internal_add_timer(base, timer);
 	}
author
#10 lore raw

Jarek Poplawski пишет:

Quoted 21 lines · Jarek Poplawski
Quoted 3 lines

...

And here is one more: this place needs more advanced debugging, but let's check something simple at the beginning... (and just like before
- could be tested with or without these earlier patches).

Jarek P.

[PATCH 4/3] (for testing only)
---

diff -Nurp linux-2.6.23-/kernel/timer.c linux-2.6.23+/kernel/timer.c

kernel/timer.c +0 -0
@@ -603,7 +603,10 @@ static int cascade(tvec_base_t *base, tv
 	 * don't have to detach them individually.

kernel/timer.c static int cascade(tvec_base_t *base, tv · line 612

Quoted 10 lines
kernel/timer.c +4 -1
 	 */
 	list_for_each_entry_safe(timer, tmp, &tv_list, entry) {
-		BUG_ON(tbase_get_base(timer->base) != base);
+		if (tbase_get_base(timer->base) != base) {
+			print_ip_sym((long)timer->function);
+			BUG_ON(1);
+		}
 		internal_add_timer(base, timer);
 	}

Hello
I add bugreport
http://bugzilla.kernel.org/show_bug.cgi?id=9632
/Oleg Nesterov <mailto:[email protected]> give me patch to debug bugplace... its look stable and i try it in few days... Slavon
/