kernel panic

DISCUSSION senthilkumaar2021 3 participants 6 messages May 18, 2010 1 message not in the archive

Subsystems (4)

Networking Network drivers netfilter Ethernet bridge

lore

Share & tools

lore

senthilkumaar2021

author

2010-05-18 04:26 UTC #0 lore raw

I am using iptables version 1.43 and getting the following error in machine while using squid in tproxy.we are using Linux machine as bridge too

i was able to capture part of the error only please help me in solving the issue

[<ffffffffa03933c2>] ? nf_nat_fn+0x138/0x14e [iptable_nat] [<ffffffffa0393585>] ? nf_nat_in+0x2f/0x6e [iptable_nat] [<ffffffffa027edaa>] ? br_nf_pre_routing_finish+0x0/ 0x2c4 [bridge]
[<ffffffffa027edfa>] br_nf_pre_routing_finish+0x50/ 0x2c4 [bridge]
[<ffffffffa027edaa>] ? br_nf_pre_routing_finish+0x0/ 0x2c4 [bridge]
[<ffffffff81339a50>] ? nf_hook_slow+0x68/0xc8 [<ffffffffa027edaa>] ? br_nf_pre_routing_finish+0x0/0x2c4 [bridge]

[<ffffffffa027f616>] br_nf_pre_routing+0x5a8/0x5c7 [bridge]
[<ffffffff813399ab>] nf_iterate+0x48/0x85
[<ffffffffa027a931>] ? br_handle_frame_finish+0x0/0x154 [bridge]
[<ffffffff81339a50>] nf_hook_slow+0x68/0xc8
[<ffffffffa027a931>] ? br_handle_frame_finish+0x0/0x154 [bridge]
[<ffffffffa027ac36>] br_handle_frame+0x1b1/0x1db [bridge]
[<ffffffff8131d54b>] netif_receive_skb+0x316/0x434
[<ffffffff8131dbfb>] napi_gro_receive+0x6e/0x83
[<ffffffffa0125bfe>] e1000_receive_skb+0x5c/0x65 [e1000e]
[<ffffffffa0125de8>] e1000_clean_rx_irq+0x1e1/0x28f [e1000e]
[<ffffffffa012730e>] e1000_clean+0x99/0x24a [e1000e]
[<ffffffff813bcfc5>] ? _spin_unlock_irqrestore+0x2c/0x43
[<ffffffff8131ba62>] net_rx_action+0xb8/0x1b4
[<ffffffff8104ed43>] __do_softirq+0x99/0x152
[<ffffffff8101284c>] call_softirq+0x1c/0x30
[<ffffffff81013a02>] do_softirq+0x52/0xb9
[<ffffffff8104e969>] irq_exit+0x53/0x8d
[<ffffffff81013d1a>] do_IRQ+0x135/0x157
[<ffffffff81011f93>] ret_from_intr+0x0/0x2e
<EOI> [<ffffffff81017e20>] ? mwait_idle+0x9e/0xc7
[<ffffffff81017e17>] ? mwait_idle+0x95/0xc7
[<ffffffff813bfd20>] ? atomic_notifier_call_chain+0x13/0x15
[<ffffffff810102f4>] ? enter_idle+0x27/0x2>

Regards
senthil

Justin P. Mattock

2010-05-18 04:47 UTC #1 lore raw

Quoted 45 lines · On 05/17/2010 09:23 PM, senthilkumaar2021 wrote:

HI
I am using iptables version 1.43 and getting the following error in machine while using squid in tproxy.we are using Linux machine as bridge too
i was able to capture part of the error only please help me in solving the issue
[<ffffffffa03933c2>] ? nf_nat_fn+0x138/0x14e [iptable_nat] [<ffffffffa0393585>] ? nf_nat_in+0x2f/0x6e [iptable_nat] [<ffffffffa027edaa>] ? br_nf_pre_routing_finish+0x0/ 0x2c4 [bridge]
[<ffffffffa027edfa>] br_nf_pre_routing_finish+0x50/ 0x2c4 [bridge]
[<ffffffffa027edaa>] ? br_nf_pre_routing_finish+0x0/ 0x2c4 [bridge]
[<ffffffff81339a50>] ? nf_hook_slow+0x68/0xc8 [<ffffffffa027edaa>] ? br_nf_pre_routing_finish+0x0/0x2c4 [bridge]
[<ffffffffa027f616>] br_nf_pre_routing+0x5a8/0x5c7 [bridge]
[<ffffffff813399ab>] nf_iterate+0x48/0x85
[<ffffffffa027a931>] ? br_handle_frame_finish+0x0/0x154 [bridge]
[<ffffffff81339a50>] nf_hook_slow+0x68/0xc8
[<ffffffffa027a931>] ? br_handle_frame_finish+0x0/0x154 [bridge]
[<ffffffffa027ac36>] br_handle_frame+0x1b1/0x1db [bridge]
[<ffffffff8131d54b>] netif_receive_skb+0x316/0x434
[<ffffffff8131dbfb>] napi_gro_receive+0x6e/0x83
[<ffffffffa0125bfe>] e1000_receive_skb+0x5c/0x65 [e1000e]
[<ffffffffa0125de8>] e1000_clean_rx_irq+0x1e1/0x28f [e1000e]
[<ffffffffa012730e>] e1000_clean+0x99/0x24a [e1000e]
[<ffffffff813bcfc5>] ? _spin_unlock_irqrestore+0x2c/0x43
[<ffffffff8131ba62>] net_rx_action+0xb8/0x1b4
[<ffffffff8104ed43>] __do_softirq+0x99/0x152
[<ffffffff8101284c>] call_softirq+0x1c/0x30
[<ffffffff81013a02>] do_softirq+0x52/0xb9
[<ffffffff8104e969>] irq_exit+0x53/0x8d
[<ffffffff81013d1a>] do_IRQ+0x135/0x157
[<ffffffff81011f93>] ret_from_intr+0x0/0x2e
<EOI> [<ffffffff81017e20>] ? mwait_idle+0x9e/0xc7
[<ffffffff81017e17>] ? mwait_idle+0x95/0xc7
[<ffffffff813bfd20>] ? atomic_notifier_call_chain+0x13/0x15
[<ffffffff810102f4>] ? enter_idle+0x27/0x2>

Quoted 8 lines

Regards
senthil
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

if you can.. find the last good kernel, then
do a bisect.
(if you need help doing a bisect, there's plenty
of people to help you out).

Justin P. Mattock

senthilkumaar2021

author

2010-05-18 04:58 UTC #2 lore raw

Thanks for the reply

Initially i was using kernel 2.6.28.5 and kernel panic occurred in that version too

tproxy is not supported in linux 2.6.32

http://wiki.squid-cache.org/Features/Tproxy4

kindly guide me how to proceed

Regards
senthil

Quoted 63 lines · Justin P. Mattock

Quoted 45 lines · On 05/17/2010 09:23 PM, senthilkumaar2021 wrote:
HI
I am using iptables version 1.43 and getting the following error in machine while using squid in tproxy.we are using Linux machine as bridge too
i was able to capture part of the error only please help me in solving the issue
[<ffffffffa03933c2>] ? nf_nat_fn+0x138/0x14e [iptable_nat] [<ffffffffa0393585>] ? nf_nat_in+0x2f/0x6e [iptable_nat] [<ffffffffa027edaa>] ? br_nf_pre_routing_finish+0x0/ 0x2c4 [bridge]
[<ffffffffa027edfa>] br_nf_pre_routing_finish+0x50/ 0x2c4 [bridge]
[<ffffffffa027edaa>] ? br_nf_pre_routing_finish+0x0/ 0x2c4 [bridge]
[<ffffffff81339a50>] ? nf_hook_slow+0x68/0xc8 [<ffffffffa027edaa>] ? br_nf_pre_routing_finish+0x0/0x2c4 [bridge]
[<ffffffffa027f616>] br_nf_pre_routing+0x5a8/0x5c7 [bridge]
[<ffffffff813399ab>] nf_iterate+0x48/0x85
[<ffffffffa027a931>] ? br_handle_frame_finish+0x0/0x154 [bridge]
[<ffffffff81339a50>] nf_hook_slow+0x68/0xc8
[<ffffffffa027a931>] ? br_handle_frame_finish+0x0/0x154 [bridge]
[<ffffffffa027ac36>] br_handle_frame+0x1b1/0x1db [bridge]
[<ffffffff8131d54b>] netif_receive_skb+0x316/0x434
[<ffffffff8131dbfb>] napi_gro_receive+0x6e/0x83
[<ffffffffa0125bfe>] e1000_receive_skb+0x5c/0x65 [e1000e]
[<ffffffffa0125de8>] e1000_clean_rx_irq+0x1e1/0x28f [e1000e]
[<ffffffffa012730e>] e1000_clean+0x99/0x24a [e1000e]
[<ffffffff813bcfc5>] ? _spin_unlock_irqrestore+0x2c/0x43
[<ffffffff8131ba62>] net_rx_action+0xb8/0x1b4
[<ffffffff8104ed43>] __do_softirq+0x99/0x152
[<ffffffff8101284c>] call_softirq+0x1c/0x30
[<ffffffff81013a02>] do_softirq+0x52/0xb9
[<ffffffff8104e969>] irq_exit+0x53/0x8d
[<ffffffff81013d1a>] do_IRQ+0x135/0x157
[<ffffffff81011f93>] ret_from_intr+0x0/0x2e
<EOI> [<ffffffff81017e20>] ? mwait_idle+0x9e/0xc7
[<ffffffff81017e17>] ? mwait_idle+0x95/0xc7
[<ffffffff813bfd20>] ? atomic_notifier_call_chain+0x13/0x15
[<ffffffff810102f4>] ? enter_idle+0x27/0x2>
Quoted 8 lines
Regards
senthil
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
if you can.. find the last good kernel, then
do a bisect.
(if you need help doing a bisect, there's plenty
of people to help you out).
Justin P. Mattock

Eric Dumazet

2010-05-18 05:05 UTC #3 lore raw

Quoted 4 lines · Le mardi 18 mai 2010 à 10:25 +0530, senthilkumaar2021 a écrit :

Hi
Thanks for the reply

Quoted 8 lines

Initially i was using kernel 2.6.28.5 and kernel panic occurred in that version too
tproxy is not supported in linux 2.6.32
http://wiki.squid-cache.org/Features/Tproxy4
kindly guide me how to proceed

Please submit full panic log, and kernel version

AFAIK tproxy is now in Linux kernel tree.

Justin P. Mattock

2010-05-18 05:09 UTC #6 lore raw

In reply to senthilkumaar2021

Quoted 7 lines · On 05/17/2010 09:55 PM, senthilkumaar2021 wrote:

Hi
Thanks for the reply
Initially i was using kernel 2.6.28.5 and kernel panic occurred in that version too

Quoted 9 lines

tproxy is not supported in linux 2.6.32
http://wiki.squid-cache.org/Features/Tproxy4
kindly guide me how to proceed
Regards
senthil

hmm.. I'm wondering if this should be discussed here then. In any case do you receive this error with 2.6.30 (even though 2.6.28 gives you this)

if so then doing a bisect might be a bit hard todo. (unless 2.6.30 is good, then a bisect can at-least point you in the right direction with what might be happening).

Justin P. Mattock

Eric Dumazet

2010-05-18 05:49 UTC #5 lore raw

In reply to a message not in the archive

Quoted 48 lines · Le mardi 18 mai 2010 à 10:46 +0530, senthilkumaar2021 a écrit :

Hi
thanks fro the reply
i was able to capture only part of the message
Machine was running squid 2.7stable6 with tproxy4 in bridge mode on Linux kernel 2.6.28.5 and the server had
kernel panic for every 10 to 12 hours. The logged error:
    [<ffffffffa0152a4b>] bnx2_poll_work+0xea0/0xfb9 [bnx2]
    [<ffffffff81029713>] enqueue_task+0x50/0x5b
    [<ffffffff8104d2ab>] getnstimeofday+0x53/0xb2
   [<ffffffffa0152f31>] bnx2_poll+0xd1/0x1ae [bnx2]
     [<ffffffff8120724a>] net_rx_action+0x9d/0x158
     [<ffffffff8103af44>] __do_softirq+0x7a/0x13d
     [<ffffffffa0151b6e>] bnx2_msi+0x40/0x47 [bnx2]
     [<ffffffff8100cf5c>] call_softirq+0x1c/0x28
     [<ffffffff8100ddc0>] do_softirq+0x2c/0x68
     [<ffffffff8103ae84>] irq_exit+0x3f/0x85
     [<ffffffff8100e06f>] do_IRQ+0x14a/0x16b
    [<ffffffff8100c216>] ret_from_intr+0x0/0xa
    <EOI> <0> [<ffffffff81174711>] acpi_idle_enter_bm+0x2a3/0x30e
    [<ffffffff81174707>] acpi_idle_enter_bm+0x299/0x30e
   [<ffffffff8106bac3>] rcu_needs_cpu+0x35/0x44
    [<ffffffff811e8269>] cpuidle_idle_call+0x7f/0xbe
     [<ffffffff8100abd4>] cpu_idle+0x4a/0x6d
    Code: 5e 5b 5d 31 c0 c3 41 55 41 54 55 53 48 89 fb 48 83 ec 68 4c 8b
     a7 98 00 00
    00 4c 8b 6f 20 48 8b b7 d0 00 00 00 8b 8f bc 00 00 00 <41> f6 44 24
     18 01 74 12
     8a 47 7d 83 e0 f8 83 c8 03 88 47 7d 41
    RIP [<ffffffffa02c405f>] br_nf_pre_routing_finish+0x25/0x2af [bridge]
    RSP <ffffffff8155fb40>
    CR2: 0000000000000018
    Kernel panic - not syncing: Fatal exception in interrupt
The kernel panic occurred in CentOS as well as Fedora 8 64 bit.
When we checked this with squid community, they replied it as the problem of the Network Card. When the NIC was changed to Intel from Broadcom, RAM replaced with new sets and kernel updated to 2.6.30.5, the find it attached). This happens once in two days. We have been trying to resolve this for over 3 weeks now but in vain, any help on this is much appreciated.
hardware detail_

Quoted 7 lines

CPU : 3.0 GHz
RAM : 4 GB RAM
HDD : 4 x 150 GB
OS : Fedora 8 64 bit
FS : ReiserFS
Max request rate : 400 per second

You could try to adapt commit e94c67436e

Maybe its same problem...

commit e94c67436efa22af7d8b7d19c885863246042543

Author: Bart De Schuymer <[email protected]>
Date:   Thu May 13 14:55:34 2010 +0200

    netfilter: bridge-netfilter: fix crash in br_nf_forward_finish()
    
    [ 4593.956206] BUG: unable to handle kernel NULL pointer dereference at 0000000000000018
    [ 4593.956219] IP: [<ffffffffa03357a4>] br_nf_forward_finish+0x154/0x170 [bridge]
    [ 4593.956232] PGD 195ece067 PUD 1ba005067 PMD 0
    [ 4593.956241] Oops: 0000 [#1] SMP
    [ 4593.956248] last sysfs file:
    /sys/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:08/ATK0110:00/hwmon/hwmon0/temp2_label
    [ 4593.956253] CPU 3
    ...
    [ 4593.956380] Pid: 29512, comm: kvm Not tainted 2.6.34-rc7-net #195 P6T DELUXE/System Product Name
    [ 4593.956384] RIP: 0010:[<ffffffffa03357a4>]  [<ffffffffa03357a4>] br_nf_forward_finish+0x154/0x170 [bridge]
    [ 4593.956395] RSP: 0018:ffff880001e63b78  EFLAGS: 00010246
    [ 4593.956399] RAX: 0000000000000608 RBX: ffff880057181700 RCX: ffff8801b813d000
    [ 4593.956402] RDX: 0000000000000000 RSI: 0000000000000002 RDI: ffff880057181700
    [ 4593.956406] RBP: ffff880001e63ba8 R08: ffff8801b9d97000 R09: ffffffffa0335650
    [ 4593.956410] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8801b813d000
    [ 4593.956413] R13: ffffffff81ab3940 R14: ffff880057181700 R15: 0000000000000002
    [ 4593.956418] FS:  00007fc40d380710(0000) GS:ffff880001e60000(0000) knlGS:0000000000000000
    [ 4593.956422] CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
    [ 4593.956426] CR2: 0000000000000018 CR3: 00000001ba1d7000 CR4: 00000000000026e0
    [ 4593.956429] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
    [ 4593.956433] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
    [ 4593.956437] Process kvm (pid: 29512, threadinfo ffff8801ba566000, task ffff8801b8003870)
    [ 4593.956441] Stack:
    [ 4593.956443]  0000000100000020 ffff880001e63ba0 ffff880001e63ba0 ffff880057181700
    [ 4593.956451] <0> ffffffffa0335650 ffffffff81ab3940 ffff880001e63bd8 ffffffffa03350e6
    [ 4593.956462] <0> ffff880001e63c40 000000000000024d ffff880057181700 0000000080000000
    [ 4593.956474] Call Trace:
    [ 4593.956478]  <IRQ>
    [ 4593.956488]  [<ffffffffa0335650>] ? br_nf_forward_finish+0x0/0x170 [bridge]
    [ 4593.956496]  [<ffffffffa03350e6>] NF_HOOK_THRESH+0x56/0x60 [bridge]
    [ 4593.956504]  [<ffffffffa0335282>] br_nf_forward_arp+0x112/0x120 [bridge]
    [ 4593.956511]  [<ffffffff813f7184>] nf_iterate+0x64/0xa0
    [ 4593.956519]  [<ffffffffa032f920>] ? br_forward_finish+0x0/0x60 [bridge]
    [ 4593.956524]  [<ffffffff813f722c>] nf_hook_slow+0x6c/0x100
    [ 4593.956531]  [<ffffffffa032f920>] ? br_forward_finish+0x0/0x60 [bridge]
    [ 4593.956538]  [<ffffffffa032f800>] ? __br_forward+0x0/0xc0 [bridge]
    [ 4593.956545]  [<ffffffffa032f86d>] __br_forward+0x6d/0xc0 [bridge]
    [ 4593.956550]  [<ffffffff813c5d8e>] ? skb_clone+0x3e/0x70
    [ 4593.956557]  [<ffffffffa032f462>] deliver_clone+0x32/0x60 [bridge]
    [ 4593.956564]  [<ffffffffa032f6b6>] br_flood+0xa6/0xe0 [bridge]
    [ 4593.956571]  [<ffffffffa032f800>] ? __br_forward+0x0/0xc0 [bridge]
    
    Don't call nf_bridge_update_protocol() for ARP traffic as skb->nf_bridge isn't
    used in the ARP case.
    
    Reported-by: Stephen Hemminger <[email protected]>
    Signed-off-by: Bart De Schuymer <[email protected]>
    Signed-off-by: Patrick McHardy <[email protected]>

net/bridge/br_netfilter.c +1 -1

index 93f80fe..4442099 100644
@@ -643,10 +643,10 @@ static int br_nf_forward_finish(struct sk_buff *skb)
 			skb->pkt_type = PACKET_OTHERHOST;
 			nf_bridge->mask ^= BRNF_PKT_TYPE;
 		}
+		nf_bridge_update_protocol(skb);
 	} else {
 		in = *((struct net_device **)(skb->cb));
 	}
-	nf_bridge_update_protocol(skb);
 	nf_bridge_push_encap_header(skb);
 
 	NF_HOOK_THRESH(NFPROTO_BRIDGE, NF_BR_FORWARD, skb, in,