Panic in ipt_do_table with 2.6.16.13-xen
I have been noticing this same problem dozens of times and have finally caught a full trace. I have run it through ksymoops, but there is no /proc/ksyms. Is there a better method for getting information out of the Code line than using ksymoops in 2.6 kernels?
The kernel is for Xen, but it does not appear to be related to Xen.
ksymoops run:
# ksymoops -m System.map -k /proc/kallsyms </root/xen-oops.log
ksymoops 2.4.9 on i686 2.6.16.13-xen. Options used
-V (default)
-k /proc/kallsyms (specified)
-l /proc/modules (default)
-o /lib/modules/2.6.16.13-xen/ (default)
-m System.map (specified)Warning (read_ksyms): no kernel symbols in ksyms, is /proc/kallsyms a valid ksyms file? No modules in ksyms, skipping objects
No ksyms, skipping lsmod
Unable to handle kernel paging request at virtual addr
c03d96a9008c2000 -> *pde = 00000001:aeead00100e1f000 -> *pme = 00000000:eb2ef0670faef000 -> *pte = 00000000:00000000
Oops: 0000 [#1] CPU: 0 EIP: 0061:[<c03d96a9>] Not tainted VLI
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010286 (2.6.16.13-xen #1)
eax: 00000002 ebx: cfcc6020 ecx: cfcc6020 edx: 00000000
esi: d1320000 edi: 00004000 ebp: d1324a00 esp: c0517d8c
ds: 007b es: 007b ss: 0069
Stack: <0>c04d056c 00000000 cfcc6020 00000000 cffa8000 c7f4fc00 d1320000 0000000 00000001 00000000 c0517e2c 80000000 c03acc04 c7f4fc00 c03da724 c0517e70 00000002 cffa8000 c7f4fc00 c04d0540 00000000 c03a2e50 00000002 c0517e70 Call Trace: [<c03acc04>] ip_forward_finish+0x0/0x36 [<c03da724>] ipt_hook+0x1c/0x20 [<c03a2e50>] nf_iterate+0x2c/0x5e [<c03acc04>] ip_forward_finish+0x0/0x36 [<c03acc04>] ip_forward_finish+0x0/0x36 [<c03a2f4b>] nf_hook_slow+0x3c/0xc3 [<c03acc04>] ip_forward_finish+0x0/0x36 [<c03acdd8>] ip_forward+0x19e/0x22e [<c03acc04>] ip_forward_finish+0x0/0x36 [<c03abcf7>] ip_rcv+0x40e/0x48f [<c037dcb5>] netif_receive_skb+0x255/0x294 [<c02e82e6>] tg3_poll+0x532/0x76c [<c037bd82>] net_rx_action+0xaa/0x17c [<c011ea17>] __do_softirq+0x73/0xf0 [<c011ead4>] do_softirq+0x40/0x64 [<c010606b>] do_IRQ+0x1f/0x25 [<c02fc87f>] evtchn_do_upcall+0x60/0x96 [<c0104a2c>] hypervisor_callback+0x2c/0x34 [<c010342e>] xen_idle+0x5e/0x7d [<c0103509>] cpu_idle+0xbc/0xd5 [<c05184e0>] start_kernel+0x344/0x34b Code: 89 44 24 18 89 c6 8b 44 24 40 8b 6c 24 18 03 74 83 0c 03 6c 83 20 c7 44 24
Quoted 3 lines
Quoted 3 lines
EIP;
c03d96a9<ipt_do_table+ad/2d0> <=====esp;
c0517d8c<init_thread_union+1d8c/2000>
Trace; c03acc04 <ip_forward_finish+0/36>
Trace; c03da724 <ipt_hook+1c/20>
Trace; c03a2e50 <nf_iterate+2c/5e>
Trace; c03acc04 <ip_forward_finish+0/36>
Trace; c03acc04 <ip_forward_finish+0/36>
Trace; c03a2f4b <nf_hook_slow+3c/c3>
Trace; c03acc04 <ip_forward_finish+0/36>
Trace; c03acdd8 <ip_forward+19e/22e>
Trace; c03acc04 <ip_forward_finish+0/36>
Trace; c03abcf7 <ip_rcv+40e/48f>
Trace; c037dcb5 <netif_receive_skb+255/294>
Trace; c02e82e6 <tg3_poll+532/76c>
Trace; c037bd82 <net_rx_action+aa/17c>
Trace; c011ea17 <__do_softirq+73/f0>
Trace; c011ead4 <do_softirq+40/64>
Trace; c010606b <do_IRQ+1f/25>
Trace; c02fc87f <evtchn_do_upcall+60/96>
Trace; c0104a2c <hypervisor_callback+2c/34>
Trace; c010342e <xen_idle+5e/7d>
Trace; c0103509 <cpu_idle+bc/d5>
Trace; c05184e0 <start_kernel+344/34b>
Code; c03d96a9 <ipt_do_table+ad/2d0>
00000000 <_EIP>:
Code; c03d96a9 <ipt_do_table+ad/2d0> <=====
0: 89 44 24 18 mov %eax,0x18(%esp) <=====
Code; c03d96ad <ipt_do_table+b1/2d0>
4: 89 c6 mov %eax,%esi
Code; c03d96af <ipt_do_table+b3/2d0>
6: 8b 44 24 40 mov 0x40(%esp),%eax
Code; c03d96b3 <ipt_do_table+b7/2d0>
a: 8b 6c 24 18 mov 0x18(%esp),%ebp
Code; c03d96b7 <ipt_do_table+bb/2d0>
e: 03 74 83 0c add 0xc(%ebx,%eax,4),%esi
Code; c03d96bb <ipt_do_table+bf/2d0>
12: 03 6c 83 20 add 0x20(%ebx,%eax,4),%ebp
Code; c03d96bf <ipt_do_table+c3/2d0>
16: c7 44 24 00 00 00 00 movl $0x0,0x0(%esp)
Code; c03d96c6 <ipt_do_table+ca/2d0>
1d: 00
1 warning issued. Results may not be reliable.
Quoted 4 lines · Matt Ayres
I have been noticing this same problem dozens of times and have finally caught a full trace. I have run it through ksymoops, but there is no
/proc/ksyms. Is there a better method for getting information out of the Code line than using ksymoops in 2.6 kernels?
CONFIG_KALLSYMS will make the kernel decode the oops itself.
Quoted 1 line
The kernel is for Xen, but it does not appear to be related to Xen.
We haven't had problems in that code for ages, so my initial feeling is that it probably is related to Xen. Do you have any other patches applied besides Xen? Please also post the full ruleset you're using and anything else that might appear special about your setup.
Quoted 9 lines · Patrick McHardy
Quoted 4 lines · Matt Ayres
I have been noticing this same problem dozens of times and have finally caught a full trace. I have run it through ksymoops, but there is no
/proc/ksyms. Is there a better method for getting information out of the Code line than using ksymoops in 2.6 kernels?
CONFIG_KALLSYMSwill make the kernel decode the oops itself.
That's odd, I had thought that too. This is what "zcat /proc/config.gz | grep KALL" shows:
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
# CONFIG_KALLSYMS_EXTRA_PASS is not set
I take it my run through ksymoops was of no help in diagnosing the problem? The panic is _always_ in ipt_do_table.
Quoted 8 lines
Quoted 1 line
The kernel is for Xen, but it does not appear to be related to Xen.
We haven't had problems in that code for ages, so my initial feeling is that it probably is related to Xen. Do you have any other patches applied besides Xen? Please also post the full ruleset you're using and anything else that might appear special about your setup.
I had initially sent my traces to the Xen guys. They have not stated it is NOT specific to Xen, just that's it's unlikely. I did not experience the problem with kernel 2.6.12, just with 2.6.16 (up to .13 bugfix release). I have completely disabled all support for SCTP (protocol/netfilter/conntrack) as I know it is still quite buggy. I know Xen touches the network code a lot, but nothing specific to iptables. I had contacted them twice before LKML as I didn't want to post patch specific problems here. I have no other patches applied besides the Xen patch.
My ruleset is pretty bland. 2 rules in the raw table to tell the system to only track my forwarded ports, 2 rules in the nat table for forwarding (intercepting) 2 ports, and then in the FORWARD tables 2 rules per VM to just account traffic.
I've CC'ed xen-devel on this in case they can provide some insight. I am not subscribed to LKML so please make sure to reply to me also in responses.
Thank you,
Matt Ayres
Quoted 9 lines · Matt Ayres
I had initially sent my traces to the Xen guys. They have not stated it is NOT specific to Xen, just that's it's unlikely. I did not experience the problem with kernel 2.6.12, just with 2.6.16 (up to .13 bugfix release). I have completely disabled all support for SCTP (protocol/netfilter/conntrack) as I know it is still quite buggy. I know Xen touches the network code a lot, but nothing specific to iptables. I had contacted them twice before LKML as I didn't want to post patch specific problems here. I have no other patches applied besides the Xen patch.
Quoted 4 lines
My ruleset is pretty bland. 2 rules in the raw table to tell the system to only track my forwarded ports, 2 rules in the nat table for forwarding (intercepting) 2 ports, and then in the FORWARD tables 2 rules per VM to just account traffic.
Can you try using a different NIC?
- James
Signature · 2 lines
James Morris <[email protected]>
Quoted 11 lines · James Morris
Quoted 9 lines · Matt Ayres
I had initially sent my traces to the Xen guys. They have not stated it is NOT specific to Xen, just that's it's unlikely. I did not experience the problem with kernel 2.6.12, just with 2.6.16 (up to .13 bugfix release). I have completely disabled all support for SCTP (protocol/netfilter/conntrack) as I know it is still quite buggy. I know Xen touches the network code a lot, but nothing specific to iptables. I had contacted them twice before LKML as I didn't want to post patch specific problems here. I have no other patches applied besides the Xen patch.
Quoted 7 lines
Quoted 4 lines
My ruleset is pretty bland. 2 rules in the raw table to tell the system to only track my forwarded ports, 2 rules in the nat table for forwarding (intercepting) 2 ports, and then in the FORWARD tables 2 rules per VM to just account traffic.
Can you try using a different NIC?
This happens on 30 different hosts. Using the same kernel I get varying uptime of "hasn't crashed since the upgrade to 2.6.16" to "crashes every day". All are Tyan S2882D boards w/ integrated Tigon3. The trace I posted to this thread indicate tg3, but in many other traces I have the trace doesn't include any driver calls. They all panic in ipt_do_table.
I would have pasted the others, but I didn't save the System.map for
either of them and they are all pretty similar.
Here is another that just crashed:
Quoted 3 lines
Quoted 3 lines
EIP;
c03d96a9<ipt_do_table+ad/2d0> <=====esp;
c0517d8c<init_thread_union+1d8c/2000>
Trace; c03acc04 <ip_forward_finish+0/36>
Trace; c03da724 <ipt_hook+1c/20>
Trace; c03a2e50 <nf_iterate+2c/5e>
Trace; c03acc04 <ip_forward_finish+0/36>
Trace; c03acc04 <ip_forward_finish+0/36>
Trace; c03a2f4b <nf_hook_slow+3c/c3>
Trace; c03acc04 <ip_forward_finish+0/36>
Trace; c03acdd8 <ip_forward+19e/22e>
Trace; c03acc04 <ip_forward_finish+0/36>
Trace; c03abcf7 <ip_rcv+40e/48f>
Trace; c037dcb5 <netif_receive_skb+255/294>
Trace; c02e82e6 <tg3_poll+532/76c>
Trace; c037bd82 <net_rx_action+aa/17c>
Trace; c011ea17 <__do_softirq+73/f0>
Trace; c011ead4 <do_softirq+40/64>
Trace; c010606b <do_IRQ+1f/25>
Trace; c02fc87f <evtchn_do_upcall+60/96>
Trace; c0104a2c <hypervisor_callback+2c/34>
Trace; c010342e <xen_idle+5e/7d>
Trace; c0103509 <cpu_idle+bc/d5>
Trace; c05184e0 <start_kernel+344/34b>
Code; c03d96a9 <ipt_do_table+ad/2d0>
00000000 <_EIP>:
Code; c03d96a9 <ipt_do_table+ad/2d0> <=====
0: 89 44 24 18 mov %eax,0x18(%esp) <=====
Code; c03d96ad <ipt_do_table+b1/2d0>
4: 89 c6 mov %eax,%esi
Code; c03d96af <ipt_do_table+b3/2d0>
6: 8b 44 24 40 mov 0x40(%esp),%eax
Code; c03d96b3 <ipt_do_table+b7/2d0>
a: 8b 6c 24 18 mov 0x18(%esp),%ebp
Code; c03d96b7 <ipt_do_table+bb/2d0>
e: 03 74 83 0c add 0xc(%ebx,%eax,4),%esi
Code; c03d96bb <ipt_do_table+bf/2d0>
12: 03 6c 83 20 add 0x20(%ebx,%eax,4),%ebp
Code; c03d96bf <ipt_do_table+c3/2d0>
16: c7 44 24 00 00 00 00 movl $0x0,0x0(%esp)
Code; c03d96c6 <ipt_do_table+ca/2d0>
1d: 00
<0>Kernel panic - not syncing: Fatal exception in interrupt
Thanks,
Matt Ayres
Quoted 10 lines · Matt Ayres
Quoted 9 lines
Quoted 6 lines
My ruleset is pretty bland. 2 rules in the raw table to tell the system to
only track my forwarded ports, 2 rules in the nat table for forwarding (intercepting) 2 ports, and then in the FORWARD tables 2 rules per VM to just
account traffic.Can you try using a different NIC?
Quoted 7 lines
This happens on 30 different hosts. Using the same kernel I get varying uptime of "hasn't crashed since the upgrade to 2.6.16" to "crashes every day". All are Tyan S2882D boards w/ integrated Tigon3. The trace I posted to this thread indicate tg3, but in many other traces I have the trace doesn't include any driver calls. They all panic in
ipt_do_table. I would have pasted the others, but I didn't save the System.map for either of them and they are all pretty similar.
I'm trying to suggest eliminating this driver & possible interaction with Xen network changes as a cause. If you can find a different type of NIC to plug in and use, or even try and change all of the params for the tg3 with ethtool, it'll help.
Signature · 2 lines
James Morris <[email protected]>
Quoted 18 lines · James Morris
Quoted 8 lines · Matt Ayres
Quoted 8 lines
Quoted 6 lines
My ruleset is pretty bland. 2 rules in the raw table to tell the system to
only track my forwarded ports, 2 rules in the nat table for forwarding (intercepting) 2 ports, and then in the FORWARD tables 2 rules per VM to just
account traffic.Can you try using a different NIC?
Quoted 7 lines
This happens on 30 different hosts. Using the same kernel I get varying uptime of "hasn't crashed since the upgrade to 2.6.16" to "crashes every day". All are Tyan S2882D boards w/ integrated Tigon3. The trace I posted to this thread indicate tg3, but in many other traces I have the trace doesn't include any driver calls. They all panic in
ipt_do_table. I would have pasted the others, but I didn't save the System.map for either of them and they are all pretty similar.
Quoted 2 lines
I'm trying to suggest eliminating this driver & possible interaction with Xen network changes as a cause. If you can find a different type of NIC to plug in and use, or even try and change all of the params for the tg3 with ethtool, it'll help.
Hi,
Thank you for the assistance. Which parameters do you suggest changing?
TSO/flow control off?
Here is my ruleset for those interested:
# iptables -t raw -L -v Chain OUTPUT (policy ACCEPT 27441 packets, 4832K bytes) pkts bytes target prot opt in out source destination
Chain PREROUTING (policy ACCEPT 195M packets, 156G bytes)
pkts bytes target prot opt in out source
destination
1332K 144M NOTRACK !tcp -- any any anywhere anywhere 54 5293 ACCEPT tcp -- any any anywhere
anywhere tcp dpt:7373
4564 223K ACCEPT tcp -- any any anywhere
anywhere tcp dpt:7322
194M 156G NOTRACK tcp -- any any anywhere
anywhere tcp dpt:!7373
194M 156G NOTRACK tcp -- any any anywhere
anywhere tcp dpt:!7322
# iptables -t nat -L -v Chain OUTPUT (policy ACCEPT 2114 packets, 155K bytes) pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 2114 packets, 155K bytes)
pkts bytes target prot opt in out source
destination
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source
destination
6 344 DNAT tcp -- eth0 any anywhere
anywhere tcp dpt:7373 to:host.ip.address:443
8 408 DNAT tcp -- eth0 any anywhere
anywhere tcp dpt:7322 to:host.ip.address:22
iptables -L -v just shows 2 rules per Virtual Machine for accounting. This averages about 100 rules in the FORWARD chain. Example:
# iptables -L -v
Chain FORWARD (policy ACCEPT 195M packets, 156G bytes)
pkts bytes target prot opt in out source
destination
0 0 all -- any any xx.xx.xx.xx
anywhere
0 0 all -- any any anywhere
xx.xx.xx.xxQuoted 10 lines · Matt Ayres
Quoted 5 lines
I'm trying to suggest eliminating this driver & possible interaction with Xen network changes as a cause. If you can find a different type of NIC to plug in and use, or even try and change all of the params for the tg3 with ethtool, it'll help.
Hi,
Thank you for the assistance. Which parameters do you suggest changing? TSO/flow control off?
Yep, anything.
Quoted 2 lines
iptables -L -v just shows 2 rules per Virtual Machine for accounting. This averages about 100 rules in the FORWARD chain. Example:
Do you know if the problem starts appearing after a certain number of hosts?
- James
Signature · 2 lines
James Morris <[email protected]>
Quoted 7 lines · James Morris
Quoted 5 lines · Matt Ayres
Quoted 5 lines
I'm trying to suggest eliminating this driver & possible interaction with Xen network changes as a cause. If you can find a different type of NIC to plug in and use, or even try and change all of the params for the tg3 with ethtool, it'll help.
Quoted 6 lines
Quoted 4 lines
Hi,
Thank you for the assistance. Which parameters do you suggest changing? TSO/flow control off?
Yep, anything.
Ok, "ethtool -K eth0 rx off tx off sg off tso off" should have turned it
all off.
Quoted 6 lines
Quoted 3 lines
iptables -L -v just shows 2 rules per Virtual Machine for accounting. This averages about 100 rules in the FORWARD chain. Example:
Do you know if the problem starts appearing after a certain number of hosts?
No... I have some servers that are running just 2.6.16-xen (no bugfix patches) for 30 days without a problem, some of these have rulesets larger than the ones that crash daily. I'd estimate this affects 90% of my servers, just some reboot daily and others can make it to 7-10 days.
Thanks,
Matt
Quoted 12 lines · Matt Ayres
Quoted 9 lines · James Morris
Quoted 7 lines · Matt Ayres
Quoted 5 lines
I'm trying to suggest eliminating this driver & possible interaction with Xen network changes as a cause. If you can find a different type of NIC to plug in and use, or even try and change all of the params for the tg3 with ethtool, it'll help.
Hi,
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Thank you for the assistance. Which parameters do you suggest changing? TSO/flow control off?
Yep, anything.
Ok, "ethtool -K eth0 rx off tx off sg off tso off" should have turned it
all off.
I think I confirmed the NIC is not the source of the problem. A few of my servers have e100/tulip NIC's due to a bug with the chipset of the on-board TG3 cards firmware and TSO. These servers that use the e100/tulip drivers also experience the ipt_do_table bug.
Thanks,
Matt
Quoted 4 lines · Matt Ayres
I think I confirmed the NIC is not the source of the problem. A few of my servers have e100/tulip NIC's due to a bug with the chipset of the on-board TG3 cards firmware and TSO. These servers that use the e100/tulip drivers also experience the
ipt_do_tablebug.
There is an identical report in the netfilter bugzilla, also crashes (on x86_64) in ipt_do_table with Xen. I haven't heard anything of similar crashes without Xen, so I doubt that the bug is in the netfilter code.
Quoted 6 lines · Patrick McHardy
Quoted 4 lines · Matt Ayres
I think I confirmed the NIC is not the source of the problem. A few of my servers have e100/tulip NIC's due to a bug with the chipset of the on-board TG3 cards firmware and TSO. These servers that use the e100/tulip drivers also experience the
ipt_do_tablebug.
Quoted 6 lines
There is an identical report in the netfilter bugzilla, also crashes (on
x86_64) inipt_do_tablewith Xen. I haven't heard anything of similar crashes without Xen, so I doubt that the bug is in the netfilter code.
Yep... too coincidental. I'd say it has _something_ to do with Xen. I've been doing different things on my side to try to reduce the severity of the problem, but I'd really like to hear what the Xen guys have to say about this now..
Thanks,
Matt
Quoted 7 lines · Matt Ayres
Quoted 5 lines
There is an identical report in the netfilter bugzilla, also crashes (on
x86_64) inipt_do_tablewith Xen. I haven't heard anything of similar crashes without Xen, so I doubt that the bug is in the netfilter code.
https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=478Yep... too coincidental. I'd say it has _something_ to do with Xen. I've been doing different things on my side to try to reduce the severity of the problem, but I'd really like to hear what the Xen guys have to say about this now..
If you can provide a vmlinux image and a backtrace we'll take a look.
-- Keir
Quoted 17 lines · Matt Ayres
Quoted 13 lines · Patrick McHardy
Quoted 4 lines · Matt Ayres
I think I confirmed the NIC is not the source of the problem. A few of my servers have e100/tulip NIC's due to a bug with the chipset of the on-board TG3 cards firmware and TSO. These servers that use the e100/tulip drivers also experience the
ipt_do_tablebug.There is an identical report in the netfilter bugzilla, also crashes (on
x86_64) inipt_do_tablewith Xen. I haven't heard anything of similar crashes without Xen, so I doubt that the bug is in the netfilter code.
Quoted 7 lines
Yep... too coincidental. I'd say it has _something_ to do with Xen. I've been doing different things on my side to try to reduce the severity of the problem, but I'd really like to hear what the Xen guys have to say about this now..
Maybe this helps: there is not too much the Xen code could be doing wrong here. If I read your crash correctly it happend in the FORWARD chain, which could mean that the outgoing device (probably the Xen virtual network driver) has some bugs, but iptables really only cares about the names at this point, which practically can't be bogus. The only other thing I can imagine is that something is wrong with the per-CPU copy of the ruleset, i.e. either smp_processor_id is returning garbage or for_each_possible_cpu misses a CPU during initialization. I have no idea if Xen really does touch this code, but other than that I don't really see what it could break.
Quoted 5 lines · Patrick McHardy
The only other thing I can imagine is that something is wrong with the per-CPU copy of the ruleset, i.e. either
smp_processor_idis returning garbage orfor_each_possible_cpumisses a CPU during initialization. I have no idea if Xen really does touch this code, but other than that I don't really see what it could break.
Of the options you consider, this sounds most likely. Really we need some more info from a crash: I'd like to get disassembly from a vmlinux image if that's possible, Matt.
Thanks,
Keir
Quoted 11 lines · Keir Fraser
Quoted 5 lines · Patrick McHardy
The only other thing I can imagine is that something is wrong with the per-CPU copy of the ruleset, i.e. either
smp_processor_idis returning garbage orfor_each_possible_cpumisses a CPU during initialization. I have no idea if Xen really does touch this code, but other than that I don't really see what it could break.Of the options you consider, this sounds most likely. Really we need some more info from a crash: I'd like to get disassembly from a vmlinux image if that's possible, Matt.
I have an un-stripped vmlinux built with kernel debugging and the corresponding System.map. I will be sending these to you privately shortly. You can see the multiple traces sent to this list.
It appears having the bandwidth accounting being performed by count rules in the FORWARD chain is causing it for my setup. I suppose I could optimize this to make the kernel spend as little time in ipt_do_table in regards to the FORWARD chain. I flushed the FORWARD chain (normally 100-120 rules) and have not experienced any crashes so far... disabling bandwidth monitoring is by no means a long term fix.
It might be more generic in the symptoms, perhaps just any chain with many rules and lots of traffic or It's just the FORWARD one that seems to be doing it for me as that is where ipt_do_table spends most of it's time.
Thanks,
Matt Ayres
Quoted 10 lines · Patrick McHardy
Maybe this helps: there is not too much the Xen code could be doing wrong here. If I read your crash correctly it happend in the FORWARD chain, which could mean that the outgoing device (probably the Xen virtual network driver) has some bugs, but iptables really only cares about the names at this point, which practically can't be bogus. The only other thing I can imagine is that something is wrong with the per-CPU copy of the ruleset, i.e. either
smp_processor_idis returning garbage orfor_each_possible_cpumisses a CPU during initialization. I have no idea if Xen really does touch this code, but other than that I don't really see what it could break.
Having looked at disassembly, the fault happens when accessing e->ip.invflags in ip_packet_match() inlined inside ipt_do_table().
e = private->entries[smp_processor_id()] +
private->hook_entry[NF_IP_FORWARD]
smp_processor_id() should be 0 (since the oops appears to occur on cpu0) and presumably all the ipt_entry structures are static once set up. Since this crash happens on a common path in ipt_do_table(), and since it happens only after the system has been up a while (I believe?), it rather looks as though something has either corrupted a pointer or unmapped memory from under iptables' feet.
-- Keir
Quoted 14 lines · Keir Fraser
Quoted 10 lines · Patrick McHardy
Maybe this helps: there is not too much the Xen code could be doing wrong here. If I read your crash correctly it happend in the FORWARD chain, which could mean that the outgoing device (probably the Xen virtual network driver) has some bugs, but iptables really only cares about the names at this point, which practically can't be bogus. The only other thing I can imagine is that something is wrong with the per-CPU copy of the ruleset, i.e. either
smp_processor_idis returning garbage orfor_each_possible_cpumisses a CPU during initialization. I have no idea if Xen really does touch this code, but other than that I don't really see what it could break.
Quoted 7 lines
Having looked at disassembly, the fault happens when accessing
e->ip.invflagsinip_packet_match()inlined insideipt_do_table().e = private->entries[smp_processor_id()] +
private->hook_entry[NF_IP_FORWARD]
smp_processor_id()should be 0 (since the oops appears to occur on cpu0) and presumably all theipt_entrystructures are static once set up. Since this crash happens on a common path inipt_do_table(), and since it happens only after the system has been up a while (I believe?), it rather looks as though something has either corrupted a pointer or unmapped memory from under iptables' feet.
As the concerned user, what does this mean to me? It will only affect SMP systems? It is a bug in Xen or netfilter?
I'd just like to understand what is going on.
Thanks,
Matt
Quoted 6 lines · Matt Ayres
Quoted 4 lines
Having looked at disassembly, the fault happens when accessing
e->ip.invflagsinip_packet_match()inlined insideipt_do_table().
e = private->entries[smp_processor_id()] +
private->hook_entry[NF_IP_FORWARD]smp_processor_id()should be 0 (since the oops appears to occur on cpu0) and presumably all theipt_entrystructures are static once set up. Since this crash happens on a common path inipt_do_table(), and since it happens only after the system has been up a while (I believe?), it rather looks as though something has either corrupted a pointer or unmapped memory from under iptables' feet.As the concerned user, what does this mean to me? It will only affect SMP systems? It is a bug in Xen or netfilter?
Probably a Xen bug, but if so then it's basically a memory corruption. It's weird it would hit the iptables rules every time though, and not be a more random crash. This might well need reproducing in a developer test-machine environment to stand a chance of tracking down.
-- Keir
Quoted 1 line
I'd just like to understand what is going on.
Gerd Hoffmann
Acked-byQuoted 4 lines
Quoted 2 lines
As the concerned user, what does this mean to me? It will only affect SMP systems? It is a bug in Xen or netfilter?
Probably a Xen bug, but if so then it's basically a memory corruption.
Might also be a netfilter bug which is simply triggered by the way how xen manages the memory. Due to ballooning you can have holes in memory, so out-of-range access may fault with xen whereas it will go unnoticed with normal kernels.
One such beast is in bridging netfilter code, additionally it triggers with certain ethernet cards only, patch below. Pinned down last week ;)
cheers,
Gerd
Subject: nf_bridge: ethernet header is 14 not 16 bytes
From: [email protected]
References: 150410
The bridge netfilter code saves two more bytes that it should. In most cases it doesn't hurt because many drivers use NET_IP_ALIGN to make the IP header aligned, so there are two extra bytes head room available.
Some drivers don't do that though (sky2 for example), so copying accesses data outside the skbuff data allocation. On xen kernels this can kill the machine with a page fault due to the way how skbuffs are allocated and the memory is managed.
include/linux/netfilter_bridge.h +2 -2
@@ -73,14 +73,14 @@ void nf_bridge_maybe_copy_header(structmemcpy (skb->data -18 , skb->nf_bridge->data,18 );skb_push (skb,4 ); }else -memcpy (skb->data -16 , skb->nf_bridge->data,16 ); +memcpy (skb->data -14 , skb->nf_bridge->data,14 ); } }static inline void nf_bridge_save_header (struct sk_buff *skb) { -int header_size =16 ; +int header_size =14 ;if (skb->protocol ==__constant_htons (ETH_P_8021Q )) header_size =18 ;
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel