kernel freeze issue
summary of current configuration:
+----- ext_if(eth1) 3c905TX ------ 34Mbit uplink
|
|
|
+------------+
| filter +
+------------+
| | |
| | +--- lan_if(eth5) rtl8169 - vlan1 ------ users/servers
| |
| +------ adm_if(eth0) 3c905TX - vlan2 ------ log server
|
+--------- core_if(eth3) 3c940
+inp_if(eth3.3) - vlan 3 +
+out_if(eth3.4) - vlan 4 +
|
|(cross-link cable)
|
+------ core_if(eth0) 3c940 ----------+
|
+-----------+
| foo +
+-----------+
|
+------ adm_if(eth1) rtl8139 - vlan2filter:
arp proxy based
route - policy routing - between local-domain1(C),local-domain2(2*C),uplink,foo(on failover this skipped)
packet filter(netfilter)
- traffic accounting(ipt_account),flood/portscan protection
- packet filter
- TTL inc
- ipset's for extra port configurations
- ipt_condition(failover control)
- we have 2 domain's so it sends redirects for the hosts spoofing that it's our router(ipt_IPALTER)
foo:
not configured because of the freezes...
problem:
filter is freezes in random intervalls(30m - 6day) - on-board watchdog(i8xx) reboot's the system i've tryed many things, remove my custom patches...but it won't help ;)
in the kernel trace i've last seen(i've a blurry image)
the kernel remove some packages from the boomerang interface
ip_rcv_finish, etc..
ipt_do_table is the last in the call trace...
notice:
crash happens when many of our beloved users use p2p software(this is also a tip)
next try:
place a cisco to monitor ext_if and lan_if with tcpdump, open another file every 10m and when filter freezes i maybe have the packet that caused the freeze (small chance - but possible ;)
my tips was:
ipt_condition - in pom it's <2.6.0 but i've read the code, and i think it's safe to use
ipt_IPALTER - w/o it also freezes, so this isn't the problem
ipset - i've a small patch on it...to enable inverted portmaps, i think it's safe
i use portmap,ipmap,macipmap from it
NAPI - yesterday i disabled it...since then no freezes
ipt_TCPMSS - it wrote some warnings in dmesg, about packet size<64 - i've moved another rule before it
-p tcp --tcp-option ! 2 -j DROP
boomerang - maybe the driver is a bit broken
pom_patches - TTL set connmark CONNMARK account condition limit
SMP - maybe, i haven't disabled it yet
today's surprise:
ip l s eth0 promisc on ==> freeze, without any trace ;)
versions:
Linux filter 2.6.14-alt #5 SMP Tue Nov 8 16:40:49 CET 2005 i686 GNU/Linux iptables-1.3.4
pom-20051031
ipset-2.2.6
some info about the system can be downloaded from
http://152.66.235.5/info-filter.tgz
this is my worst nightmare, any suggestions? ;)
Hi,
Quoted 8 lines · Zoltan Nagy
problem:
filter is freezes in random intervalls(30m - 6day) - on-board
watchdog(i8xx) reboot's the system i've tryed many things, remove my custom patches...but it won't help ;) in the kernel trace i've last seen(i've a blurry image)
the kernel remove some packages from the boomerang interfaceip_rcv_finish, etc..
ipt_do_tableis the last in the call trace...
Please try to get at least a backtrace, withouth that probably nobody will
be able to help you. The above information (ip_rcv_finish and ipt_do_table being present in the trace) is completely useless, as processing _any_ IPv4 packet could lead to receiving such a backtrace...
Quoted 6 lines
my tips was:
ipt_condition- in pom it's <2.6.0 but i've read the code, and i think
it's safe to usetoday's surprise:
ip l s eth0 promisc on ==> freeze, without any trace ;)
You could still try getting a backtrace with SysRq + P.
Signature · 2 lines
Regards, Krisztian Kovacs
Quoted 6 lines · KOVACS Krisztian
Quoted 2 lines
today's surprise:
ip l s eth0 promisc on ==> freeze, without any trace ;)You could still try getting a backtrace with SysRq + P.
tried, sysrq not responding...
regards
Quoted 7 lines · Zoltan Nagy
Quoted 4 lines · KOVACS Krisztian
Quoted 2 lines
today's surprise:
ip l s eth0 promisc on ==> freeze, without any trace ;)
Quoted 6 lines
Quoted 2 lines
You could still try getting a backtrace with SysRq + P.
tried, sysrq not responding...
regards
try booting with these kernel parameters nmi_watchdog=1 console=ttyS0,9600n8 console=/dev/tty0
it will setup nmi_watchdog and will also send all you syslog stuff out to the serial port.
Then connect serial port to another machine to capture the oops/crash
Quoted 3 lines
last night, netconsole captured a trace ;)
i'm not an expert in interpreting kernel traces
any suggestions?
regard.kirk
Nov 10 19:11:45 filter netconsole: network logging started Nov 10 23:41:47 filter Unable to handle kernel paging request Nov 10 23:41:47 filter at virtual address e0fd5000 Nov 10 23:41:47 filter printing eip: Nov 10 23:41:47 filter c03fb036 Nov 10 23:41:47 filter *pde = 1fe27067 Nov 10 23:41:47 filter *pte = 00000000 Nov 10 23:41:47 filter Oops: 0000 [#1] Nov 10 23:41:47 filter SMP Nov 10 23:41:47 filter Nov 10 23:41:47 filter Modules linked in: Nov 10 23:41:47 filter netconsole Nov 10 23:41:47 filter i8xx_tco Nov 10 23:41:47 filter Nov 10 23:41:47 filter CPU: 0 Nov 10 23:41:47 filter EIP: 0060:[<c03fb036>] Not tainted VLI Nov 10 23:41:47 filter EFLAGS: 00010213 (2.6.14-alt) Nov 10 23:41:47 filter EIP is at match+0x586/0x8b0 Nov 10 23:41:47 filter eax: 00000001 ebx: 00000800 ecx: 00000800 edx: 00000000 Nov 10 23:41:47 filter esi: e0fd3000 edi: e0fd3000 ebp: e0fbb000 esp: c04f7d8c Nov 10 23:41:47 filter ds: 007b es: 007b ss: 0068 Nov 10 23:41:47 filter Process swapper (pid: 0, threadinfo=c04f6000 task=c048cbe0) Nov 10 23:41:47 filter Nov 10 23:41:47 filter Stack: Nov 10 23:41:47 filter 979b87c9 Nov 10 23:41:47 filter 00000800 Nov 10 23:41:47 filter 00000048 Nov 10 23:41:47 filter e10268c6 Nov 10 23:41:47 filter c012a46d Nov 10 23:41:47 filter c04b44a0 Nov 10 23:41:47 filter 00000282 Nov 10 23:41:47 filter 00000000 Nov 10 23:41:47 filter Nov 10 23:41:47 filter Nov 10 23:41:47 filter 00000020 Nov 10 23:41:47 filter 000002ee Nov 10 23:41:47 filter e0fc8500 Nov 10 23:41:47 filter 0000001e Nov 10 23:41:47 filter 085975c0 Nov 10 23:41:47 filter e0fbb0d8 Nov 10 23:41:47 filter e0fbd000 Nov 10 23:41:47 filter e0fd6000 Nov 10 23:41:47 filter Nov 10 23:41:47 filter Nov 10 23:41:47 filter 000007ff Nov 10 23:41:47 filter 000007fe Nov 10 23:41:47 filter 000007fe Nov 10 23:41:47 filter e0fd3000 Nov 10 23:41:47 filter 6c025a8c Nov 10 23:41:47 filter 6bf3acd9 Nov 10 23:41:47 filter 003d26b1 Nov 10 23:41:47 filter 00000000 Nov 10 23:41:47 filter Nov 10 23:41:47 filter Call Trace: Nov 10 23:41:47 filter [<c012a46d>] Nov 10 23:41:47 filter notifier_call_chain+0x2d/0x50 Nov 10 23:41:47 filter Nov 10 23:41:47 filter [<c03f1cd8>] Nov 10 23:41:47 filter ipt_do_table+0x1d8/0x370 Nov 10 23:41:47 filter Nov 10 23:41:47 filter [<c03b3260>] Nov 10 23:41:47 filter ip_rcv_finish+0x0/0x270 Nov 10 23:41:47 filter Nov 10 23:41:47 filter [<c03f7ef6>] Nov 10 23:41:47 filter ipt_hook+0x36/0x40 Nov 10 23:41:47 filter Nov 10 23:41:47 filter [<c040cfc8>] Nov 10 23:41:47 filter nf_iterate+0x78/0x90 Nov 10 23:41:47 filter Nov 10 23:41:47 filter [<c03b3260>] Nov 10 23:41:47 filter ip_rcv_finish+0x0/0x270 Nov 10 23:41:47 filter Nov 10 23:41:47 filter [<c03b3260>] Nov 10 23:41:47 filter ip_rcv_finish+0x0/0x270 Nov 10 23:41:47 filter Nov 10 23:41:47 filter [<c040d04e>] Nov 10 23:41:47 filter nf_hook_slow+0x6e/0x110 Nov 10 23:41:47 filter Nov 10 23:41:47 filter [<c03b3260>] Nov 10 23:41:47 filter ip_rcv_finish+0x0/0x270 Nov 10 23:41:48 filter Nov 10 23:41:48 filter [<c03b3260>] Nov 10 23:41:48 filter ip_rcv_finish+0x0/0x270 Nov 10 23:41:48 filter Nov 10 23:41:48 filter [<c03b2fcb>] Nov 10 23:41:48 filter ip_rcv+0x40b/0x530 Nov 10 23:41:48 filter Nov 10 23:41:48 filter [<c03b3260>] Nov 10 23:41:48 filter ip_rcv_finish+0x0/0x270 Nov 10 23:41:48 filter Nov 10 23:41:48 filter [<c039cd78>] Nov 10 23:41:48 filter netif_receive_skb+0x1c8/0x270 Nov 10 23:41:48 filter Nov 10 23:41:48 filter [<c039ceb2>] Nov 10 23:41:48 filter process_backlog+0x92/0x120 Nov 10 23:41:48 filter Nov 10 23:41:48 filter [<c039cfd5>] Nov 10 23:41:48 filter net_rx_action+0x95/0x190 Nov 10 23:41:48 filter Nov 10 23:41:48 filter [<c0121c16>] Nov 10 23:41:48 filter __do_softirq+0xd6/0xf0 Nov 10 23:41:48 filter Nov 10 23:41:48 filter [<c0121c65>] Nov 10 23:41:48 filter do_softirq+0x35/0x40 Nov 10 23:41:48 filter Nov 10 23:41:48 filter [<c010522e>] Nov 10 23:41:48 filter do_IRQ+0x1e/0x30 Nov 10 23:41:48 filter Nov 10 23:41:48 filter [<c01039e6>] Nov 10 23:41:48 filter common_interrupt+0x1a/0x20 Nov 10 23:41:48 filter Nov 10 23:41:48 filter [<c0100db0>] Nov 10 23:41:48 filter default_idle+0x0/0x30 Nov 10 23:41:48 filter Nov 10 23:41:48 filter [<c0100dd3>] Nov 10 23:41:48 filter default_idle+0x23/0x30 Nov 10 23:41:48 filter Nov 10 23:41:48 filter [<c0100e90>] Nov 10 23:41:48 filter cpu_idle+0x70/0x80 Nov 10 23:41:48 filter Nov 10 23:41:48 filter [<c04f899f>] Nov 10 23:41:48 filter start_kernel+0x15f/0x180 Nov 10 23:41:48 filter Nov 10 23:41:48 filter [<c04f8390>] Nov 10 23:41:48 filter unknown_bootoption+0x0/0x1e0 Nov 10 23:41:48 filter Nov 10 23:41:48 filter Code: Nov 10 23:41:48 filter 84 Nov 10 23:41:48 filter 80 Nov 10 23:41:48 filter 01 Nov 10 23:41:48 filter 00 Nov 10 23:41:48 filter 00 Nov 10 23:41:48 filter 8b Nov 10 23:41:48 filter 5c Nov 10 23:41:48 filter 24 Nov 10 23:41:48 filter 48 Nov 10 23:41:48 filter 8b Nov 10 23:41:48 filter 0d Nov 10 23:41:48 filter 4c Nov 10 23:41:48 filter f5 Nov 10 23:41:48 filter 55 Nov 10 23:41:48 filter c0 Nov 10 23:41:48 filter 8b Nov 10 23:41:48 filter 74 Nov 10 23:41:48 filter 24 Nov 10 23:41:48 filter 4c Nov 10 23:41:48 filter 43 Nov 10 23:41:48 filter 89 Nov 10 23:41:48 filter d8 Nov 10 23:41:48 filter 99 Nov 10 23:41:48 filter f7 Nov 10 23:41:48 filter f9 Nov 10 23:41:48 filter 83 Nov 10 23:41:48 filter 3c Nov 10 23:41:48 filter 96 Nov 10 23:41:48 filter ff Nov 10 23:41:48 filter 0f Nov 10 23:41:48 filter 84 Nov 10 23:41:48 filter be Nov 10 23:41:48 filter 00 Nov 10 23:41:48 filter last message repeated 2 times Nov 10 23:41:48 filter 89 Nov 10 23:41:48 filter 4c Nov 10 23:41:48 filter 24 Nov 10 23:41:48 filter 04 Nov 10 23:41:48 filter 8b Nov 10 23:41:48 filter 7c Nov 10 23:41:48 filter 24 Nov 10 23:41:48 filter 4c Nov 10 23:41:48 filter b> Nov 10 23:41:48 filter 14 Nov 10 23:41:48 filter 9f Nov 10 23:41:48 filter 8d Nov 10 23:41:48 filter 04 Nov 10 23:41:48 filter d5 Nov 10 23:41:48 filter 00 Nov 10 23:41:48 filter last message repeated 3 times Nov 10 23:41:48 filter 29 Nov 10 23:41:48 filter d0 Nov 10 23:41:48 filter 8b Nov 10 23:41:48 filter 54 Nov 10 23:41:48 filter 24 Nov 10 23:41:48 filter 38 Nov 10 23:41:48 filter 8b Nov 10 23:41:48 filter 04 Nov 10 23:41:48 filter 82 Nov 10 23:41:48 filter 89 Nov 10 23:41:48 filter 04 Nov 10 23:41:48 filter Nov 10 23:41:48 filter Nov 10 23:41:48 filter Kernel panic - not syncing: Fatal exception in interrupt Nov 10 23:41:48 filter
General comment on the format: it would help if you would put the text and list portions in standard (>80 columns) format. ASCII drawings are harder to fit sometimes, but even those will come across better under 80 columns. I use a GUI MUA so I was able to resize to view your post properly, but remember, much of your audience might be using mutt.
Quoted 4 lines · Zoltan Nagy
problem:
filter is freezes in random intervalls(30m - 6day) - on-board
watchdog(i8xx) reboot's the system i've tryed many things, remove my custom patches...but it won't help ;) in the kernel trace i've last
I would suspect hardware first.
Quoted 2 lines
crash happens when many of our beloved users use p2p software(this is also a tip)
snip
Quoted 2 lines
today's surprise:
ip l s eth0 promisc on ==> freeze, without any trace ;)
Did you try swapping out that NIC?
Quoted 3 lines
some info about the system can be downloaded from
http://152.66.235.5/info-filter.tgz
this is my worst nightmare, any suggestions? ;)
You do have my sympathy, but I didn't download the file. I would be likely to do that if it was something I could view in the browser, like text or HTML.
Signature · 2 lines
mail to this address is discarded unless "/dev/rob0"
or "not-spam" is in Subject: headerhi
Quoted 6 lines · /dev/rob0 wrote:
General comment on the format: it would help if you would put the text and list portions in standard (>80 columns) format. ASCII drawings are harder to fit sometimes, but even those will come across better under 80 columns. I use a GUI MUA so I was able to resize to view your post properly, but remember, much of your audience might be using mutt.
ok, i use 212x83 terminals ;)
Quoted 9 lines · Zoltan Nagy
Quoted 5 lines
problem:
filter is freezes in random intervalls(30m - 6day) - on-board
watchdog(i8xx) reboot's the system i've tryed many things, remove my custom patches...but it won't help ;) in the kernel trace i've lastI would suspect hardware first.
another part of the story...problem started with a rtl8139 if on the side of the micro it have been failed time to time, the transciever has gone mad, and packet traversal has become 20% ;) so...we tried a 3c940 to put it on the router side - the card don't take up the link...
we tried rtl8169 - like the 940
so, we bought 2 brand new 3c905tx, i'm not sure(because at this time was a hw change, and i've rewrote nearly everything in a new style... so i don't suspected hw issue, but i will try an intel100 ;)
Quoted 5 lines
Did you try swapping out that NIC?
yes ;)
Quoted 3 lines
You do have my sympathy, but I didn't download the file. I would be likely to do that if it was something I could view in the browser, like text or HTML.
i think people like to read source in there own ways, i prefer use less.. /there's a bit too many files in it/
...and i hate viewing sources in browser - i hate white bg color ;)
regards.kirk