Troubles with NFS & ip_conntrack: packets go to wrong mac
Subsystems (3)
I've been experiencing strange problems with nfs and ip_conntrack for a while now, unfortunately noone so far was able to help. The problem occurs when ip_conntrack is loaded on the nfs server. nfs reads hang and the clients start logging
UDP: short packet: 192.168.0.1:0 0/120 to 192.168.0.23:0
UDP: short packet: 192.168.0.1:6439 28562/120 to 192.168.0.23:60558
There are two ways to make it work: 1. remove ip_conntrack or 2. set mtu to 1484 on the nfs server. One suspicion was ip_conntrack breaking udp path mtu discovery since it seems to defragment packets with DF|MF and refragment them (with possibly different mtu) at POSTROUTING. This doesn't seem to be the problem, but i noted the nfs server sends out fragments with wrong destination mac.
This is a packet captured on 192.168.0.23:
0:e0:7d:74:ab:cc 0:e0:7d:74:ab:cd 0800 1514: 192.168.0.1 > 192.168.0.223: (frag 44777:1480@4440+) (ttl 64, len 1500, bad cksum 2294!)
This happens every 1-30 seconds. I never saw more than one misdirected fragment per packet.
Relevent parts from neighbour table:
192.168.0.223 dev eth0 lladdr 00:e0:29:3c:c1:c9 nud reachable 192.168.0.23 dev eth0 lladdr 00:e0:7d:74:ab:cd nud reachable
On 192.168.0.223 packets for 192.168.0.23 show up. Both clients time out during reassembly.
I placed some printks though the netfilter code and ip_output.c but couldn't find any further pointers. I looked for broken checksums (something seems to alter the ip after checksumming)
in ip_finish_output2, but everything is ok there. I can't see anything netfilter related touching packets after that. Any help and/or pointers where to look further would be appreciated.
Regards,
Patrick
Quoted 4 lines · Patrick McHardy
I've been experiencing strange problems with nfs and
ip_conntrackfor a while now, unfortunately noone so far was able to help. The problem occurs whenip_conntrackis loaded on the nfs server. nfs reads hang and the clients start logging
Just for consistency: Can you please report this to
http://bugzilla.netfilter.org/ ?
Addiditional information like what kind of machine (UP/SMP), and network board might be interesting.
Quoted 2 lines
Regards,
Patrick
Signature · 4 lines
- Harald Welte <[email protected]> http://www.gnumonks.org/ ============================================================================ "If this were a dictatorship, it'd be a heck of a lot easier, just so long as I'm the dictator." -- George W. Bush Dec 18, 2000
Some of the information in my last mail was wrong, i'll cut-and-paste the information entered into the netfilter bugtracking system, maybe someone has a suggestion. The important part is the packets checksum turns out to be right if the destination ip is replaced by the ip the mac belongs to, so the mac is correct.
--cut-n-paste--
With ip_conntrack loaded on the nfs server, client reads time out. Clients start logging:
UDP: short packet: 192.168.0.1:0 0/120 to 192.168.0.23:0
UDP: short packet: 192.168.0.1:6439 28562/120 to 192.168.0.23:60558
tcpdump shows corrupted packets:
0:e0:7d:74:ab:cc 0:e0:7d:74:ab:cd 0800 1514: 192.168.0.1 > 192.168.0.223: (frag 42878:1480@4440+) (ttl 64, len 1500, bad cksum 29ff!) 0:e0:7d:74:ab:cc 0:e0:7d:74:ab:cd 0800 1514: 192.168.0.1 > 192.168.0.223: (frag 42879:1480@4440+) (ttl 64, len 1500, bad cksum 29fe!)
0:e0:7d:74:ab:cd is not the mac of 192.168.0.223 but 192.168.0.23. Both are nfs-clients. If more nfs-clients show up more incorrect destination ips appear. The packets checksum is correct if the (incorrect) destination ip is replaced by the correct destination ip. Only single fragments have incorrect destination, the remaining fragments of a packet are fine. Packets verified (checksum) in ip_finish_output2 show no corruption.
Neighbour table of nfs server:
192.168.0.223 dev eth0 lladdr 00:e0:29:3c:c1:c9 nud reachable 192.168.0.23 dev eth0 lladdr 00:e0:7d:74:ab:cd nud reachable
The problem goes away as soon as ip_conntrack is unloaded. Another possibility is to set the interface mtu to 1486 on the nfs server.
CPU: AMD-K6(tm) 3D processor, 256MB RAM
Kernel: 2.4.21-pre3 with few netfilter patches applied, problem also exists in vanilla kernel, first noticed around 2.4.17
lspci:
00:00.0 Host bridge: Acer Laboratories Inc. [ALi] M1541 (rev 04) 00:01.0 PCI bridge: Acer Laboratories Inc. [ALi] M5243 (rev 04) 00:07.0 ISA bridge: Acer Laboratories Inc. [ALi] M1533 PCI to ISA Bridge [Aladdin IV] (rev c3)
00:08.0 VGA compatible unclassified device: S3 Inc. 86c864 [Vision 864 DRAM] vers 0 00:09.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139 (rev 10) 00:0a.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139 (rev 10) 00:0f.0 IDE interface: Acer Laboratories Inc. [ALi] M5229 IDE (rev c2)
--end-cut-n-paste--
Bye,
Patrick
Quoted 31 lines · Patrick McHardy
I've been experiencing strange problems with nfs and
ip_conntrackfor a while now, unfortunately noone so far was able to help. The problem occurs whenip_conntrackis loaded on the nfs server. nfs reads hang and the clients start loggingUDP: short packet: 192.168.0.1:0 0/120 to 192.168.0.23:0
UDP: short packet: 192.168.0.1:6439 28562/120 to 192.168.0.23:60558There are two ways to make it work: 1. remove
ip_conntrackor 2. set mtu to 1484 on the nfs server. One suspicion wasip_conntrackbreaking udp path mtu discovery since it seems to defragment packets with DF|MF and refragment them (with possibly different mtu) at POSTROUTING. This doesn't seem to be the problem, but i noted the nfs server sends out fragments with wrong destination mac.This is a packet captured on 192.168.0.23:
0:e0:7d:74:ab:cc 0:e0:7d:74:ab:cd 0800 1514: 192.168.0.1 > 192.168.0.223: (frag 44777:1480@4440+) (ttl 64, len 1500, bad cksum 2294!)This happens every 1-30 seconds. I never saw more than one misdirected fragment per packet.
Relevent parts from neighbour table:
192.168.0.223 dev eth0 lladdr 00:e0:29:3c:c1:c9 nud reachable 192.168.0.23 dev eth0 lladdr 00:e0:7d:74:ab:cd nud reachableOn 192.168.0.223 packets for 192.168.0.23 show up. Both clients time out during reassembly.
I placed some printks though the netfilter code andip_output.cbut couldn't find any further pointers. I looked for broken checksums (something seems to alter the ip after checksumming)
inip_finish_output2, but everything is ok there. I can't see anything netfilter related touching packets after that. Any help and/or pointers where to look further would be appreciated.Regards,
Patrick