Skip to contentSkip to conversation

Intermittent NAT failure when multiple hosts send UDP packets

DISCUSSION Bernardo Innocenti 2 participants 6 messages Sep 17, 2005 – Sep 27, 2005 netfilter

lore
Share & tools
lore
#0 lore raw

This smells like a bug in UDP ip_nat_proto_udp.c or nearby. I'm seeing this on 2.6.12-1.1447_FC4, but code in 2.6.13 is still the same.

I've setup SNAT the usual way:

iptables -A POSTROUTING -t nat -o ppp0 -j SNAT --to-source 151.38.19.110

When multiple clients in the LAN send UDP packets to the same port of the same remote host, I see something like this in my /proc/net/ip_conntrack:

 udp      17 170 src=10.3.3.2 dst=194.185.88.60 sport=5060 dport=5060 src=194.185.88.60 dst=151.38.19.110 sport=5060 dport=5060 [ASSURED] use=1
 udp      17 29 src=10.3.3.2 dst=212.97.59.76 sport=5060 dport=5060 [UNREPLIED] src=212.97.59.76 dst=151.38.19.110 sport=5060 dport=5060 use=1
 udp      17 177 src=10.3.3.250 dst=194.185.88.60 sport=5060 dport=5060 src=194.185.88.60 dst=151.38.19.110 sport=5060 dport=1024 [ASSURED] use=1

In the last line, the destination port has been properly remapped from 5060 to 1024 to distingish between incoming packets.

However, I see packets going out over ppp0 without the source address properly rewritten to 151.38.19.110:

04:38:28.739514 IP 10.3.3.2.5060 > 194.185.88.60.5060: UDP, length 536

This doesn't happen when there's just a single host sending to port 5060. Sometimes I must restart the interface to trigger this bug.

Signature · 2 lines
  // Bernardo Innocenti - Develer S.r.l., R&D dept.
\X/  http://www.develer.com/
#1 lore raw

Never mind, it was fixed in 2.6.13, probably by this patch:

https://lists.netfilter.org/pipermail/netfilter-devel/2004-March/014412.html

Quoted 25 lines · Bernardo Innocenti

This smells like a bug in UDP ip_nat_proto_udp.c or nearby. I'm seeing this on 2.6.12-1.1447_FC4, but code in 2.6.13 is still the same.

I've setup SNAT the usual way:

iptables -A POSTROUTING -t nat -o ppp0 -j SNAT --to-source 151.38.19.110

When multiple clients in the LAN send UDP packets to the same port of the same remote host, I see something like this in my /proc/net/ip_conntrack:

 udp      17 170 src=10.3.3.2 dst=194.185.88.60 sport=5060 dport=5060 src=194.185.88.60 dst=151.38.19.110 sport=5060 dport=5060 [ASSURED] use=1
 udp      17 29 src=10.3.3.2 dst=212.97.59.76 sport=5060 dport=5060 [UNREPLIED] src=212.97.59.76 dst=151.38.19.110 sport=5060 dport=5060 use=1
 udp      17 177 src=10.3.3.250 dst=194.185.88.60 sport=5060 dport=5060 src=194.185.88.60 dst=151.38.19.110 sport=5060 dport=1024 [ASSURED] use=1

In the last line, the destination port has been properly remapped from 5060 to 1024 to distingish between incoming packets.

However, I see packets going out over ppp0 without the source address properly rewritten to 151.38.19.110:

04:38:28.739514 IP 10.3.3.2.5060 > 194.185.88.60.5060: UDP, length 536

This doesn't happen when there's just a single host sending to port 5060. Sometimes I must restart the interface to trigger this bug.

Signature · 2 lines
  // Bernardo Innocenti - Develer S.r.l., R&D dept.
\X/  http://www.develer.com/
#2 lore raw

I'm sorry to say that this bug has shown up again on 2.6.13 too, so it's not fixed at all.

It's quite hard to trigger, but after it does, packets are consistently routed with the source IP untranslated.

Quoted 34 lines · Bernardo Innocenti

Never mind, it was fixed in 2.6.13, probably by this patch:

https://lists.netfilter.org/pipermail/netfilter-devel/2004-March/014412.html

Quoted 25 lines · Bernardo Innocenti

This smells like a bug in UDP ip_nat_proto_udp.c or nearby. I'm seeing this on 2.6.12-1.1447_FC4, but code in 2.6.13 is still the same.

I've setup SNAT the usual way:

iptables -A POSTROUTING -t nat -o ppp0 -j SNAT --to-source 151.38.19.110

When multiple clients in the LAN send UDP packets to the same port of the same remote host, I see something like this in my /proc/net/ip_conntrack:

udp      17 170 src=10.3.3.2 dst=194.185.88.60 sport=5060 dport=5060 src=194.185.88.60 dst=151.38.19.110 sport=5060 dport=5060 [ASSURED] use=1
udp      17 29 src=10.3.3.2 dst=212.97.59.76 sport=5060 dport=5060 [UNREPLIED] src=212.97.59.76 dst=151.38.19.110 sport=5060 dport=5060 use=1
udp      17 177 src=10.3.3.250 dst=194.185.88.60 sport=5060 dport=5060 src=194.185.88.60 dst=151.38.19.110 sport=5060 dport=1024 [ASSURED] use=1

In the last line, the destination port has been properly remapped from 5060 to 1024 to distingish between incoming packets.

However, I see packets going out over ppp0 without the source address properly rewritten to 151.38.19.110:

04:38:28.739514 IP 10.3.3.2.5060 > 194.185.88.60.5060: UDP, length 536

This doesn't happen when there's just a single host sending to port 5060. Sometimes I must restart the interface to trigger this bug.

Signature · 2 lines
  // Bernardo Innocenti - Develer S.r.l., R&D dept.
\X/  http://www.develer.com/
#3 lore raw
Quoted 5 lines · Bernardo Innocenti

I'm sorry to say that this bug has shown up again on 2.6.13 too, so it's not fixed at all.

It's quite hard to trigger, but after it does, packets are consistently routed with the source IP untranslated.

Please try "echo 255 > /proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid" and modprobe ipt_LOG to see if conntrack ignores them because of invalid checksums or something.

#4 lore raw
Quoted 5 lines · Patrick McHardy
Quoted 3 lines · Bernardo Innocenti

I'm sorry to say that this bug has shown up again on 2.6.13 too, so it's not fixed at all.

Quoted 8 lines
Quoted 2 lines

It's quite hard to trigger, but after it does, packets are consistently routed with the source IP untranslated.

Please try "echo 255 > /proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid" and modprobe ipt_LOG to see if conntrack ignores them because of invalid checksums or something.

It doesn't seem to be the case. I only see a few occasional errors, probably caused by miserable hosts crawling with worms:

ip_ct_tcp: invalid packet ignored IN= OUT= SRC=83.40.242.177 DST=151.38.19.110 LEN=48 TOS=0x18 PREC=0x20 TTL=114 ID=42553 DF PROTO=TCP SPT=3017 DPT=4662 SEQ=3667342556 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT (0204055001010402)
device eth1 left promiscuous mode
ip_ct_tcp: SEQ is under the lower bound (already ACKed data retransmitted) IN= OUT= SRC=62.241.4.1 DST=151.38.19.110 LEN=40 TOS=0x18 PREC=0x20 TTL=39 ID=62928 PROTO=TCP SPT=995 DPT=41567 SEQ=0 ACK=1313396353 WINDOW=0 RES=0x00 ACK RST URGP=0
ip_ct_icmp: bad ICMP checksum IN= OUT= SRC=68.192.221.135 DST=151.38.19.110 LEN=81 TOS=0x18 PREC=0x20 TTL=49 ID=30671 PROTO=ICMP TYPE=3 CODE=1 [SRC=151.38.19.110 DST=68.192.221.135 LEN=53 TOS=0x00 PREC=0x00 TTL=52 ID=32600 DF PROTO=UDP SPT=13049 DPT=12464 LEN=33 ]
ip_ct_tcp: invalid state IN= OUT= SRC=168.226.95.26 DST=151.38.19.110 LEN=52 TOS=0x18 PREC=0x20 TTL=106 ID=26190 DF PROTO=TCP SPT=4662 DPT=40358 SEQ=2466472710 ACK=1425847173 WINDOW=65535 RES=0x00 ACK URGP=0 OPT (0101080A00048B3304B5C54C)
ip_ct_tcp: invalid packet ignored IN= OUT= SRC=82.224.25.130 DST=151.38.19.110 LEN=48 TOS=0x18 PREC=0x20 TTL=116 ID=17411 DF PROTO=TCP SPT=3565 DPT=4662 SEQ=1051168262 ACK=0 WINDOW=16384 RES=0x00 SYN URGP=0 OPT (0204058401010402)

By the way, all the rest of my TCP and UDP traffic is still being translated and routed correctly:

 # tcpdump -n -i ppp0 host sip.squillo.it -v -v -v
 01:52:38.020005 IP (tos 0x0, ttl  63, id 0, offset 0, flags [DF], proto 17, length: 454) 151.38.19.110.1024 > 194.185.88.60.5060: UDP, length 426
 01:52:38.096282 IP (tos 0x38, ttl  55, id 23343, offset 0, flags [DF], proto 17, length: 475) 194.185.88.60.5060 > 151.38.19.110.1024: UDP, length 447
 01:52:38.100324 IP (tos 0x38, ttl  55, id 23344, offset 0, flags [DF], proto 17, length: 560) 194.185.88.60.5060 > 151.38.19.110.1024: UDP, length 532
 01:52:38.110865 IP (tos 0x0, ttl  63, id 1, offset 0, flags [DF], proto 17, length: 597) 151.38.19.110.1024 > 194.185.88.60.5060: UDP, length 569
 01:52:38.216213 IP (tos 0x38, ttl  55, id 23353, offset 0, flags [DF], proto 17, length: 475) 194.185.88.60.5060 > 151.38.19.110.1024: UDP, length 447
 01:52:38.238967 IP (tos 0x38, ttl  55, id 23354, offset 0, flags [DF], proto 17, length: 553) 194.185.88.60.5060 > 151.38.19.110.1024: UDP, length 525

It's just this one connection that doesn't:

 # tcpdump -n -i ppp0 host sip.squillo.it -v -v -v
 01:44:04.936695 IP (tos 0x38, ttl  63, id 8416, offset 0, flags [DF], proto 17, length: 564) 10.3.3.2.5060 > 194.185.88.60.5060: UDP, length 536
 01:44:06.937433 IP (tos 0x38, ttl  63, id 8418, offset 0, flags [DF], proto 17, length: 564) 10.3.3.2.5060 > 194.185.88.60.5060: UDP, length 536
 01:44:10.936911 IP (tos 0x38, ttl  63, id 8420, offset 0, flags [DF], proto 17, length: 564) 10.3.3.2.5060 > 194.185.88.60.5060: UDP, length 536

If I stop transmitting packets until the conntrack timer expires, then everything works again normally for a few hours.

If I read ip_nat_core.c correctly, manip_pkt() is responsible for doing the IP address translation, while udp_manip_pkt() takes care of the port in the UDP packet. I'm staring at this code without a clue how it could selectively skip one of the tuples.

Signature · 2 lines
  // Bernardo Innocenti - Develer S.r.l., R&D dept.
\X/  http://www.develer.com/
#5 lore raw
Quoted 6 lines · Bernardo Innocenti
Quoted 5 lines · Patrick McHardy
Quoted 2 lines · Bernardo Innocenti

It's quite hard to trigger, but after it does, packets are consistently routed with the source IP untranslated.

Quoted 7 lines
Quoted 4 lines

Please try "echo 255 > /proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid" and modprobe ipt_LOG to see if conntrack ignores them because of invalid checksums or something.

It doesn't seem to be the case. I only see a few occasional errors, probably caused by miserable hosts crawling with worms:

PROBLEM SOLVED! I'm glad to say It's *almost* not a kernel bug, more like a missing feature.

In my ip-up.local script, I do:

echo "Loading NAT modules:"
/sbin/modprobe ip_conntrack
/sbin/modprobe ip_conntrack_ftp
/sbin/modprobe ip_conntrack_irc
/sbin/modprobe iptable_nat
/sbin/modprobe ip_nat_ftp
/sbin/modprobe ip_nat_irc

[...several filtering and QoS rules...]

$iptab -A POSTROUTING -t nat -o $IFNAME -j SNAT --to-source $IPLOCAL

My ip-down.local attempts to do the opposite:

echo "Flushing all current rules:"
$iptab -F
$iptab -F -t nat
echo "Clearing all chains:"
$iptab -X
$iptab -Z
$iptab -X -t nat
$iptab -Z -t nat

echo "Removing NAT modules:"
/sbin/rmmod ip_nat_ftp ip_nat_irc iptable_nat ip_conntrack ip_conntrack_ftp ip_conntrack_irc

Note the order of the modules in the last line: ip_conntrack cannot be unloaded because it's still being used by ip_conntrack_ftp and ip_conntrack_irc.

So, whenever the PPP link goes down, ip_conntrack remains loaded with all connections still being tracked until the timer expires!

If ppp0 goes up again soon enough, the script reloads the ip_nat modules, but the existing connections are no longer being translated.

Would it be possible to do something at module initialization time to recover those connections? Meanwhile, I've fixed my rmmod to make sure ip_conntrack really gets unloaded.

Signature · 2 lines
  // Bernardo Innocenti - Develer S.r.l., R&D dept.
\X/  http://www.develer.com/