Invalid SACK numbers in NAT'ed packets
Hi all,
I have the following issue that I need a comment on.
The setup. I have a Linux router with one NIC connected to a LAN and the other connected directly to a Cisco 800 Series router. The Cisco box is connected further to the ISP. The Cisco box creates an IPsec tunnel to a remote site. It also does SNAT on packets coming through it from the LAN to the remote site. It looks as follows:
LAN -> Linux router -> Cisco (NAT) -> IPsec tunnel -> remote site
The Linux router has the following IP Tables rules:
-A FORWARD -i eth0 -o eth3 -j ACCEPT
-A FORWARD -i eth3 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
with the default policy for the FORWARD chain being DROP. Here eth0 is the LAN interface and eth3 is the interface that goes to the Cisco router. Thus packets coming from the LAN to the Cisco and further to remote site are allowed, as well as replies to them, but the remote site can't intiate a connection to the LAN. So far so good, we can connect from the LAN to the remote site successfully.
The problem. When uploading large files by FTP from the LAN to the remote site, the connection randomly stalls.
Analysis of tcp dumps shows that some ACK packets coming back from the remote site are not forwarded by the Linux box from eth3 to eth0. Apparently, Linux doesn't think they are part of the established connection. Adding this rule that lets all ACK packets through fixes the problem:
-A FORWARD -i eth3 -o eth0 -p tcp --tcp-flags ACK ACK -j ACCEPT
I want to understand why these ACK packets are not allowed by the original state match.
Further analysis of tcp dumps reveals the following. The Cisco router changes TCP sequence numbers when it does NAT. So, the sequence numbers and ACKs that go between the LAN and the Cisco box are not the same as the ones on the other side, between the Cisco box and the remote site. The problem is, the Cisco box does NOT change the sequence numbers within SACK TCP option! So, whenever a packet is lost, and the remote site sends an ACK packet that contains a SACK option, the Linux router sees a SACK option referencing a packet it never saw before. My guess is that this is the reason why the packet is not considered as belonging to an established connection by netfilter.
The questions:
1. Does my explanation look plausible? Can an invalid sequence number in SACK lead to the packet not being considered as belonging to an established connection?
2. Has anyone come across such issue with a Cisco hardware before? Can it be fixed by some configuration?
3. If not, can it be worked around at the Linux side; i.e. somehow ignore invalid SACKs or prohibit SACKs on this particular connection (I am reluctant to turn them off altogether with a sysctl).
Thanks a lot,
Leonid
Quoted 11 lines · Leonid Zeitlin
Further analysis of tcp dumps reveals the following. The Cisco router changes TCP sequence numbers when it does NAT. So, the sequence numbers and ACKs that go between the LAN and the Cisco box are not the same as the ones on the other side, between the Cisco box and the remote site. The problem is, the Cisco box does NOT change the sequence numbers within SACK TCP option! So, whenever a packet is lost, and the remote site sends an ACK packet that contains a SACK option, the Linux router sees a SACK option referencing a packet it never saw before. My guess is that this is the reason why the packet is not considered as belonging to an established connection by netfilter.
Quoted 4 lines
The questions:
1. Does my explanation look plausible? Can an invalid sequence number in SACK lead to the packet not being considered as belonging to an established connection?
Yes, TCP connection tracking in netfilter checks the SACK option values as well.
Quoted 2 lines
2. Has anyone come across such issue with a Cisco hardware before? Can it be fixed by some configuration?
It was reported a few times. Try to upgrade IOS image as it's a Cisco bug
Quoted 3 lines
3. If not, can it be worked around at the Linux side; i.e. somehow ignore invalid SACKs or prohibit SACKs on this particular connection (I am reluctant to turn them off altogether with a sysctl).
or use IPV4OPTSTRIP for the SYN packets sent/received in this direction as a selective workaround for the problem.
Best regards,
Jozsef
-
E-mail : [email protected], [email protected] PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
Hi Jozsef,
Thanks for your answer.
Quoted 2 lines
or use IPV4OPTSTRIP for the SYN packets sent/received in this direction as a selective workaround for the problem.
What is IPV4OPTSTRIP? How can I get it? It's not in standard iptables (not the one that I have anyway), and I can't find it at the netfilter site either.
Thanks,
Leonid
Quoted 5 lines · Leonid Zeitlin
Quoted 2 lines
or use IPV4OPTSTRIP for the SYN packets sent/received in this direction as a selective workaround for the problem.
What is IPV4OPTSTRIP? How can I get it? It's not in standard iptables (not the one that I have anyway), and I can't find it at the netfilter site either.
It's a target extension which can be found in patch-o-matic-ng. But sorry, I mixed up: it strips off IPv4 options and not TCP options, so it'd not help.
Best regards,
Jozsef
-
E-mail : [email protected], [email protected] PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
Thanks, Jozsef, I see.
It appears that short of writing a custom netfilter extension, there's no way to turn off SACKs on a particular connection. Is this right?
Thanks,
Leonid
----- Original Message ----- From: "Jozsef Kadlecsik" <[email protected]>
To: "Leonid Zeitlin" <[email protected]>
Cc: <[email protected]>
Sent: Thursday, April 24, 2008 12:33 PM
Subject: Re: Invalid SACK numbers in NAT'ed packets
Quoted 20 lines · Leonid Zeitlin
Quoted 6 lines
Quoted 3 lines
or use IPV4OPTSTRIP for the SYN packets sent/received in this direction as a selective workaround for the problem.
What is IPV4OPTSTRIP? How can I get it? It's not in standard iptables (not the one that I have anyway), and I can't find it at the netfilter site either.
It's a target extension which can be found in patch-o-matic-ng. But sorry, I mixed up: it strips off IPv4 options and not TCP options, so it'd not help.
Best regards,
Jozsef
-
E-mail : [email protected], [email protected] PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear PhysicsH-1525 Budapest 114, POB. 49, Hungary
Quoted 2 lines · Leonid Zeitlin
It appears that short of writing a custom netfilter extension, there's no way to turn off SACKs on a particular connection. Is this right?
Yes, exactly. Actually, writing a new extension to erase any TCP option isn't that hard: just replace the option with noop and recalculate the checksum.
Best regards,
Jozsef
-
E-mail : [email protected], [email protected] PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
Quoted 8 lines · Jozsef Kadlecsik
Quoted 2 lines · Leonid Zeitlin
It appears that short of writing a custom netfilter extension, there's no way to turn off SACKs on a particular connection. Is this right?
Yes, exactly. Actually, writing a new extension to erase any TCP option isn't that hard: just replace the option with noop and recalculate the checksum.
There is already a TCPOPTSTRIP target.
Quoted 4 lines · Jan Engelhardt
Quoted 2 lines · Jozsef Kadlecsik
On Fri, 25 Apr 2008, Leonid Zeitlin wrote:
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
It appears that short of writing a custom netfilter extension, there's no way to turn off SACKs on a particular connection. Is this right?
Yes, exactly. Actually, writing a new extension to erase any TCP option isn't that hard: just replace the option with noop and recalculate the checksum.
There is already a TCPOPTSTRIP target.
True, it was added to 2.6.25. I should have checked the facts before posting.
Best regards,
Jozsef
-
E-mail : [email protected], [email protected] PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
If I am reading the news correctly, it is available in the latest 2.6.25 kernel, right?
Thanks,
Leonid
----- Original Message ----- From: "Jan Engelhardt" <[email protected]>
To: "Jozsef Kadlecsik" <[email protected]>
Cc: "Leonid Zeitlin" <[email protected]>; <[email protected]>; <[email protected]>
Sent: Friday, April 25, 2008 1:57 PM
Subject: Re: Invalid SACK numbers in NAT'ed packets
Quoted 14 lines · Jozsef Kadlecsik
Quoted 9 lines · Leonid Zeitlin
Quoted 3 lines
It appears that short of writing a custom netfilter extension, there's no way to turn off SACKs on a particular connection. Is this right?
Yes, exactly. Actually, writing a new extension to erase any TCP option isn't that hard: just replace the option with noop and recalculate the checksum.
There is already a TCPOPTSTRIP target.