Skip to contentSkip to conversation

mss to pmtu clamping partially broken?

DISCUSSION Andreas Steinmetz 5 participants 14 messages Jun 29, 2007 – Jul 2, 2007 netfilter

lore
Share & tools
lore
#0 lore raw

There seems to be a problem with mss to pmtu clamping for incoming syn packets on reply to an outgoing connection on a ppp interface. The mss of the outgoing syn packets is always always clamped to the pmtu, I did check this with a target host I do have access to. The incoming syn reply to such a packet, however, is mss clamped only sometimes and this seems to depend on the DSL line used.

The kernels tested were 2.6.20.1, 2.6.20.3 and 2.6.22rc6.

Test setup: Two DSL lines, otherwise identical setup (same masquerading linux gateway, same DSL account, same DSL modem, same DSL line provider, same target host, request from and tcpdump on the same client).

Linux Client<->Masquerading Linux Gateway<->DSL Modem<->DSL Line<->...

DSL line 1, working:

22:26:39.319281 IP (tos 0x0, ttl 64, id 22377, offset 0, flags [DF],
length: 48
) 192.168.0.253.1164 > 64.34.165.170.80: S [tcp sum ok] 1465827859:1465827859(0)
win 5840 <mss 1460,nop,nop,sackOK>
22:26:39.459314 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF],
length: 48) 64
.34.165.170.80 > 192.168.0.253.1164: S [tcp sum ok] 3667852791:3667852791(0) ack
1465827860 win 5840 <mss 1452,nop,nop,sackOK>

The tcpdump on the client shows that the mss of the incoming syn reply packet is clamped to the ppp interface mtu.

DSL line 2, not working:

22:03:57.725998 IP (tos 0x0, ttl 64, id 55984, offset 0, flags [DF],
length: 48
) 192.168.0.253.1600 > 64.34.165.170.80: S [tcp sum ok] 36968258:36968258(0) win
5840 <mss 1460,nop,nop,sackOK>
22:03:57.866966 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF],
length: 48) 64
.34.165.170.80 > 192.168.0.253.1600: S [tcp sum ok] 2226854208:2226854208(0) ack
36968259 win 5840 <mss 1460,nop,nop,sackOK>

The tcpdump on the client shows that the mss of the incoming syn reply packet is *NOT* clamped to the ppp interface mtu.

Signature · 1 line
Andreas Steinmetz                       SPAMmers use [email protected]
#1 lore raw
Quoted 44 lines · Andreas Steinmetz

There seems to be a problem with mss to pmtu clamping for incoming syn packets on reply to an outgoing connection on a ppp interface. The mss of the outgoing syn packets is always always clamped to the pmtu, I did check this with a target host I do have access to. The incoming syn reply to such a packet, however, is mss clamped only sometimes and this seems to depend on the DSL line used.

The kernels tested were 2.6.20.1, 2.6.20.3 and 2.6.22rc6.

Test setup: Two DSL lines, otherwise identical setup (same masquerading linux gateway, same DSL account, same DSL modem, same DSL line provider, same target host, request from and tcpdump on the same client).

Linux Client<->Masquerading Linux Gateway<->DSL Modem<->DSL Line<->...

DSL line 1, working:

22:26:39.319281 IP (tos 0x0, ttl 64, id 22377, offset 0, flags [DF],
length: 48
) 192.168.0.253.1164 > 64.34.165.170.80: S [tcp sum ok] 1465827859:1465827859(0)
win 5840 <mss 1460,nop,nop,sackOK>
22:26:39.459314 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF],
length: 48) 64
.34.165.170.80 > 192.168.0.253.1164: S [tcp sum ok] 3667852791:3667852791(0) ack
1465827860 win 5840 <mss 1452,nop,nop,sackOK>

The tcpdump on the client shows that the mss of the incoming syn reply packet is clamped to the ppp interface mtu.

DSL line 2, not working:

22:03:57.725998 IP (tos 0x0, ttl 64, id 55984, offset 0, flags [DF],
length: 48
) 192.168.0.253.1600 > 64.34.165.170.80: S [tcp sum ok] 36968258:36968258(0) win
5840 <mss 1460,nop,nop,sackOK>
22:03:57.866966 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF],
length: 48) 64
.34.165.170.80 > 192.168.0.253.1600: S [tcp sum ok] 2226854208:2226854208(0) ack
36968259 win 5840 <mss 1460,nop,nop,sackOK>

Quoted 2 lines

The tcpdump on the client shows that the mss of the incoming syn reply packet is *NOT* clamped to the ppp interface mtu.

You forgot to mention *how* you're clamping the MSS. Using TCPMSS? Do you have a rule for incoming packets?

#2 lore raw
Quoted 45 lines · Patrick McHardy
Quoted 44 lines · Andreas Steinmetz

There seems to be a problem with mss to pmtu clamping for incoming syn packets on reply to an outgoing connection on a ppp interface. The mss of the outgoing syn packets is always always clamped to the pmtu, I did check this with a target host I do have access to. The incoming syn reply to such a packet, however, is mss clamped only sometimes and this seems to depend on the DSL line used.

The kernels tested were 2.6.20.1, 2.6.20.3 and 2.6.22rc6.

Test setup: Two DSL lines, otherwise identical setup (same masquerading linux gateway, same DSL account, same DSL modem, same DSL line provider, same target host, request from and tcpdump on the same client).

Linux Client<->Masquerading Linux Gateway<->DSL Modem<->DSL Line<->...

DSL line 1, working:

22:26:39.319281 IP (tos 0x0, ttl 64, id 22377, offset 0, flags [DF],
length: 48
) 192.168.0.253.1164 > 64.34.165.170.80: S [tcp sum ok] 1465827859:1465827859(0)
win 5840 <mss 1460,nop,nop,sackOK>
22:26:39.459314 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF],
length: 48) 64
.34.165.170.80 > 192.168.0.253.1164: S [tcp sum ok] 3667852791:3667852791(0) ack
1465827860 win 5840 <mss 1452,nop,nop,sackOK>

The tcpdump on the client shows that the mss of the incoming syn reply packet is clamped to the ppp interface mtu.

DSL line 2, not working:

22:03:57.725998 IP (tos 0x0, ttl 64, id 55984, offset 0, flags [DF],
length: 48
) 192.168.0.253.1600 > 64.34.165.170.80: S [tcp sum ok] 36968258:36968258(0) win
5840 <mss 1460,nop,nop,sackOK>
22:03:57.866966 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF],
length: 48) 64
.34.165.170.80 > 192.168.0.253.1600: S [tcp sum ok] 2226854208:2226854208(0) ack
36968259 win 5840 <mss 1460,nop,nop,sackOK>

Quoted 7 lines
Quoted 2 lines

The tcpdump on the client shows that the mss of the incoming syn reply packet is *NOT* clamped to the ppp interface mtu.

You forgot to mention *how* you're clamping the MSS. Using TCPMSS? Do you have a rule for incoming packets?

The relevant iptables commands I do use for masquerading and clamping are:

iptables -t nat -A POSTROUTING -o ppp0 -j MASQUERADE

iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS \
	--clamp-mss-to-pmtu
Signature · 1 line
Andreas Steinmetz                       SPAMmers use [email protected]
#3 lore raw
Quoted 14 lines · Andreas Steinmetz
Quoted 10 lines · Patrick McHardy
Quoted 3 lines · Andreas Steinmetz

[...]
The tcpdump on the client shows that the mss of the incoming syn reply packet is *NOT* clamped to the ppp interface mtu.

You forgot to mention *how* you're clamping the MSS. Using TCPMSS? Do you have a rule for incoming packets?

Quoted 5 lines

The relevant iptables commands I do use for masquerading and clamping are:

iptables -t nat -A POSTROUTING -o ppp0 -j MASQUERADE

iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS \
	--clamp-mss-to-pmtu

Two things here:

- tcpdumps on ppp0 will show unclamped packets since they haven't been forwarded yet

- assuming you have ethernet internally, the PMTU from your router to the internal hosts is 1500, so it won't do any clamping.

Does that explain it?

A useful thing for TCPMSS for routers would be to clamp to the minimum of the PMTU of both directions. But thats not supported so far.

#4 lore raw
Quoted 20 lines · Patrick McHardy
Quoted 12 lines · Andreas Steinmetz
Quoted 9 lines · Patrick McHardy
Quoted 3 lines · Andreas Steinmetz

[...]
The tcpdump on the client shows that the mss of the incoming syn reply packet is *NOT* clamped to the ppp interface mtu.

You forgot to mention *how* you're clamping the MSS. Using TCPMSS? Do you have a rule for incoming packets?

Quoted 5 lines

The relevant iptables commands I do use for masquerading and clamping are:

iptables -t nat -A POSTROUTING -o ppp0 -j MASQUERADE

iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS \
	--clamp-mss-to-pmtu
Quoted 5 lines

Two things here:

- tcpdumps on ppp0 will show unclamped packets since they haven't been forwarded yet

That is true, I know this.

Quoted 3 lines

- assuming you have ethernet internally, the PMTU from your router to the internal hosts is 1500, so it won't do any clamping.

Yep, internal PMTU is 1500, still the incoming packets are clamped to 1452 on the one line and not clamped on the other.

Quoted 6 lines

Does that explain it?

A useful thing for TCPMSS for routers would be to clamp to the minimum of the PMTU of both directions. But thats not supported so far.

I wonder, as somteimes it gets clamped. If it would never have been clamped I wouldn't have asked.

Signature · 1 line
Andreas Steinmetz                       SPAMmers use [email protected]
#5 lore raw
Quoted 13 lines · Andreas Steinmetz
Quoted 3 lines · Patrick McHardy

- assuming you have ethernet internally, the PMTU from your router to the internal hosts is 1500, so it won't do any clamping.

Yep, internal PMTU is 1500, still the incoming packets are clamped to 1452 on the one line and not clamped on the other.

Quoted 2 lines

Does that explain it?

Quoted 8 lines
Quoted 4 lines

A useful thing for TCPMSS for routers would be to clamp to the minimum of the PMTU of both directions. But thats not supported so far.

I wonder, as somteimes it gets clamped. If it would never have been clamped I wouldn't have asked.

Its possible that one of your ISPs is doing clamping. You could check on ppp0 if thats the case. Or maybe for some reason the PMTU value for the internal host is smaller than 1500. You can check that by doing "ip route get <internal host>".

#6 lore raw
Quoted 18 lines · Patrick McHardy
Quoted 17 lines · Andreas Steinmetz
Quoted 3 lines · Patrick McHardy

- assuming you have ethernet internally, the PMTU from your router to the internal hosts is 1500, so it won't do any clamping.

Yep, internal PMTU is 1500, still the incoming packets are clamped to 1452 on the one line and not clamped on the other.

Quoted 6 lines

Does that explain it?

A useful thing for TCPMSS for routers would be to clamp to the minimum of the PMTU of both directions. But thats not supported so far.

Quoted 5 lines
Quoted 2 lines

I wonder, as somteimes it gets clamped. If it would never have been clamped I wouldn't have asked.

Its possible that one of your ISPs is doing clamping. You could

This would be fun as it is the same ISP for both lines. I'll check next week as the lines are located 40km away.

Quoted 4 lines

check on ppp0 if thats the case. Or maybe for some reason the PMTU value for the internal host is smaller than 1500. You can check that by doing "ip route get <internal host>".

No. Unmodified internal network in both test cases.

Quoted 1 line
Signature · 1 line
Andreas Steinmetz                       SPAMmers use [email protected]
#8 lore raw
In reply to Andreas Steinmetz
Quoted 7 lines · Andreas Steinmetz

There seems to be a problem with mss to pmtu clamping for incoming syn packets on reply to an outgoing connection on a ppp interface. The mss of the outgoing syn packets is always always clamped to the pmtu, I did check this with a target host I do have access to. The incoming syn reply to such a packet, however, is mss clamped only sometimes and this seems to depend on the DSL line used.

Do you really need clamping? It's a hack, since TCP should do MSS negotiation itself. (Of course it may happen that some routers are broken.) But usually not for incoming packets.

Jan

#7 lore raw
In reply to Patrick McHardy
Quoted 6 lines · Patrick McHardy

Its possible that one of your ISPs is doing clamping. You could check on ppp0 if thats the case. Or maybe for some reason the PMTU value for the internal host is smaller than 1500. You can check that by doing "ip route get <internal host>".

Oh well, thew fun with ISPs. Same provider, clamping on one line but not the other. This is fun :-(

Signature · 1 line
Andreas Steinmetz                       SPAMmers use [email protected]
#9 lore raw
In reply to Jan Engelhardt
Quoted 3 lines · Jan Engelhardt

Do you really need clamping? It's a hack, since TCP should do MSS negotiation itself. (Of course it may happen that some routers are broken.) But usually not for incoming packets.

You never know when you hit ICMP blackholes, broken routers and other evil things. Better safe than sorry so clamping is the way to go for me.

Signature · 1 line
Andreas Steinmetz                       SPAMmers use [email protected]
#10 lore raw
Quoted 7 lines · Andreas Steinmetz
Quoted 3 lines · Jan Engelhardt

Do you really need clamping? It's a hack, since TCP should do MSS negotiation itself. (Of course it may happen that some routers are broken.) But usually not for incoming packets.

You never know when you hit ICMP blackholes, broken routers and other evil things. Better safe than sorry so clamping is the way to go for me.

I encourage you to report PMTUD Blackholes to the MSS Initiative at
http://www.phildev.net/mss/

We'll notify them, and if we can't get them to fix it, blacklist them. We have more fixed sites than blacklisted sites, so it's at least somewhat successful.

Signature · 9 lines
Phil Dibowitz                             [email protected]
Open Source software and tech docs        Insanity Palace of Metallica
http://www.phildev.net/                   http://www.ipom.com/

"Never write it in C if you can do it in 'awk';
 Never do it in 'awk' if 'sed' can handle it;
 Never use 'sed' when 'tr' can do the job;
 Never invoke 'tr' when 'cat' is sufficient;
 Avoid using 'cat' whenever possible" -- Taylor's Laws of Programming
#11 lore raw
Quoted 11 lines · Phil Dibowitz
Quoted 7 lines · Andreas Steinmetz
Quoted 3 lines · Jan Engelhardt

Do you really need clamping? It's a hack, since TCP should do MSS negotiation itself. (Of course it may happen that some routers are broken.) But usually not for incoming packets.

You never know when you hit ICMP blackholes, broken routers and other evil things. Better safe than sorry so clamping is the way to go for me.

I encourage you to report PMTUD Blackholes to the MSS Initiative at
http://www.phildev.net/mss/

Any chances for similar initiative for "SACK vandals"? ;)

Best regards,

Krzysztof Olędzki

#12 lore raw
Quoted 12 lines · Krzysztof Oledzki
Quoted 8 lines · Phil Dibowitz
Quoted 7 lines · Andreas Steinmetz
Quoted 5 lines · Jan Engelhardt

Do you really need clamping? It's a hack, since TCP should do MSS negotiation
itself. (Of course it may happen that some routers are broken.) But usually not
for incoming packets.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

You never know when you hit ICMP blackholes, broken routers and other evil things. Better safe than sorry so clamping is the way to go for me.

I encourage you to report PMTUD Blackholes to the MSS Initiative at
http://www.phildev.net/mss/

Any chances for similar initiative for "SACK vandals"? ;)

There's already a counterpart for ECN blackholes, so I'm not opposed to it. However, keeping up with new reports, re-testing existing offenders, etc. takes up a good chunk of time, so I don't have the time to do it myself. I'm happy to reference such a site, however.

Though - I'm not familiar with the problem of SACK vandals either. There appears to be a thread on here, I'll go read it...

Signature · 9 lines
Phil Dibowitz                             [email protected]
Open Source software and tech docs        Insanity Palace of Metallica
http://www.phildev.net/                   http://www.ipom.com/

"Never write it in C if you can do it in 'awk';
 Never do it in 'awk' if 'sed' can handle it;
 Never use 'sed' when 'tr' can do the job;
 Never invoke 'tr' when 'cat' is sufficient;
 Avoid using 'cat' whenever possible" -- Taylor's Laws of Programming
#13 lore raw
Quoted 19 lines · Phil Dibowitz
Quoted 18 lines · Krzysztof Oledzki
Quoted 13 lines · Phil Dibowitz
Quoted 9 lines · Andreas Steinmetz
Quoted 5 lines · Jan Engelhardt

Do you really need clamping? It's a hack, since TCP should do MSS negotiation
itself. (Of course it may happen that some routers are broken.) But usually not
for incoming packets.

You never know when you hit ICMP blackholes, broken routers and other evil things. Better safe than sorry so clamping is the way to go for me.

I encourage you to report PMTUD Blackholes to the MSS Initiative at
http://www.phildev.net/mss/

Quoted 6 lines
Quoted 1 line

Any chances for similar initiative for "SACK vandals"? ;)

There's already a counterpart for ECN blackholes, so I'm not opposed to it. However, keeping up with new reports, re-testing existing offenders, etc. takes up a good chunk of time, so I don't have the time to do it myself. I'm happy to reference such a site, however.

Indeed and it seems there are more important issues, like similar window scaling problem for example. :(

Best regards,

Krzysztof Olędzki