IPsec PMTUD problem
Subsystems (2)
I noticed a problem with PMTUD between two IPsec tunnel endpoints. When sending a packet larger than the PMTU with IP_DF from one tunnel endpoint to the other, xfrm4_output sends an ICMP frag. required with the IPsec MTU. Since the addresses match the tunnel endpoints, this updates the MTU for the XFRM route with the value that was calculated for the entire bundle, which in turn causes a decrease for the bundle, resulting in further ICMP frag. required messages until the minimum is reached.
For example with an IPsec tunnel 10.0.0.1 <-> 10.0.0.2:
uml:~# ping 10.0.0.1 -M do -s 1500 -c 20 PING 10.0.0.1 (10.0.0.1) 1500(1528) bytes of data.
Quoted 8 lines
From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 1428) From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 1356) From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 1284) From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 1212) From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 1140) From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 1068) From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 996) From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 924)
Quoted 8 lines
From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 852) From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 780) From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 708) From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 636) From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 564) From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 552) From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 492) From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 492)
...
I can't come up with a way to fix this, so I'm hoping someone else has an idea.
Hi Patrick:
Quoted 8 lines · Patrick McHardy
I noticed a problem with PMTUD between two IPsec tunnel endpoints. When sending a packet larger than the PMTU with
IP_DFfrom one tunnel endpoint to the other,xfrm4_outputsends an ICMP frag. required with the IPsec MTU. Since the addresses match the tunnel endpoints, this updates the MTU for the XFRM route with the value that was calculated for the entire bundle, which in turn causes a decrease for the bundle, resulting in further ICMP frag. required messages until the minimum is reached.
I presume you're using the same pair of addresses inside and outside the tunnel? If so the problem is that the kernel doesn't distinguish between internal ICMP errors and external ones. So when an MTU update occurs for the internal pair the external pair is also affected.
We'd need some field in the routing cache to distinguish the two pairs.
Of course the easy work-around is to use distinct addresses within IPsec tunnels.
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtQuoted 12 lines · Herbert Xu
Quoted 8 lines · Patrick McHardy
I noticed a problem with PMTUD between two IPsec tunnel endpoints. When sending a packet larger than the PMTU with
IP_DFfrom one tunnel endpoint to the other,xfrm4_outputsends an ICMP frag. required with the IPsec MTU. Since the addresses match the tunnel endpoints, this updates the MTU for the XFRM route with the value that was calculated for the entire bundle, which in turn causes a decrease for the bundle, resulting in further ICMP frag. required messages until the minimum is reached.
Quoted 5 lines
I presume you're using the same pair of addresses inside and outside the tunnel? If so the problem is that the kernel doesn't distinguish between internal ICMP errors and external ones. So when an MTU update occurs for the internal pair the external pair is also affected.
Exactly.
Quoted 2 lines
We'd need some field in the routing cache to distinguish the two pairs.
I'm not sure I understand how this would work, the ICMP message looks the same in both cases. Or are you suggesting to differentiate based on the source of the ICMP message?
Quoted 2 lines
Of course the easy work-around is to use distinct addresses within IPsec tunnels.
Yes, that would work as a workaround, but it still seems like something worth fixing.
Quoted 13 lines · Herbert Xu
Quoted 3 lines · Patrick McHardy
I'm not sure I understand how this would work, the ICMP message looks the same in both cases. Or are you suggesting to differentiate based on the source of the ICMP message?
Actually you're right, this can't work in the general case. Even if we had real devices for IPsec tunnels, there is still no way to reliably figure out which device we should attribute a given MTU event to if the same address appears on more than one device.
Quoted 8 lines
Quoted 2 lines
Yes, that would work as a workaround, but it still seems like something worth fixing.
One possible solution is to not send MTU errors to ourselves since we it wouldn't give us any new information. We'd need to audit the users of
icmp_sendto make sure that there isn't a legitimate case where we'd want to do that.
One such case is delivery of errors to sockets. We'd need to make sure the errors are delivered some other way.
Quoted 7 lines · Herbert Xu
Quoted 5 lines · Patrick McHardy
Quoted 4 lines
One possible solution is to not send MTU errors to ourselves since we it wouldn't give us any new information. We'd need to audit the users of
icmp_sendto make sure that there isn't a legitimate case where we'd want to do that.
Quoted 6 lines
Quoted 2 lines
One such case is delivery of errors to sockets. We'd need to make sure the errors are delivered some other way.
Alternatively we can still send the ICMP error but avoid a PMTU update if we received it from ourselves.
That sounds easier. I'm currently working in that area anyway, I'll give it a try.
Quoted 3 lines · Patrick McHardy
That sounds easier. I'm currently working in that area anyway, I'll give it a try.
Thanks Patrick!
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtQuoted 8 lines · Patrick McHardy
Quoted 5 lines
One possible solution is to not send MTU errors to ourselves since we it wouldn't give us any new information. We'd need to audit the users of
icmp_sendto make sure that there isn't a legitimate case where we'd want to do that.One such case is delivery of errors to sockets. We'd need to make sure the errors are delivered some other way.
Alternatively we can still send the ICMP error but avoid a PMTU update if we received it from ourselves.
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtQuoted 4 lines · Patrick McHardy
I'm not sure I understand how this would work, the ICMP message looks the same in both cases. Or are you suggesting to differentiate based on the source of the ICMP message?
Actually you're right, this can't work in the general case. Even if we had real devices for IPsec tunnels, there is still no way to reliably figure out which device we should attribute a given MTU event to if the same address appears on more than one device.
Quoted 2 lines
Yes, that would work as a workaround, but it still seems like something worth fixing.
One possible solution is to not send MTU errors to ourselves since we it wouldn't give us any new information. We'd need to audit the users of icmp_send to make sure that there isn't a legitimate case where we'd want to do that.
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt