xfrm: is pmtu broken with ESP tunneling?

Hi,

I am using Openswan to configure an IPSec VPN (using the xfrm/netkey backend). Large HTTP POST requests from the client seem to get stuck, because the outgoing packets are 1530 bytes (before being wrapped into ESP packets). The problem goes away by setting sysctl net.ipv4.ip_no_pmtu_disc=1.

May have something to do with it:
The tunneled network is 10.6.6.6/32 and I am SNAT'ing some destinations to that IP, so they get routed through the tunnel. Any other networks are not to go through the tunnel.

iptables -t nat -A POSTROUTING -d "${R}" -j SNAT --to-source 10.6.6.6

It seems quite clear to me that xfrm is doing something wrong here.

Ortwin

Hi!

This setting will shrink the path mtu to min_pmtu when a frag needed icmp is received. It sounds like we calculate the path mtu incorreclty in case of fragmentation.

Can you send a ip route get <ip> to the problematic target to see how far off the calculated value is?

Thanks,

Hannes

The UDP+ESP encapsulation adds 60 bytes to the original packet size.

ifconfig wla0 shows an mtu of 1500.

The size of the first big packet on the interface:
net.ipv4.ip_no_pmtu_disc=1: packet length is 1300 net.ipv4.ip_no_pmtu_disc=0: packet length is 1500

Length is without the ESP wrapper and UDP encapsulation. The packets are so big that they can't even leave the wireless interface and never show up on the router. So no ICMP packets are received. PMTU can't work with initial packets of that size.

dump question: which layer discard these packets? qdisc? why no notification to the sender?

When I increase the mtu of the interface to 2000 with ifconfig, then I start seeing ICMP fragmentation needed from the next hop, indicating 1500 as the mtu as response to a 1560 byte UDP[ESP] packet.

The next UDP[ESP] packet is shorter: 1360 bytes. It gets hard to see what's going on after that, but the connection is still not working.

So, instead of somehow losing these packets on the way out of the interface should the kernel not start with a lower mtu in the first place? Now it seems it is trying with the maximum of the interface and expecting to scale down with pmtu - which can ever happen.

That command doesn't return anything useful. No hint on the mtu here.

BTW, instead of disabling pmtu, setting mtu explicitly also helps:
ip route add 10.6.6.0/24 via ${localip} mtu 1300

Thanks,

Ortwin

Could you try either dropwatch or perf script net_dropmonitor and flood the network with the problematic packets. From the traces we could see where the packets get dropped without notification in the kernel.

Strange that the problem disappears if you enable no_pmtu_disc then.

Thanks,

Hannes

Not much to see, unfortunately. The COUNT doesn't reflect the number packets that I am missing.

LOCATION OFFSET COUNT

ieee80211_iface_work 208 1

It seems with PMTU the initial mtu is the one of the device (1500). So the original packet will have that size, but is subsequently wrapped into ESP and UDP, which add to that size. And the final packet is then larger than the device MTU... I know nothing about the ip / xfrm kernel code, so it's hard for me to verify if that theory is real.

Thanks,

Ortwin