virtio GSO makes IPv6 unbearably slow

Hi,

I have a really weird issue all of the sudden on _one_ of my two KVM hosts. The other one, while running on a different hardware in a different network, is configured in a very similar way and does not show these issues (so far).

Host:
- AMD Athlon64 3500+
- Debian testing amd64
- Kernel 2.6.32-trunk from Debian
- Debian qemu-kvm 0.11.1+dfsg-1

The system has a routed uplink (Intel e100) and an internal bridge, that connects all the VMs (only virtual ports). Networking in the VMs is usually configured like this:

-net nic,model=virtio,macaddr=00:16:3E:7C:30:AA -net tap,ifname=vm.compile,script=no

The guests are Debian amd64, either testing or stable, running a custom stripped 2.6.31.6 or 2.6.32.7. I've also tested older kernels (2.6.28.2 and 2.6.30.4) and have seen the same problem.

Problem:
IPv6 tx throughput from a Guest to a host in the internet is extremely low (around 16kB/s). IPv6 rx is fine, as well as throughput in IPv4 (both ways). Also fine is throughput from the guest to the host and from the host to the internet (also both ways).

Running tcpdump on the tap-device on the host while doing an scp to another system shows this:

20:43:27.130438 IP6 GUEST.59864 > DEST.22: Flags [.], seq 70785:73641, ack 2128, win 327, options [nop,nop,TS val 4294920148 ecr 63373335], length 2856 20:43:27.130506 IP6 HOST > GUEST: ICMP6, packet too big, mtu 1500, length 1240
20:43:27.131965 IP6 DEST.22 > GUEST.59864: Flags [.], ack 69357, win 501, options [nop,nop,TS val 63373335 ecr 4294920144], length 0 20:43:27.131996 IP6 DEST.22 > GUEST.59864: Flags [.], ack 70785, win 501, options [nop,nop,TS val 63373335 ecr 4294920144], length 0 20:43:27.132651 IP6 GUEST.59864 > DEST.22: Flags [.], seq 73641:76497, ack 2128, win 327, options [nop,nop,TS val 4294920148 ecr 63373335], length 2856 20:43:27.132704 IP6 HOST > GUEST: ICMP6, packet too big, mtu 1500, length 1240
20:43:27.346347 IP6 GUEST.59864 > DEST.22: Flags [.], seq 70785:72213, ack 2128, win 327, options [nop,nop,TS val 4294920202 ecr 63373335], length 1428 20:43:27.360411 IP6 DEST.22 > GUEST.59864: Flags [.], ack 72213, win 501, options [nop,nop,TS val 63373358 ecr 4294920202], length 0 20:43:27.361045 IP6 GUEST.59864 > DEST.22: Flags [.], seq 76497:79353, ack 2128, win 327, options [nop,nop,TS val 4294920205 ecr 63373358], length 2856

So, the guest sends packets of 2856 bytes, which are too large to pass through the eth0 of the host (1500 bytes). Thus, the host rejects the packet and sends back an ICMPv6 packet too big, which sometimes makes the guest send at least one packet with the small MTU, but sometimes it doesn't. This is repeated all over and slows down the process.

According to ethtool GSO is enabled on the guest, but disabling it using ethtool does not have any effect. But giving the kernel "virtio_net.gso=0" in append fixes the issue completely.

Anyone having any ideas?

Best Regards,
Bernhard

I ran into a similar problem when using Intel e1000e driver on the host. Found a bug in most of the intel ethernet drivers when handling large GSO IPv6 packets. You can see the fix i submitted here

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=8e1e8a4779cb23c1d9f51e9223795e07ec54d77a

But you seem to be using Intel e100 driver. So i am not sure you are running into the same issue. You could try disabling both tso and gso on the guest using ethtool.

On the host, do you have gso/tso enabled on the uplink e100 driver?

Thanks
Sridhar

Disabling anything inside the guest doesn't help anything, as told above. I have to disable gso using a kernel parameter in the virtio_net driver.

On the host everything is disabled.

Offload parameters for eth0:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off

Best Regards,
Bernhard