TAP MTU >= 4055 problem?
Hey all,
I'm using KVM to setup a virtual machine. Using this machine i'm attempting to setup a connection to host machine using the tap interface. I setup the interface using the standard tap method and then start the KVM using the following:
kvm -net nic,model=e1000 -net tap,ifname=qtap0,script/etc/kvm/kvm-ifup -drive file=./image.img
When the virual machine starts i setup the IP addresses (host = 192.168.100.1 vm = 192.168.100.2) and I am able to bring between the host and virtual machine. However, what i need to do is to be able to send jumbo frames between the two. I therefore increase the MTU size on both the host machine and virutal machine to 9000.
Now when i try to do from within the host
ping -s 4096 192.168.100.1
I go no respone. So i started with a lower packet size and figured out below a size of 4054 packets were sent and recevied (without ip fragmentation), however, as soon as the packets were >= 4055 it stopped working.
Is this a known problem? Have I set something up incorrectly?
Thanks for your help
P.S. I'm not on the kvm mailing list so please cc me with any replies
Quoted 6 lines · Matthew Faulkner
I go no respone. So i started with a lower packet size and figured out below a size of 4054 packets were sent and recevied (without ip fragmentation), however, as soon as the packets were >= 4055 it stopped working.
Is this a known problem? Have I set something up incorrectly?
are you using a bridge? it has some problems with long packets.
if you can, try without a bridge. if not, make sure that you set the MTU of both ethX and tapX to 9000 before adding anything to the bridge. i think the bridge itself has its own MTU that is set according to the interfaces it gets; but i don't know if it can change MTU when an interface changes.
Signature · 1 line
Javier
I'm not using a bridge. And i have set the mtu on both tapX and ethX to 9000.
Having done a little packet sniffing from the hostmachine -> virtual machine I can see that packets of the correct (large) size are being sent. Whilst sniffing on the virtual machine I don't see any packets. This indicates to me that the network emulation drivers in KVM aren't setup correctly?
I'm using the e1000 drivers.
Anyone any thoughts?
2008/10/30 Javier Guerra <[email protected]>:
Quoted 21 lines
Quoted 6 lines · Matthew Faulkner
I go no respone. So i started with a lower packet size and figured out below a size of 4054 packets were sent and recevied (without ip fragmentation), however, as soon as the packets were >= 4055 it stopped working.
Is this a known problem? Have I set something up incorrectly?
are you using a bridge? it has some problems with long packets.
if you can, try without a bridge. if not, make sure that you set the MTU of both ethX and tapX to 9000 before adding anything to the bridge. i think the bridge itself has its own MTU that is set according to the interfaces it gets; but i don't know if it can change MTU when an interface changes.
--
Javier
For additionial information, i've placed some printf's within KVMs e1000 emualted drivers.
It turns out the within the e1000 receive function (e1000_receive) the correct size is received. Somewhere between there and the upper layers something goes wrong. I am continuing to investiage this problem, but if anyone else shed some light on this I would be greatful.
2008/10/30 Matthew Faulkner <[email protected]>:
Quoted 35 lines
I'm not using a bridge. And i have set the mtu on both tapX and ethX to 9000.
Having done a little packet sniffing from the hostmachine -> virtual machine I can see that packets of the correct (large) size are being sent. Whilst sniffing on the virtual machine I don't see any packets. This indicates to me that the network emulation drivers in KVM aren't setup correctly?
I'm using the e1000 drivers.
Anyone any thoughts?
2008/10/30 Javier Guerra <[email protected]>:
Quoted 21 lines
Quoted 6 lines · Matthew Faulkner
I go no respone. So i started with a lower packet size and figured out below a size of 4054 packets were sent and recevied (without ip fragmentation), however, as soon as the packets were >= 4055 it stopped working.
Is this a known problem? Have I set something up incorrectly?
are you using a bridge? it has some problems with long packets.
if you can, try without a bridge. if not, make sure that you set the MTU of both ethX and tapX to 9000 before adding anything to the bridge. i think the bridge itself has its own MTU that is set according to the interfaces it gets; but i don't know if it can change MTU when an interface changes.
--
Javier
Hey all,
I've solved the problem. Turns out the tap buffers were set at the size 4096. When pulling a packet off that is greater than 4096 the 2nd packet (for some reason) is thrown aaway or ignored.
I've done a hacky solutoin by simply increasing this number of 65536.
2008/10/30 Matthew Faulkner <[email protected]>:
Quoted 46 lines
For additionial information, i've placed some printf's within KVMs e1000 emualted drivers.
It turns out the within the e1000 receive function (
e1000_receive) the correct size is received. Somewhere between there and the upper layers something goes wrong. I am continuing to investiage this problem, but if anyone else shed some light on this I would be greatful.2008/10/30 Matthew Faulkner <[email protected]>:
Quoted 35 lines
I'm not using a bridge. And i have set the mtu on both tapX and ethX to 9000.
Having done a little packet sniffing from the hostmachine -> virtual machine I can see that packets of the correct (large) size are being sent. Whilst sniffing on the virtual machine I don't see any packets. This indicates to me that the network emulation drivers in KVM aren't setup correctly?
I'm using the e1000 drivers.
Anyone any thoughts?
2008/10/30 Javier Guerra <[email protected]>:
Quoted 21 lines
Quoted 6 lines · Matthew Faulkner
I go no respone. So i started with a lower packet size and figured out below a size of 4054 packets were sent and recevied (without ip fragmentation), however, as soon as the packets were >= 4055 it stopped working.
Is this a known problem? Have I set something up incorrectly?
are you using a bridge? it has some problems with long packets.
if you can, try without a bridge. if not, make sure that you set the MTU of both ethX and tapX to 9000 before adding anything to the bridge. i think the bridge itself has its own MTU that is set according to the interfaces it gets; but i don't know if it can change MTU when an interface changes.
--
Javier
PATCH 1/1 kvm: qemu: increase the size of the tap buffer [was Re: TAP MTU >= 4055 problem?]
Hi Matthew,
Quoted 7 lines · Matthew Faulkner
Hey all,
I've solved the problem. Turns out the tap buffers were set at the size 4096. When pulling a packet off that is greater than 4096 the 2nd packet (for some reason) is thrown aaway or ignored.
I've done a hacky solutoin by simply increasing this number of 65536.
Thanks for debugging - that makes a whole lot of sense.
To explain in more detail - if you set the tap MTU size to 9000 and do ping -s 4055
1) the host's networking stack will construct an ethernet frame of 4097 (4055 bytes ICMP data, 8 bytes ICMP header, 20 bytes TCP header and 14 bytes ethernet header)
2) qemu will read the frame from the tap device into it's 4096 byte buffer causing the kernel to truncate it to 4096 bytes
3) qemu passes that on to the guest and the guest's IP stack will see that there is less data in the buffer than is specified in the IP header. It then silently drops the packet (see InTruncatedPkts in /proc/net/netstat)
It sucks that the kernel just silently truncates and doesn't give userspace a chance to retry with a larger buffer. We could add that as an extension, I guess.
Best short term thing to do is just increase the tap buffer size. Patch below. A similar patch should go into qemu upstream.
Cheers,
Mark.
From: Mark McLoughlin <[email protected]>
Subject: [PATCH] kvm: qemu: increase size of tun/tap buffer
As debugged by Matthew Faulkner, with a 4k byte tap buffer if you increase the MTU on the tap device to greater than 4k, the packets read by qemu into the tap buffer will be truncated and the guest will discard the packet.
With GSO enabled, we use a 64k tap buffer, so let's just use a 64k buffer in all cases.
Also, remove the obtuse logic for figuring out the max GSO buffer size. We shouldn't receive IP packets larger than 64k, so let's just use 17 pages to make sure we've enough room for headers.
--- qemu/vl.c | 14 ++++++-------- 1 files changed, 6 insertions(+), 8 deletions(-)
qemu/vl.c +6 -8
@@ -4413,15 +4413,13 @@ void tap_using_vnet_hdr(void *opaque, int using_vnet_hdr) #else /* !defined(_WIN32) */ -#ifndef IFF_VNET_HDR -#define TAP_BUFSIZE 4096 -#else +/* Maximum GSO packet size (64k) plus plenty of room for + * the ethernet and virtio_net headers + */ +#define TAP_BUFSIZE (4096 + 65536) + +#ifdef IFF_VNET_HDR #include <linux/virtio_net.h> -#define ETH_HLEN 14 -#define ETH_DATA_LEN 1500 -#define MAX_PACKET_LEN (ETH_HLEN + ETH_DATA_LEN) -#define MAX_SKB_FRAGS ((65536/TARGET_PAGE_SIZE) + 2) -#define TAP_BUFSIZE (sizeof(struct virtio_net_hdr) + MAX_PACKET_LEN + (MAX_SKB_FRAGS*TARGET_PAGE_SIZE)) #endif typedef struct TAPState {
Signature · 1 line
1.6.0.1
Avi Kivity
AppliedRe: [PATCH] kvm: qemu: increase the size of the tap buffer [was Re: TAP MTU >= 4055 problem?]
Quoted 5 lines · Mark McLoughlin
It sucks that the kernel just silently truncates and doesn't give userspace a chance to retry with a larger buffer. We could add that as an extension, I guess.
What we want even more is to give the kernel a set of buffers (iovec) and have it fill them as much as it can with a number of packets, giving us some info as to where frames begin. Manage the whole thing using aio reads.
Quoted 5 lines
As debugged by Matthew Faulkner, with a 4k byte tap buffer if you increase the MTU on the tap device to greater than 4k, the packets read by qemu into the tap buffer will be truncated and the guest will discard the packet.
Quoted 8 lines
With GSO enabled, we use a 64k tap buffer, so let's just use a 64k buffer in all cases.
Also, remove the obtuse logic for figuring out the max GSO buffer size. We shouldn't receive IP packets larger than 64k, so let's just use 17 pages to make sure we've enough room for headers.
Applied, thanks.
Signature · 1 line
error compiling committee.c: too many arguments to function