persistent tun & different virtual NICs & dead guest network

DISCUSSION Michael Tokarev 3 participants 4 messages Apr 4, 2009 – Jul 9, 2009 KVM

Share & tools

Michael Tokarev

author

2009-04-04 10:26 UTC #0 lore raw

Hello.

2 days debugging an.. issue here, and finally got it. To make the long and painful (it was for me anyway) story short...

kvm provides a way to control various offload settings on the "host side" of the tun network device (I mean the `-net tap' setup) from within guest. I.e., guest can set/clear various offload bits according to its capabilities/wishes.

The problem is that different virtual NICs as used by kvm/qemu expects and sets different offload bits for the virtual NIC. And sets only those bits which - as they "think" - differs from the default (all-off).

This means that when changing virtual NIC model AND using persistent tun device, it's very likely to get inconsistent flags.

For example, here's how the offload settings on the host looks like after using e1000 driver in guest (freshly created persistent tun device):

rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: off
large receive offload: off

Here's the same setting when using virtio_net instead:

rx-checksumming: on
tx-checksumming: off
scatter-gather: off
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
large receive offload: off

I.e., only rx-checksumming. When using virtio_net from 2.6.29, which supports LRO, it also turns on large receive offload.

Now, say, I tried a host with e1000 driver, and it turned on tx, sg and tso bits. And now I'm trying to run a guest with new virtio-net NIC instead. It turns on lro bit, but the network does not work anyway:
almost any packet that's being sent from host to the guest has incorrect checksum - because the NIC is marked as able to do tx-checksumming but it does not do it. The network is dead.

Now, after trying that and this, not understanding what's going on etc, let's reboot back with e1000 NIC which worked a few minutes ago... just to discover that it does not work anymore too! Because previous attempt with virtio_net resulted in lro being on, but the driver does not support it! So now, we've non- working network again, and now, it does not matter which driver we'll try: neither of them will work because the offload settings are broken.

It's more: one can't control this stuff from the host side using standard ethtool: it says that the operation is not supported (I wonder how kvm performs the settings changes).

The solution here is to re-create the tun device before changing the virtual NIC model. But it isn't always possible, esp. when guests are being run from non-root user (where persistent tun devices are most useful).

Can this be fixed somehow please?

I think all the settings should be reset to 0 when opening the tun device.

Thanks.

/mjt,
who lost 2 more days and had another sleepless night trying to understand what's going wrong...

Avi Kivity

2009-04-05 11:58 UTC #1 lore raw

(cc qemu-devel)

Quoted 78 lines · Michael Tokarev

Hello.
2 days debugging an.. issue here, and finally got it. To make the long and painful (it was for me anyway) story short...
kvm provides a way to control various offload settings on the "host side" of the tun network device (I mean the `-net tap' setup) from within guest. I.e., guest can set/clear various offload bits according to its capabilities/wishes.
The problem is that different virtual NICs as used by kvm/qemu expects and sets different offload bits for the virtual NIC. And sets only those bits which - as they "think" - differs from the default (all-off).
This means that when changing virtual NIC model AND using persistent tun device, it's very likely to get inconsistent flags.
For example, here's how the offload settings on the host looks like after using e1000 driver in guest (freshly created persistent tun device):
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: off
large receive offload: off
Here's the same setting when using virtio_net instead:
rx-checksumming: on
tx-checksumming: off
scatter-gather: off
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
large receive offload: off
I.e., only rx-checksumming. When using virtio_net from 2.6.29, which supports LRO, it also turns on large receive offload.
Now, say, I tried a host with e1000 driver, and it turned on tx, sg and tso bits. And now I'm trying to run a guest with new virtio-net NIC instead. It turns on lro bit, but the network does not work anyway:
almost any packet that's being sent from host to the guest has incorrect checksum - because the NIC is marked as able to do tx-checksumming but it does not do it. The network is dead.
Now, after trying that and this, not understanding what's going on etc, let's reboot back with e1000 NIC which worked a few minutes ago... just to discover that it does not work anymore too! Because previous attempt with virtio_net resulted in lro being on, but the driver does not support it! So now, we've non- working network again, and now, it does not matter which driver we'll try: neither of them will work because the offload settings are broken.
It's more: one can't control this stuff from the host side using standard ethtool: it says that the operation is not supported (I wonder how kvm performs the settings changes).
The solution here is to re-create the tun device before changing the virtual NIC model. But it isn't always possible, esp. when guests are being run from non-root user (where persistent tun devices are most useful).

Quoted 4 lines

Can this be fixed somehow please?
I think all the settings should be reset to 0 when opening the tun device.

This should definitely be fixed. I'll look at writing a patch.

Signature · 1 line

error compiling committee.c: too many arguments to function

Avi Kivity

2009-04-05 12:12 UTC #2 lore raw

Quoted 5 lines · Avi Kivity

Quoted 2 lines
I think all the settings should be reset to 0
when opening the tun device.
This should definitely be fixed. I'll look at writing a patch.

Okay, that's not in upstream qemu, so I committed a fix to kvm-userspace.git.

Attached if you want to test it.

Signature · 1 line

error compiling committee.c: too many arguments to function

Quoted 1 line

From 25971710409c374e9486c960c297f324a9164a65 Mon Sep 17 00:00:00 2001

From: Avi Kivity <[email protected]>
Date: Sun, 5 Apr 2009 15:08:55 +0300
Subject: [PATCH] kvm: qemu: clear tap features on initialization

tap features change how tap interprets data, so they must be cleared on initialization to prevent old settings from interfering with new guest instances.

Signed-off-by Avi Kivity <[email protected]>

---
 qemu/net.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

qemu/net.c +1 -0

index d753fa0..703d01c 100644
@@ -930,6 +930,7 @@ static TAPState *net_tap_fd_init(VLANState *vlan,
 #endif
 #ifdef TUNSETOFFLOAD
     s->vc->set_offload = tap_set_offload;
+    tap_set_offload(s->vc, 0, 0, 0, 0);
 #endif
     qemu_set_fd_handler2(s->fd, tap_can_send, tap_send, NULL, s);
     snprintf(s->vc->info_str, sizeof(s->vc->info_str), "fd=%d", fd);

Signature · 1 line

1.6.0.6

Michael S. Tsirkin

2009-07-09 18:14 UTC #3 lore raw

Quoted 35 lines · Avi Kivity

Quoted 5 lines · Avi Kivity
Quoted 2 lines
I think all the settings should be reset to 0
when opening the tun device.
This should definitely be fixed. I'll look at writing a patch.
Okay, that's not in upstream qemu, so I committed a fix to kvm-userspace.git.
Attached if you want to test it.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 1 line
From 25971710409c374e9486c960c297f324a9164a65 Mon Sep 17 00:00:00 2001
From: Avi Kivity <[email protected]>
Date: Sun, 5 Apr 2009 15:08:55 +0300
Subject: [PATCH] kvm: qemu: clear tap features on initialization
tap features change how tap interprets data, so they must be cleared on initialization to prevent old settings from interfering with new guest instances.
Signed-off-by Avi Kivity <[email protected]>
---
 qemu/net.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
qemu/net.c +0 -0
index d753fa0..703d01c 100644

qemu/net.c static TAPState *net_tap_fd_init(VLANState *vlan, · line 933

Quoted 4 lines

qemu/net.c +1 -0

@@ -930,6 +930,7 @@ static TAPState *net_tap_fd_init(VLANState *vlan,
 #endif
 #ifdef TUNSETOFFLOAD
     s->vc->set_offload = tap_set_offload;
+    tap_set_offload(s->vc, 0, 0, 0, 0);

BTW, should not these bits be restored on load?
I couldn't find code that does this.

qemu/net.c static TAPState *net_tap_fd_init(VLANState *vlan, · line 936

Quoted 3 lines

qemu/net.c +0 -0

 #endif
     qemu_set_fd_handler2(s->fd, tap_can_send, tap_send, NULL, s);
     snprintf(s->vc->info_str, sizeof(s->vc->info_str), "fd=%d", fd);

Looks good to me. Just a thought: do we want kernel to give us an option to tie the offload bits to the character device, so that it gets cleaned automatically when qemu dies?

Signature · 1 line

MST