network shutdown under heavy load

Hello, we notice that when we stress any of our guests, in this case they are all fedora, the KVM network will shutdown.. anyone experience this?

Quoted 3 lines · On 12/14/2009 05:49 PM, rek2 wrote:

Hello, we notice that when we stress any of our guests, in this case they are all fedora, the KVM network will shutdown.. anyone experience this?

Herbert?

What's the exact guest kernel version? When the network is down, please get onto the guest console to determine which direction (if not both) of the network is not functioning.

You can run tcpdump in the guest/host and execute pings on both sides to see which direction is blocked.

Cheers,

I been told that today the network when down again and one of the guys here had to log using the console and restart it for that particular guests..

on the guest:
uname -a
Linux XXXX 2.6.27.25-170.2.72.fc10.x86_64 #1 SMP Sun Jun 21 18:39:34 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

Next time it goes down I will try to run a sniffer and try both sides.

Quoted 11 lines · On Thu, Dec 17, 2009 at 01:15:46PM -0500, rek2 wrote:

I been told that today the network when down again and one of the guys here had to log using the console and restart it for that particular guests..

on the guest:
uname -a
Linux XXXX 2.6.27.25-170.2.72.fc10.x86_64 #1 SMP Sun Jun 21 18:39:34 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

Next time it goes down I will try to run a sniffer and try both sides.

OK I'm fairly sure this version has a buggy virtio-net. Does this patch (if it applies :) help?

drivers/net/virtio_net.c +3 -1

index 9eec5a5..74b3854 100644
@@ -521,8 +521,10 @@ static void xmit_tasklet(unsigned long data)
  		vi->svq->vq_ops->kick(vi->svq);
  		vi->last_xmit_skb = NULL;
  	}
-	if (vi->free_in_tasklet)
+	if (vi->free_in_tasklet) {
  		free_old_xmit_skbs(vi);
+		netif_wake_queue(vi->dev);
+	}
  	netif_tx_unlock_bh(vi->dev);
  }
 

Cheers,

You say this version.. is there a newer version with this patch already apply to it?

Thanks

Quoted 31 lines · Herbert Xu

Quoted 10 lines · On Thu, Dec 17, 2009 at 01:15:46PM -0500, rek2 wrote:

I been told that today the network when down again and one of the guys here had to log using the console and restart it for that particular guests..

on the guest:
uname -a
Linux XXXX 2.6.27.25-170.2.72.fc10.x86_64 #1 SMP Sun Jun 21 18:39:34 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

Next time it goes down I will try to run a sniffer and try both sides.

OK I'm fairly sure this version has a buggy virtio-net. Does this patch (if it applies :) help?

drivers/net/virtio_net.c +3 -1

index 9eec5a5..74b3854 100644
@@ -521,8 +521,10 @@ static void xmit_tasklet(unsigned long data)
          vi->svq->vq_ops->kick(vi->svq);
          vi->last_xmit_skb = NULL;
      }
-    if (vi->free_in_tasklet)
+    if (vi->free_in_tasklet) {
          free_old_xmit_skbs(vi);
+        netif_wake_queue(vi->dev);
+    }
      netif_tx_unlock_bh(vi->dev);
  }
 

Cheers,

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hi guys, it happen again (in this server I didn't patch with the fix you guys sent) but I did this so if it happen i can test with tcpdump.. seems that the guest can receive packages but can't sent... when I open a tcpdump I saw traffic coming in, but not out.

Hope this helps..
also I need to know if the patch you guys sent me will be in newer versions, if not I like to know since I can't update.

This isn't in 2.6.27.y. Herbert, can you send it there?

Quoted 2 lines · Avi Kivity

This isn't in 2.6.27.y. Herbert, can you send it there?

It appears that now that TX is fixed we have a similar problem with RX. Once I figure that one out I'll send them together.

Who is maintaining that BTW, [email protected]?

Quoted 5 lines · Herbert Xu

Quoted 1 line · Avi Kivity

This isn't in 2.6.27.y. Herbert, can you send it there?

It appears that now that TX is fixed we have a similar problem with RX. Once I figure that one out I'll send them together.

Thanks.

Quoted 1 line

Who is maintaining that BTW, [email protected]?

Yes.

Quoted 7 lines · Avi Kivity

Quoted 5 lines · Herbert Xu

Quoted 1 line · Avi Kivity

This isn't in 2.6.27.y. Herbert, can you send it there?

It appears that now that TX is fixed we have a similar problem with RX. Once I figure that one out I'll send them together.

I've been experiencing the network shutdown issue also. I've been running netperf tests across 10GbE adapters with Qemu 0.12.1.2, RHEL5.4 guests and 2.6.32 kernel (from kvm.git) guests. I instrumented Qemu to print out some network statistics. It appears that at some point in the netperf test the receiving guest ends up having the 10GbE device "receive_disabled" variable in its VLANClientState structure stuck at 1. From looking at the code it appears that the virtio-net driver in the guest should cause qemu_flush_queued_packets in net.c to eventually run and clear the "receive_disabled" variable but it's not happening. I don't seem to have these issues when I have a lot of debug settings active in the guest kernel which results in very low/poor network performance - maybe some kind of race condition?

(Mark cc'd, sound familiar?)

Quoted 20 lines · * Tom Lendacky ([email protected]) wrote:

Quoted 6 lines · Avi Kivity

Quoted 5 lines · Herbert Xu

Quoted 1 line · Avi Kivity

This isn't in 2.6.27.y. Herbert, can you send it there?

It appears that now that TX is fixed we have a similar problem with RX. Once I figure that one out I'll send them together.

I've been experiencing the network shutdown issue also. I've been running netperf tests across 10GbE adapters with Qemu 0.12.1.2, RHEL5.4 guests and 2.6.32 kernel (from kvm.git) guests. I instrumented Qemu to print out some network statistics. It appears that at some point in the netperf test the receiving guest ends up having the 10GbE device "receive_disabled" variable in its VLANClientState structure stuck at 1. From looking at the code it appears that the virtio-net driver in the guest should cause qemu_flush_queued_packets in net.c to eventually run and clear the "receive_disabled" variable but it's not happening. I don't seem to have these issues when I have a lot of debug settings active in the guest kernel which results in very low/poor network performance - maybe some kind of race condition?

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 23 lines · Chris Wright

(Mark cc'd, sound familiar?)

Quoted 20 lines · * Tom Lendacky ([email protected]) wrote:

Quoted 6 lines · Avi Kivity

Quoted 5 lines · Herbert Xu

Quoted 1 line · Avi Kivity

This isn't in 2.6.27.y. Herbert, can you send it there?

It appears that now that TX is fixed we have a similar problem with RX. Once I figure that one out I'll send them together.

I've been experiencing the network shutdown issue also. I've been running netperf tests across 10GbE adapters with Qemu 0.12.1.2, RHEL5.4 guests and 2.6.32 kernel (from kvm.git) guests. I instrumented Qemu to print out some network statistics. It appears that at some point in the netperf test the receiving guest ends up having the 10GbE device "receive_disabled" variable in its VLANClientState structure stuck at 1. From looking at the code it appears that the virtio-net driver in the guest should cause qemu_flush_queued_packets in net.c to eventually run and clear the "receive_disabled" variable but it's not happening. I don't seem to have these issues when I have a lot of debug settings active in the guest kernel which results in very low/poor network performance - maybe some kind of race condition?

Ok, here's an update. After realizing that none of the ethtool offload options were enabled in my guest, I found that I needed to be using the -netdev option on the qemu command line. Once I did that, some ethtool offload options were enabled and the deadlock did not appear when I did networking between guests on different machines. However, the deadlock did appear when I did networking between guests on the same machine.

Quoted 26 lines · * Tom Lendacky ([email protected]) wrote:

Quoted 24 lines · Chris Wright

(Mark cc'd, sound familiar?)

Quoted 21 lines · * Tom Lendacky ([email protected]) wrote:

Quoted 6 lines · Avi Kivity

Quoted 5 lines · Herbert Xu

Quoted 1 line · Avi Kivity

This isn't in 2.6.27.y. Herbert, can you send it there?

It appears that now that TX is fixed we have a similar problem with RX. Once I figure that one out I'll send them together.

I've been experiencing the network shutdown issue also. I've been running netperf tests across 10GbE adapters with Qemu 0.12.1.2, RHEL5.4 guests and 2.6.32 kernel (from kvm.git) guests. I instrumented Qemu to print out some network statistics. It appears that at some point in the netperf test the receiving guest ends up having the 10GbE device "receive_disabled" variable in its VLANClientState structure stuck at 1. From looking at the code it appears that the virtio-net driver in the guest should cause qemu_flush_queued_packets in net.c to eventually run and clear the "receive_disabled" variable but it's not happening. I don't seem to have these issues when I have a lot of debug settings active in the guest kernel which results in very low/poor network performance - maybe some kind of race condition?

Quoted 6 lines

Ok, here's an update. After realizing that none of the ethtool offload options were enabled in my guest, I found that I needed to be using the -netdev option on the qemu command line. Once I did that, some ethtool offload options were enabled and the deadlock did not appear when I did networking between guests on different machines. However, the deadlock did appear when I did networking between guests on the same machine.

What does your full command line look like? And when the networking stops does your same receive_disabled hack make things work?

thanks,
-chris
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 34 lines · Chris Wright

Quoted 26 lines · * Tom Lendacky ([email protected]) wrote:

Quoted 24 lines · Chris Wright

(Mark cc'd, sound familiar?)

Quoted 21 lines · * Tom Lendacky ([email protected]) wrote:

Quoted 6 lines · Avi Kivity

Quoted 5 lines · Herbert Xu

Quoted 1 line · Avi Kivity

This isn't in 2.6.27.y. Herbert, can you send it there?

It appears that now that TX is fixed we have a similar problem with RX. Once I figure that one out I'll send them together.

I've been experiencing the network shutdown issue also. I've been running netperf tests across 10GbE adapters with Qemu 0.12.1.2, RHEL5.4 guests and 2.6.32 kernel (from kvm.git) guests. I instrumented Qemu to print out some network statistics. It appears that at some point in the netperf test the receiving guest ends up having the 10GbE device "receive_disabled" variable in its VLANClientState structure stuck at 1. From looking at the code it appears that the virtio-net driver in the guest should cause qemu_flush_queued_packets in net.c to eventually run and clear the "receive_disabled" variable but it's not happening. I don't seem to have these issues when I have a lot of debug settings active in the guest kernel which results in very low/poor network performance
- maybe some kind of race condition?

Quoted 6 lines

Ok, here's an update. After realizing that none of the ethtool offload options were enabled in my guest, I found that I needed to be using the -netdev option on the qemu command line. Once I did that, some ethtool offload options were enabled and the deadlock did not appear when I did networking between guests on different machines. However, the deadlock did appear when I did networking between guests on the same machine.

Quoted 2 lines

What does your full command line look like? And when the networking stops does your same receive_disabled hack make things work?

The command line when using the -net option for the tap device is:

/usr/local/bin/qemu-system-x86_64 -name cape-vm001 -m 1024 -drive file=/autobench/var/tmp/cape-vm001- raw.img,if=virtio,index=0,media=disk,boot=on -net nic,model=virtio,vlan=0,macaddr=00:16:3E:00:62:51 -net tap,vlan=0,script=/autobench/var/tmp/ifup-kvm- br0,downscript=/autobench/var/tmp/ifdown-kvm-br0 -net nic,model=virtio,vlan=1,macaddr=00:16:3E:00:62:D1 -net tap,vlan=1,script=/autobench/var/tmp/ifup-kvm- br1,downscript=/autobench/var/tmp/ifdown-kvm-br1 -vnc :1 -monitor telnet::5701,server,nowait -snapshot -daemonize

when using the -netdev option for the tap device:

/usr/local/bin/qemu-system-x86_64 -name cape-vm001 -m 1024 -drive file=/autobench/var/tmp/cape-vm001- raw.img,if=virtio,index=0,media=disk,boot=on -net nic,model=virtio,vlan=0,macaddr=00:16:3E:00:62:51,netdev=cape-vm001-eth0 - netdev tap,id=cape-vm001-eth0,script=/autobench/var/tmp/ifup-kvm- br0,downscript=/autobench/var/tmp/ifdown-kvm-br0 -net nic,model=virtio,vlan=1,macaddr=00:16:3E:00:62:D1,netdev=cape-vm001-eth1 - netdev tap,id=cape-vm001-eth1,script=/autobench/var/tmp/ifup-kvm- br1,downscript=/autobench/var/tmp/ifdown-kvm-br1 -vnc :1 -monitor telnet::5701,server,nowait -snapshot -daemonize

The first ethernet device is a 1GbE device for communicating with the automation infrastructure we have. The second ethernet device is the 10GbE device that the netperf tests run on.

I can get the networking to work again by bringing down the interfaces and reloading the virtio_net module (modprobe -r virtio_net / modprobe virtio_net).

I haven't had a chance yet to run the tests against a modified version of
qemu that does not set the receive_disabled variable.

Tom

Quoted 6 lines

thanks,
-chris
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 76 lines · Tom Lendacky

Quoted 40 lines · Chris Wright

Quoted 31 lines · * Tom Lendacky ([email protected]) wrote:

Quoted 29 lines · Chris Wright

(Mark cc'd, sound familiar?)

Quoted 25 lines · * Tom Lendacky ([email protected]) wrote:

Quoted 9 lines · Avi Kivity

Quoted 7 lines · Herbert Xu

Quoted 2 lines · Avi Kivity

This isn't in 2.6.27.y. Herbert, can you send it there?

It appears that now that TX is fixed we have a similar problem with RX. Once I figure that one out I'll send them together.

I've been experiencing the network shutdown issue also. I've been running netperf tests across 10GbE adapters with Qemu 0.12.1.2, RHEL5.4 guests and 2.6.32 kernel (from kvm.git) guests. I instrumented Qemu to print out some network statistics. It appears that at some point in the netperf test the receiving guest ends up having the 10GbE device "receive_disabled" variable in its VLANClientState structure stuck at 1. From looking at the code it appears that the virtio-net driver in the guest should cause qemu_flush_queued_packets in net.c to eventually run and clear the "receive_disabled" variable but it's not happening. I don't seem to have these issues when I have a lot of debug settings active in the guest kernel which results in very low/poor network performance
- maybe some kind of race condition?

Quoted 7 lines

Ok, here's an update. After realizing that none of the ethtool offload options were enabled in my guest, I found that I needed to be using the -netdev option on the qemu command line. Once I did that, some ethtool offload options were enabled and the deadlock did not appear when I did networking between guests on different machines. However, the deadlock did appear when I did networking between guests on the same machine.

Quoted 3 lines

What does your full command line look like? And when the networking stops does your same receive_disabled hack make things work?

The command line when using the -net option for the tap device is:

/usr/local/bin/qemu-system-x86_64 -name cape-vm001 -m 1024 -drive file=/autobench/var/tmp/cape-vm001- raw.img,if=virtio,index=0,media=disk,boot=on -net nic,model=virtio,vlan=0,macaddr=00:16:3E:00:62:51 -net tap,vlan=0,script=/autobench/var/tmp/ifup-kvm- br0,downscript=/autobench/var/tmp/ifdown-kvm-br0 -net nic,model=virtio,vlan=1,macaddr=00:16:3E:00:62:D1 -net tap,vlan=1,script=/autobench/var/tmp/ifup-kvm- br1,downscript=/autobench/var/tmp/ifdown-kvm-br1 -vnc :1 -monitor telnet::5701,server,nowait -snapshot -daemonize

when using the -netdev option for the tap device:

/usr/local/bin/qemu-system-x86_64 -name cape-vm001 -m 1024 -drive file=/autobench/var/tmp/cape-vm001- raw.img,if=virtio,index=0,media=disk,boot=on -net nic,model=virtio,vlan=0,macaddr=00:16:3E:00:62:51,netdev=cape-vm001-eth0 - netdev tap,id=cape-vm001-eth0,script=/autobench/var/tmp/ifup-kvm- br0,downscript=/autobench/var/tmp/ifdown-kvm-br0 -net nic,model=virtio,vlan=1,macaddr=00:16:3E:00:62:D1,netdev=cape-vm001-eth1 - netdev tap,id=cape-vm001-eth1,script=/autobench/var/tmp/ifup-kvm- br1,downscript=/autobench/var/tmp/ifdown-kvm-br1 -vnc :1 -monitor telnet::5701,server,nowait -snapshot -daemonize

The first ethernet device is a 1GbE device for communicating with the automation infrastructure we have. The second ethernet device is the 10GbE device that the netperf tests run on.

Quoted 7 lines

I can get the networking to work again by bringing down the interfaces and reloading the virtio_net module (modprobe -r virtio_net / modprobe virtio_net).

I haven't had a chance yet to run the tests against a modified version of
qemu that does not set the receive_disabled variable.

I got a chance to run with the setting of the receive_diabled variable commented out and I still run into the problem. It's easier to reproduce when running netperf between two guests on the same machine. I instrumented qemu and virtio a little bit to try and track this down. What I'm seeing is that, with two guests on the same machine, the receiving (netserver) guest eventually gets into a condition where the tap read poll callback is disabled and never re-enabled. So packets are never delivered from tap to qemu and to the guest. On the sending (netperf) side the transmit queue eventually runs out of capacity and it can no longer send packets (I believe this is unique to having the guests on the same machine). And as before, bringing down the interfaces, reloading the virtio_net module, and restarting the interfaces clears things up.

Tom

Quoted 10 lines

Tom

Quoted 7 lines

thanks,
-chris
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 6 lines

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

it seems, that i encounter the same bug.

I've a guest with an high network load, and after some time, it seems that there's "no more network".
Under the guest, I can't ping anymore the gateway.
If i restart the guest, everything work fine again.

Quoted 6 lines · * Jean-Philippe Menil ([email protected]) wrote:

it seems, that i encounter the same bug.

I've a guest with an high network load, and after some time, it seems that there's "no more network". Under the guest, I can't ping anymore the gateway.
If i restart the guest, everything work fine again.

How reproducible is this (and can you also recover the network by simply rmmod/modprobe'ing the virtio_net driver)? Tom posted a patch that you could try if you have an easy way to trigger.

http://marc.info/?l=kvm&m=126564542625725&w=2

thanks,
-chris
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 9 lines · Chris Wright

Quoted 6 lines · * Jean-Philippe Menil ([email protected]) wrote:

it seems, that i encounter the same bug.

I've a guest with an high network load, and after some time, it seems that there's "no more network". Under the guest, I can't ping anymore the gateway.
If i restart the guest, everything work fine again.

Quoted 13 lines

How reproducible is this (and can you also recover the network by simply rmmod/modprobe'ing the virtio_net driver)? Tom posted a patch that you could try if you have an easy way to trigger.

http://marc.info/?l=kvm&m=126564542625725&w=2

thanks,
-chris

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Thanks,

i've just seen the patch.
I will test it.

Many thanks.