Network shutdown under load

DISCUSSION Applied Tom Lendacky 6 participants 8 messages Jan 29, 2010 – Feb 20, 2010 KVM

Share & tools

Tom Lendacky

author

2010-01-29 20:07 UTC #0 lore raw

There's been some discussion of this already in the kvm list, but I want to summarize what I've found and also include the qemu-devel list in an effort to find a solution to this problem.

Running a netperf test between two kvm guests results in the guest's network interface shutting down. I originally found this using kvm guests on two different machines that were connected via a 10GbE link. However, I found this problem can be easily reproduced using two guests on the same machine.

I am running the 2.6.32 level of the kvm.git tree and the 0.12.1.2 level of the qemu-kvm.git tree.

The setup includes two bridges, br0 and br1.

The commands used to start the guests are as follows:
usr/local/bin/qemu-system-x86_64 -name cape-vm001 -m 1024 -drive file=/autobench/var/tmp/cape-vm001- raw.img,if=virtio,index=0,media=disk,boot=on -net nic,model=virtio,vlan=0,macaddr=00:16:3E:00:62:51,netdev=cape-vm001-eth0 - netdev tap,id=cape-vm001-eth0,script=/autobench/var/tmp/ifup-kvm- br0,downscript=/autobench/var/tmp/ifdown-kvm-br0 -net nic,model=virtio,vlan=1,macaddr=00:16:3E:00:62:D1,netdev=cape-vm001-eth1 - netdev tap,id=cape-vm001-eth1,script=/autobench/var/tmp/ifup-kvm- br1,downscript=/autobench/var/tmp/ifdown-kvm-br1 -vnc :1 -monitor telnet::5701,server,nowait -snapshot -daemonize

usr/local/bin/qemu-system-x86_64 -name cape-vm002 -m 1024 -drive file=/autobench/var/tmp/cape-vm002- raw.img,if=virtio,index=0,media=disk,boot=on -net nic,model=virtio,vlan=0,macaddr=00:16:3E:00:62:61,netdev=cape-vm002-eth0 - netdev tap,id=cape-vm002-eth0,script=/autobench/var/tmp/ifup-kvm- br0,downscript=/autobench/var/tmp/ifdown-kvm-br0 -net nic,model=virtio,vlan=1,macaddr=00:16:3E:00:62:E1,netdev=cape-vm002-eth1 - netdev tap,id=cape-vm002-eth1,script=/autobench/var/tmp/ifup-kvm- br1,downscript=/autobench/var/tmp/ifdown-kvm-br1 -vnc :2 -monitor telnet::5702,server,nowait -snapshot -daemonize

The ifup-kvm-br0 script takes the (first) qemu created tap device and brings it up and adds it to bridge br0. The ifup-kvm-br1 script take the (second) qemu created tap device and brings it up and adds it to bridge br1.

Each ethernet device within a guest is on it's own subnet. For example:
guest 1 eth0 has addr 192.168.100.32 and eth1 has addr 192.168.101.32 guest 2 eth0 has addr 192.168.100.64 and eth1 has addr 192.168.101.64

On one of the guests run netserver:
netserver -L 192.168.101.32 -p 12000

On the other guest run netperf:
netperf -L 192.168.101.64 -H 192.168.101.32 -p 12000 -t TCP_STREAM -l 60 -c
-C -- -m 16K -M 16K

It may take more than one netperf run (I find that my second run almost always causes the shutdown) but the network on the eth1 links will stop working.

I did some debugging and found that in qemu on the guest running netserver:
- the receive_disabled variable is set and never gets reset
- the read_poll event handler for the eth1 tap device is disabled and never
re-enabled
These conditions result in no packets being read from the tap device and sent to the guest - effectively shutting down the network. Network connectivity can be restored by shutting down the guest interfaces, unloading the virtio_net module, re-loading the virtio_net module and re-starting the guest interfaces.

I'm continuing to work on debugging this, but would appreciate if some folks with more qemu network experience could try to recreate and debug this.

If my kernel config matters, I can provide that.

Thanks,
Tom

RW

2010-02-02 19:42 UTC #1 lore raw

Hi,

we're currently having this problem on two production servers that 2-4 times a day one interface shuts down. We've four KVMs running on two hosts (2x2). All VMs have eth0 and eth1 running virtio_net. All eth0's are connected to bridge br0 and all eth1's are connected to br1 on the host. Here are the startup options for one VM (the others are quite similar [of course other mac address, ...]):

/usr/bin/kvm -m 8192 -smp 8 -cpu host -daemonize -k de -vnc 127.0.0.1:1 -monitor telnet:172.18.105.46:4444,server,nowait -localtime -pidfile /tmp/kvm-dodoma.pid -drive file=/data/kvm/kvmimages/dodoma.qcow2,if=virtio,cache=none,boot=on -drive file=/data/kvm/kvmimages/dodoma-vdb.qcow2,if=virtio,cache=none -net nic,vlan=104,model=virtio,macaddr=00:ff:48:e5:4b:8d -net tap,vlan=104,ifname=tap.b.dodoma,script=no -net nic,vlan=96,model=virtio,macaddr=00:ff:48:e5:4b:8f -net tap,vlan=96,ifname=tap.f.dodoma,script=no

I've tried the very latest Gentoo kernel 2.6.30 on the host and guest (all VMs and hosts running Gentoo btw.). With kernel 2.6.31 on host and 2.6.30 on guest the problem still exist. I've tried KVM 0.11.1, 0.12.1.2 and 0.12.2 running with kernel 2.6.30 and 2.6.31 on the host side.

Interestingly all the VMs almost have the same network traffic (in and out) but the VMs running Apache bind to eth1 have the biggest problems. They shut down eth1 2-4 times a day. eth0 is running fine despite that it is doing almost the same traffic amount but this traffic comes from the database where as eth1 sends the traffic to the proxy (Varnish). So incoming traffic seems to work fine here but outgoing traffic is problematic. On the other hand the VMs running Varnish getting all the traffic through eth1. Here I've "only" seen one shutdown of eth1 in 48 hours.

Is there anything I can help to debug this problem? Is there already a fix available? Otherwise I really have to install KVM-88 which runs fine on some other hosts.

Thanks!
Robert

Quoted 77 lines · Tom Lendacky

There's been some discussion of this already in the kvm list, but I want to summarize what I've found and also include the qemu-devel list in an effort to find a solution to this problem.
Running a netperf test between two kvm guests results in the guest's network interface shutting down. I originally found this using kvm guests on two different machines that were connected via a 10GbE link. However, I found this problem can be easily reproduced using two guests on the same machine.
I am running the 2.6.32 level of the kvm.git tree and the 0.12.1.2 level of the qemu-kvm.git tree.
The setup includes two bridges, br0 and br1.
The commands used to start the guests are as follows:
usr/local/bin/qemu-system-x86_64 -name cape-vm001 -m 1024 -drive file=/autobench/var/tmp/cape-vm001- raw.img,if=virtio,index=0,media=disk,boot=on -net nic,model=virtio,vlan=0,macaddr=00:16:3E:00:62:51,netdev=cape-vm001-eth0 - netdev tap,id=cape-vm001-eth0,script=/autobench/var/tmp/ifup-kvm- br0,downscript=/autobench/var/tmp/ifdown-kvm-br0 -net nic,model=virtio,vlan=1,macaddr=00:16:3E:00:62:D1,netdev=cape-vm001-eth1 - netdev tap,id=cape-vm001-eth1,script=/autobench/var/tmp/ifup-kvm- br1,downscript=/autobench/var/tmp/ifdown-kvm-br1 -vnc :1 -monitor telnet::5701,server,nowait -snapshot -daemonize
usr/local/bin/qemu-system-x86_64 -name cape-vm002 -m 1024 -drive file=/autobench/var/tmp/cape-vm002- raw.img,if=virtio,index=0,media=disk,boot=on -net nic,model=virtio,vlan=0,macaddr=00:16:3E:00:62:61,netdev=cape-vm002-eth0 - netdev tap,id=cape-vm002-eth0,script=/autobench/var/tmp/ifup-kvm- br0,downscript=/autobench/var/tmp/ifdown-kvm-br0 -net nic,model=virtio,vlan=1,macaddr=00:16:3E:00:62:E1,netdev=cape-vm002-eth1 - netdev tap,id=cape-vm002-eth1,script=/autobench/var/tmp/ifup-kvm- br1,downscript=/autobench/var/tmp/ifdown-kvm-br1 -vnc :2 -monitor telnet::5702,server,nowait -snapshot -daemonize
The ifup-kvm-br0 script takes the (first) qemu created tap device and brings it up and adds it to bridge br0. The ifup-kvm-br1 script take the (second) qemu created tap device and brings it up and adds it to bridge br1.
Each ethernet device within a guest is on it's own subnet. For example:
guest 1 eth0 has addr 192.168.100.32 and eth1 has addr 192.168.101.32 guest 2 eth0 has addr 192.168.100.64 and eth1 has addr 192.168.101.64
On one of the guests run netserver:
netserver -L 192.168.101.32 -p 12000
On the other guest run netperf:
netperf -L 192.168.101.64 -H 192.168.101.32 -p 12000 -t TCP_STREAM -l 60 -c
-C -- -m 16K -M 16K
It may take more than one netperf run (I find that my second run almost always causes the shutdown) but the network on the eth1 links will stop working.
I did some debugging and found that in qemu on the guest running netserver:
- the receive_disabled variable is set and never gets reset
- the read_poll event handler for the eth1 tap device is disabled and never
re-enabled
These conditions result in no packets being read from the tap device and sent to the guest - effectively shutting down the network. Network connectivity can be restored by shutting down the guest interfaces, unloading the virtio_net module, re-loading the virtio_net module and re-starting the guest interfaces.
I'm continuing to work on debugging this, but would appreciate if some folks with more qemu network experience could try to recreate and debug this.
If my kernel config matters, I can provide that.
Thanks,
Tom
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Tom Lendacky

author

2010-02-08 16:10 UTC #2 lore raw

In reply to Tom Lendacky

Fix a race condition where qemu finds that there are not enough virtio ring buffers available and the guest make more buffers available before qemu can enable notifications.

Signed-off-by Tom Lendacky <[email protected]>

Signed-off-by Anthony Liguori <[email protected]>

 hw/virtio-net.c |   10 +++++++++-
 1 files changed, 9 insertions(+), 1 deletions(-)

hw/virtio-net.c +9 -1

index 6e48997..5c0093e 100644
@@ -379,7 +379,15 @@ static int virtio_net_has_buffers(VirtIONet *n, int bufsize)
         (n->mergeable_rx_bufs &&
          !virtqueue_avail_bytes(n->rx_vq, bufsize, 0))) {
         virtio_queue_set_notification(n->rx_vq, 1);
-        return 0;
+
+        /* To avoid a race condition where the guest has made some buffers
+         * available after the above check but before notification was
+         * enabled, check for available buffers again.
+         */
+        if (virtio_queue_empty(n->rx_vq) ||
+            (n->mergeable_rx_bufs &&
+             !virtqueue_avail_bytes(n->rx_vq, bufsize, 0)))
+            return 0;
     }
 
     virtio_queue_set_notification(n->rx_vq, 0);

Quoted 81 lines · Tom Lendacky

There's been some discussion of this already in the kvm list, but I want to summarize what I've found and also include the qemu-devel list in an effort
to find a solution to this problem.
Running a netperf test between two kvm guests results in the guest's
network interface shutting down. I originally found this using kvm guests on two different machines that were connected via a 10GbE link. However, I found this problem can be easily reproduced using two guests on the same machine.
I am running the 2.6.32 level of the kvm.git tree and the 0.12.1.2 level of the qemu-kvm.git tree.
The setup includes two bridges, br0 and br1.
The commands used to start the guests are as follows:
usr/local/bin/qemu-system-x86_64 -name cape-vm001 -m 1024 -drive file=/autobench/var/tmp/cape-vm001- raw.img,if=virtio,index=0,media=disk,boot=on -net nic,model=virtio,vlan=0,macaddr=00:16:3E:00:62:51,netdev=cape-vm001-eth0 - netdev tap,id=cape-vm001-eth0,script=/autobench/var/tmp/ifup-kvm- br0,downscript=/autobench/var/tmp/ifdown-kvm-br0 -net nic,model=virtio,vlan=1,macaddr=00:16:3E:00:62:D1,netdev=cape-vm001-eth1 - netdev tap,id=cape-vm001-eth1,script=/autobench/var/tmp/ifup-kvm- br1,downscript=/autobench/var/tmp/ifdown-kvm-br1 -vnc :1 -monitor telnet::5701,server,nowait -snapshot -daemonize
usr/local/bin/qemu-system-x86_64 -name cape-vm002 -m 1024 -drive file=/autobench/var/tmp/cape-vm002- raw.img,if=virtio,index=0,media=disk,boot=on -net nic,model=virtio,vlan=0,macaddr=00:16:3E:00:62:61,netdev=cape-vm002-eth0 - netdev tap,id=cape-vm002-eth0,script=/autobench/var/tmp/ifup-kvm- br0,downscript=/autobench/var/tmp/ifdown-kvm-br0 -net nic,model=virtio,vlan=1,macaddr=00:16:3E:00:62:E1,netdev=cape-vm002-eth1 - netdev tap,id=cape-vm002-eth1,script=/autobench/var/tmp/ifup-kvm- br1,downscript=/autobench/var/tmp/ifdown-kvm-br1 -vnc :2 -monitor telnet::5702,server,nowait -snapshot -daemonize
The ifup-kvm-br0 script takes the (first) qemu created tap device and
brings it up and adds it to bridge br0. The ifup-kvm-br1 script take the (second) qemu created tap device and brings it up and adds it to bridge br1.
Each ethernet device within a guest is on it's own subnet. For example:
guest 1 eth0 has addr 192.168.100.32 and eth1 has addr 192.168.101.32 guest 2 eth0 has addr 192.168.100.64 and eth1 has addr 192.168.101.64
On one of the guests run netserver:
netserver -L 192.168.101.32 -p 12000
On the other guest run netperf:
netperf -L 192.168.101.64 -H 192.168.101.32 -p 12000 -t TCP_STREAM -l 60
-c -C -- -m 16K -M 16K
It may take more than one netperf run (I find that my second run almost
always causes the shutdown) but the network on the eth1 links will stop working.
I did some debugging and found that in qemu on the guest running netserver:
- the receive_disabled variable is set and never gets reset
- the read_poll event handler for the eth1 tap device is disabled and never re-enabled
These conditions result in no packets being read from the tap device and
sent to the guest - effectively shutting down the network. Network connectivity can be restored by shutting down the guest interfaces, unloading the virtio_net module, re-loading the virtio_net module and re-starting the guest interfaces.
I'm continuing to work on debugging this, but would appreciate if some
folks with more qemu network experience could try to recreate and debug this.
If my kernel config matters, I can provide that.
Thanks,
Tom
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Anthony Liguori

2010-02-08 20:58 UTC #3 lore raw

Quoted 7 lines · Tom Lendacky

Fix a race condition where qemu finds that there are not enough virtio ring buffers available and the guest make more buffers available before qemu can enable notifications.
Signed-off-by Tom Lendacky<[email protected]>
Signed-off-by Anthony Liguori<[email protected]>

I've walked through the changes in this series and I'm pretty certain that this is the only problem. I'd appreciate if others could review though.

Regards,

Anthony Liguori

Quoted 109 lines

  hw/virtio-net.c |   10 +++++++++-
  1 files changed, 9 insertions(+), 1 deletions(-)
hw/virtio-net.c +9 -1
index 6e48997..5c0093e 100644
@@ -379,7 +379,15 @@ static int virtio_net_has_buffers(VirtIONet *n, int bufsize)
          (n->mergeable_rx_bufs&&
           !virtqueue_avail_bytes(n->rx_vq, bufsize, 0))) {
          virtio_queue_set_notification(n->rx_vq, 1);
-        return 0;
+
+        /* To avoid a race condition where the guest has made some buffers
+         * available after the above check but before notification was
+         * enabled, check for available buffers again.
+         */
+        if (virtio_queue_empty(n->rx_vq) ||
+            (n->mergeable_rx_bufs&&
+             !virtqueue_avail_bytes(n->rx_vq, bufsize, 0)))
+            return 0;
      }
 
      virtio_queue_set_notification(n->rx_vq, 0);
Quoted 82 lines · Tom Lendacky
There's been some discussion of this already in the kvm list, but I want to summarize what I've found and also include the qemu-devel list in an effort
to find a solution to this problem.
Running a netperf test between two kvm guests results in the guest's
network interface shutting down. I originally found this using kvm guests on two different machines that were connected via a 10GbE link. However, I found this problem can be easily reproduced using two guests on the same machine.
I am running the 2.6.32 level of the kvm.git tree and the 0.12.1.2 level of the qemu-kvm.git tree.
The setup includes two bridges, br0 and br1.
The commands used to start the guests are as follows:
usr/local/bin/qemu-system-x86_64 -name cape-vm001 -m 1024 -drive file=/autobench/var/tmp/cape-vm001- raw.img,if=virtio,index=0,media=disk,boot=on -net nic,model=virtio,vlan=0,macaddr=00:16:3E:00:62:51,netdev=cape-vm001-eth0 - netdev tap,id=cape-vm001-eth0,script=/autobench/var/tmp/ifup-kvm- br0,downscript=/autobench/var/tmp/ifdown-kvm-br0 -net nic,model=virtio,vlan=1,macaddr=00:16:3E:00:62:D1,netdev=cape-vm001-eth1 - netdev tap,id=cape-vm001-eth1,script=/autobench/var/tmp/ifup-kvm- br1,downscript=/autobench/var/tmp/ifdown-kvm-br1 -vnc :1 -monitor telnet::5701,server,nowait -snapshot -daemonize
usr/local/bin/qemu-system-x86_64 -name cape-vm002 -m 1024 -drive file=/autobench/var/tmp/cape-vm002- raw.img,if=virtio,index=0,media=disk,boot=on -net nic,model=virtio,vlan=0,macaddr=00:16:3E:00:62:61,netdev=cape-vm002-eth0 - netdev tap,id=cape-vm002-eth0,script=/autobench/var/tmp/ifup-kvm- br0,downscript=/autobench/var/tmp/ifdown-kvm-br0 -net nic,model=virtio,vlan=1,macaddr=00:16:3E:00:62:E1,netdev=cape-vm002-eth1 - netdev tap,id=cape-vm002-eth1,script=/autobench/var/tmp/ifup-kvm- br1,downscript=/autobench/var/tmp/ifdown-kvm-br1 -vnc :2 -monitor telnet::5702,server,nowait -snapshot -daemonize
The ifup-kvm-br0 script takes the (first) qemu created tap device and
brings it up and adds it to bridge br0. The ifup-kvm-br1 script take the (second) qemu created tap device and brings it up and adds it to bridge br1.
Each ethernet device within a guest is on it's own subnet. For example:
   guest 1 eth0 has addr 192.168.100.32 and eth1 has addr 192.168.101.32
   guest 2 eth0 has addr 192.168.100.64 and eth1 has addr 192.168.101.64
On one of the guests run netserver:
   netserver -L 192.168.101.32 -p 12000
On the other guest run netperf:
   netperf -L 192.168.101.64 -H 192.168.101.32 -p 12000 -t TCP_STREAM -l 60
-c -C -- -m 16K -M 16K
It may take more than one netperf run (I find that my second run almost
always causes the shutdown) but the network on the eth1 links will stop working.
I did some debugging and found that in qemu on the guest running netserver:
- the receive_disabled variable is set and never gets reset
- the read_poll event handler for the eth1 tap device is disabled and never re-enabled
These conditions result in no packets being read from the tap device and
sent to the guest - effectively shutting down the network. Network connectivity can be restored by shutting down the guest interfaces, unloading the virtio_net module, re-loading the virtio_net module and re-starting the guest interfaces.
I'm continuing to work on debugging this, but would appreciate if some
folks with more qemu network experience could try to recreate and debug this.
If my kernel config matters, I can provide that.
Thanks,
Tom
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Herbert Xu

2010-02-08 21:18 UTC #4 lore raw

In reply to Tom Lendacky

Quoted 7 lines · Tom Lendacky

Fix a race condition where qemu finds that there are not enough virtio ring buffers available and the guest make more buffers available before qemu can enable notifications.
Signed-off-by Tom Lendacky <[email protected]>
Signed-off-by Anthony Liguori <[email protected]>

Good cath!

Quoted 5 lines

hw/virtio-net.c +0 -0
index 6e48997..5c0093e 100644

hw/virtio-net.c static int virtio_net_has_buffers(VirtIONet *n, int bufsize) · line 386

Quoted 9 lines

hw/virtio-net.c +5 -1

@@ -379,7 +379,15 @@ static int virtio_net_has_buffers(VirtIONet *n, int bufsize)
         (n->mergeable_rx_bufs &&
          !virtqueue_avail_bytes(n->rx_vq, bufsize, 0))) {
         virtio_queue_set_notification(n->rx_vq, 1);
-        return 0;
+
+        /* To avoid a race condition where the guest has made some buffers
+         * available after the above check but before notification was
+         * enabled, check for available buffers again.
+         */

We should also add a full memory barrier right here to avoid out-of-order loads.

hw/virtio-net.c static int virtio_net_has_buffers(VirtIONet *n, int bufsize) · line 391

Quoted 5 lines

hw/virtio-net.c +4 -0

+        if (virtio_queue_empty(n->rx_vq) ||
+            (n->mergeable_rx_bufs &&
+             !virtqueue_avail_bytes(n->rx_vq, bufsize, 0)))
+            return 0;
     }

Cheers,

Signature · 4 lines

Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

RW

2010-02-09 20:29 UTC #5 lore raw

In reply to Tom Lendacky

Thanks for the patch! It seems to solve the problem that under load (> 50 MBit/s) the network goes down. I've applied the patch to KVM 0.12.2 running Gentoo. Host and guest is running kernel 2.6.32 currently (kernel 2.6.30 in guest and 2.6.32 in host works also for us).

Another host doing the same jobs with the same amount of traffic and configuration but with KVM 0.11.1 was shutting down the network interface every 5-10 minutes today while the patched 0.12.2 was running fine. During the time the 0.11.1 KVMs were down the patched one delivered >200 MBit/s without problems. Now both hosts running with the patched version. We're expecting much more traffic tomorrow so if the network is still up on thursday I would say the bug is fixed.

Thanks for that patch! It really was a lifesaver today :-)

- Robert

Quoted 121 lines · Tom Lendacky

Fix a race condition where qemu finds that there are not enough virtio ring buffers available and the guest make more buffers available before qemu can enable notifications.
Signed-off-by Tom Lendacky <[email protected]>
Signed-off-by Anthony Liguori <[email protected]>
 hw/virtio-net.c |   10 +++++++++-
 1 files changed, 9 insertions(+), 1 deletions(-)
hw/virtio-net.c +9 -1
index 6e48997..5c0093e 100644
@@ -379,7 +379,15 @@ static int virtio_net_has_buffers(VirtIONet *n, int bufsize)
         (n->mergeable_rx_bufs &&
          !virtqueue_avail_bytes(n->rx_vq, bufsize, 0))) {
         virtio_queue_set_notification(n->rx_vq, 1);
-        return 0;
+
+        /* To avoid a race condition where the guest has made some buffers
+         * available after the above check but before notification was
+         * enabled, check for available buffers again.
+         */
+        if (virtio_queue_empty(n->rx_vq) ||
+            (n->mergeable_rx_bufs &&
+             !virtqueue_avail_bytes(n->rx_vq, bufsize, 0)))
+            return 0;
     }
 
     virtio_queue_set_notification(n->rx_vq, 0);
Quoted 72 lines · Tom Lendacky
There's been some discussion of this already in the kvm list, but I want to summarize what I've found and also include the qemu-devel list in an effort
to find a solution to this problem.
Running a netperf test between two kvm guests results in the guest's
network interface shutting down. I originally found this using kvm guests on two different machines that were connected via a 10GbE link. However, I found this problem can be easily reproduced using two guests on the same machine.
I am running the 2.6.32 level of the kvm.git tree and the 0.12.1.2 level of the qemu-kvm.git tree.
The setup includes two bridges, br0 and br1.
The commands used to start the guests are as follows:
usr/local/bin/qemu-system-x86_64 -name cape-vm001 -m 1024 -drive file=/autobench/var/tmp/cape-vm001- raw.img,if=virtio,index=0,media=disk,boot=on -net nic,model=virtio,vlan=0,macaddr=00:16:3E:00:62:51,netdev=cape-vm001-eth0 - netdev tap,id=cape-vm001-eth0,script=/autobench/var/tmp/ifup-kvm- br0,downscript=/autobench/var/tmp/ifdown-kvm-br0 -net nic,model=virtio,vlan=1,macaddr=00:16:3E:00:62:D1,netdev=cape-vm001-eth1 - netdev tap,id=cape-vm001-eth1,script=/autobench/var/tmp/ifup-kvm- br1,downscript=/autobench/var/tmp/ifdown-kvm-br1 -vnc :1 -monitor telnet::5701,server,nowait -snapshot -daemonize
usr/local/bin/qemu-system-x86_64 -name cape-vm002 -m 1024 -drive file=/autobench/var/tmp/cape-vm002- raw.img,if=virtio,index=0,media=disk,boot=on -net nic,model=virtio,vlan=0,macaddr=00:16:3E:00:62:61,netdev=cape-vm002-eth0 - netdev tap,id=cape-vm002-eth0,script=/autobench/var/tmp/ifup-kvm- br0,downscript=/autobench/var/tmp/ifdown-kvm-br0 -net nic,model=virtio,vlan=1,macaddr=00:16:3E:00:62:E1,netdev=cape-vm002-eth1 - netdev tap,id=cape-vm002-eth1,script=/autobench/var/tmp/ifup-kvm- br1,downscript=/autobench/var/tmp/ifdown-kvm-br1 -vnc :2 -monitor telnet::5702,server,nowait -snapshot -daemonize
The ifup-kvm-br0 script takes the (first) qemu created tap device and
brings it up and adds it to bridge br0. The ifup-kvm-br1 script take the (second) qemu created tap device and brings it up and adds it to bridge br1.
Each ethernet device within a guest is on it's own subnet. For example:
guest 1 eth0 has addr 192.168.100.32 and eth1 has addr 192.168.101.32 guest 2 eth0 has addr 192.168.100.64 and eth1 has addr 192.168.101.64
On one of the guests run netserver:
netserver -L 192.168.101.32 -p 12000
On the other guest run netperf:
netperf -L 192.168.101.64 -H 192.168.101.32 -p 12000 -t TCP_STREAM -l 60
-c -C -- -m 16K -M 16K
It may take more than one netperf run (I find that my second run almost
always causes the shutdown) but the network on the eth1 links will stop working.
I did some debugging and found that in qemu on the guest running netserver:
- the receive_disabled variable is set and never gets reset
- the read_poll event handler for the eth1 tap device is disabled and never re-enabled
These conditions result in no packets being read from the tap device and
sent to the guest - effectively shutting down the network. Network connectivity can be restored by shutting down the guest interfaces, unloading the virtio_net module, re-loading the virtio_net module and re-starting the guest interfaces.
I'm continuing to work on debugging this, but would appreciate if some
folks with more qemu network experience could try to recreate and debug this.
Quoted 10 lines
If my kernel config matters, I can provide that.
Thanks,
Tom
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Anthony Liguori

Applied

2010-02-10 19:31 UTC #6 lore raw

In reply to Tom Lendacky

Quoted 7 lines · Tom Lendacky

Fix a race condition where qemu finds that there are not enough virtio ring buffers available and the guest make more buffers available before qemu can enable notifications.
Signed-off-by Tom Lendacky<[email protected]>
Signed-off-by Anthony Liguori<[email protected]>

Applied. Thanks. We should audit the code for proper barrier support. Right now, I think there's a lot of places that we're missing them.

Regards,

Anthony Liguori

Quoted 115 lines

  hw/virtio-net.c |   10 +++++++++-
  1 files changed, 9 insertions(+), 1 deletions(-)
hw/virtio-net.c +9 -1
index 6e48997..5c0093e 100644
@@ -379,7 +379,15 @@ static int virtio_net_has_buffers(VirtIONet *n, int bufsize)
          (n->mergeable_rx_bufs&&
           !virtqueue_avail_bytes(n->rx_vq, bufsize, 0))) {
          virtio_queue_set_notification(n->rx_vq, 1);
-        return 0;
+
+        /* To avoid a race condition where the guest has made some buffers
+         * available after the above check but before notification was
+         * enabled, check for available buffers again.
+         */
+        if (virtio_queue_empty(n->rx_vq) ||
+            (n->mergeable_rx_bufs&&
+             !virtqueue_avail_bytes(n->rx_vq, bufsize, 0)))
+            return 0;
      }
 
      virtio_queue_set_notification(n->rx_vq, 0);
Quoted 72 lines · Tom Lendacky
There's been some discussion of this already in the kvm list, but I want to summarize what I've found and also include the qemu-devel list in an effort
to find a solution to this problem.
Running a netperf test between two kvm guests results in the guest's
network interface shutting down. I originally found this using kvm guests on two different machines that were connected via a 10GbE link. However, I found this problem can be easily reproduced using two guests on the same machine.
I am running the 2.6.32 level of the kvm.git tree and the 0.12.1.2 level of the qemu-kvm.git tree.
The setup includes two bridges, br0 and br1.
The commands used to start the guests are as follows:
usr/local/bin/qemu-system-x86_64 -name cape-vm001 -m 1024 -drive file=/autobench/var/tmp/cape-vm001- raw.img,if=virtio,index=0,media=disk,boot=on -net nic,model=virtio,vlan=0,macaddr=00:16:3E:00:62:51,netdev=cape-vm001-eth0 - netdev tap,id=cape-vm001-eth0,script=/autobench/var/tmp/ifup-kvm- br0,downscript=/autobench/var/tmp/ifdown-kvm-br0 -net nic,model=virtio,vlan=1,macaddr=00:16:3E:00:62:D1,netdev=cape-vm001-eth1 - netdev tap,id=cape-vm001-eth1,script=/autobench/var/tmp/ifup-kvm- br1,downscript=/autobench/var/tmp/ifdown-kvm-br1 -vnc :1 -monitor telnet::5701,server,nowait -snapshot -daemonize
usr/local/bin/qemu-system-x86_64 -name cape-vm002 -m 1024 -drive file=/autobench/var/tmp/cape-vm002- raw.img,if=virtio,index=0,media=disk,boot=on -net nic,model=virtio,vlan=0,macaddr=00:16:3E:00:62:61,netdev=cape-vm002-eth0 - netdev tap,id=cape-vm002-eth0,script=/autobench/var/tmp/ifup-kvm- br0,downscript=/autobench/var/tmp/ifdown-kvm-br0 -net nic,model=virtio,vlan=1,macaddr=00:16:3E:00:62:E1,netdev=cape-vm002-eth1 - netdev tap,id=cape-vm002-eth1,script=/autobench/var/tmp/ifup-kvm- br1,downscript=/autobench/var/tmp/ifdown-kvm-br1 -vnc :2 -monitor telnet::5702,server,nowait -snapshot -daemonize
The ifup-kvm-br0 script takes the (first) qemu created tap device and
brings it up and adds it to bridge br0. The ifup-kvm-br1 script take the (second) qemu created tap device and brings it up and adds it to bridge br1.
Each ethernet device within a guest is on it's own subnet. For example:
   guest 1 eth0 has addr 192.168.100.32 and eth1 has addr 192.168.101.32
   guest 2 eth0 has addr 192.168.100.64 and eth1 has addr 192.168.101.64
On one of the guests run netserver:
   netserver -L 192.168.101.32 -p 12000
On the other guest run netperf:
   netperf -L 192.168.101.64 -H 192.168.101.32 -p 12000 -t TCP_STREAM -l 60
-c -C -- -m 16K -M 16K
It may take more than one netperf run (I find that my second run almost
always causes the shutdown) but the network on the eth1 links will stop working.
I did some debugging and found that in qemu on the guest running netserver:
- the receive_disabled variable is set and never gets reset
- the read_poll event handler for the eth1 tap device is disabled and never re-enabled
These conditions result in no packets being read from the tap device and
sent to the guest - effectively shutting down the network. Network connectivity can be restored by shutting down the guest interfaces, unloading the virtio_net module, re-loading the virtio_net module and re-starting the guest interfaces.
I'm continuing to work on debugging this, but would appreciate if some
folks with more qemu network experience could try to recreate and debug this.
Quoted 10 lines
If my kernel config matters, I can provide that.
Thanks,
Tom
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Lothar Behrens

2010-02-20 06:57 UTC #7 lore raw

In reply to Tom Lendacky

Hi,

as I have read that the bug has been fixed, when would it available as a new release for the public or at least as a RPM for openSuSE 11.1?

Or is it better to build one myself?

What I also have encountered is that some java code would bring the qemu process into a race condition one could probably solve by a sigstop and then a sigcont.

Is there any relation to this network shutdown bug?

Thanks

Lothar

-- | Rapid Prototyping | XSLT Codegeneration | http://www.lollisoft.de
Lothar Behrens
Heinrich-Scheufelen-Platz 2
73252 Lenningen