Network poblems using virtio and tap.
Hey.
I'm observing some network problems when using virtio and a tap if. The trouble is a drop of network troughput from abot 90Mbit/s to about 90kbit/s, after about 1.7 TB of transferred data.
Setup
Host:
- Fedora 8, nearly all updates
- Intel(R) Xeon(R) CPU X3210 @ 2.13GHz
- 8GB RAM
- 2 Guests
Guest:
- Fedora 9 all updates
- virtio nic using tap
- samba sharing a mountpoint using an iscsi target
Steps to reproduce:
1. Setup a guets using virtio and tap
2. Mount an iscsi target and wokr on the fs or create some traffic on the nic. 2.a I used:
# while true ; do sync ; \
time dd if=/dev/zero of=test.bin bs=1M count=1024 ; sync ;
time md5sum test.bin ; done
3. Wait a couple of days or about 1.7 TB of data on a 100Mbit/s.Expected results:
Everything runs fine, even after 2.5 TB of transferred data or more.
Actual results:
Everything runs fine up to some point.
Then: Troughput drops a bit (from 11 MB/s to about 6MB/s). At some point later the troughput drops to about 90kBit/s. No messages on guest and host. No errors on ifconfig, no dmesg errors. Ping latency increases over time. Logged here:
http://www.meinsandkasten.de/log-virtio.pdf
Did someone observ something similar or so?
Greetings
fabian
Signature · 1 line
Fabian Deutsch <[email protected]>
Hello.
It seems to be a problem of virtio. Using e1000 again, I don't get into trouble anymore.
- fabian
Am Montag, den 01.09.2008, 14:42 +0200 schrieb Fabian Deutsch:
Quoted 43 lines
Hey.
I'm observing some network problems when using virtio and a tap if. The trouble is a drop of network troughput from abot 90Mbit/s to about 90kbit/s, after about 1.7 TB of transferred data.
Setup
Host:
- Fedora 8, nearly all updates
- Intel(R) Xeon(R) CPU X3210 @ 2.13GHz
- 8GB RAM
- 2 GuestsGuest:
- Fedora 9 all updates
- virtio nic using tap
- samba sharing a mountpoint using an iscsi targetSteps to reproduce:
1. Setup a guets using virtio and tap
2. Mount an iscsi target and wokr on the fs or create some traffic on the nic. 2.a I used:# while true ; do sync ; \ time dd if=/dev/zero of=test.bin bs=1M count=1024 ; sync ; time md5sum test.bin ; done 3. Wait a couple of days or about 1.7 TB of data on a 100Mbit/s.Expected results:
Everything runs fine, even after 2.5 TB of transferred data or more.Actual results:
Everything runs fine up to some point.
Then: Troughput drops a bit (from 11 MB/s to about 6MB/s). At some point later the troughput drops to about 90kBit/s. No messages on guest and host. No errors on ifconfig, no dmesg errors. Ping latency increases over time. Logged here:
http://www.meinsandkasten.de/log-virtio.pdfDid someone observ something similar or so?
Greetings
fabian
Am Montag, 1. September 2008 schrieb Fabian Deutsch:
Quoted 4 lines
Hey.
I'm observing some network problems when using virtio and a tap if. The trouble is a drop of network troughput from abot 90Mbit/s to about
90kbit/s,
Quoted 9 lines
after about 1.7 TB of transferred data.
Setup
Host:
- Fedora 8, nearly all updates
- Intel(R) Xeon(R) CPU X3210 @ 2.13GHz
- 8GB RAM
- 2 Guests
Quoted 4 lines
Guest:
- Fedora 9 all updates
- virtio nic using tap
- samba sharing a mountpoint using an iscsi target
Can you tell us what host kernel and what guest kernel you are using? The were some changes to tun/tap for virtio lately which could make a difference.
Christian
Am Samstag, den 06.09.2008, 09:40 +0200 schrieb Christian Borntraeger:
Quoted 5 lines
Am Montag, 1. September 2008 schrieb Fabian Deutsch:
Quoted 4 lines
Hey.
I'm observing some network problems when using virtio and a tap if. The trouble is a drop of network troughput from abot 90Mbit/s to
about
Quoted 10 lines
90kbit/s,
Quoted 9 lines
after about 1.7 TB of transferred data.
Setup
Host:
- Fedora 8, nearly all updates
- Intel(R) Xeon(R) CPU X3210 @ 2.13GHz
- 8GB RAM
- 2 Guests
Quoted 6 lines
Quoted 4 lines
Guest:
- Fedora 9 all updates
- virtio nic using tap
- samba sharing a mountpoint using an iscsi targetCan you tell us what host kernel and what guest kernel you are using?
The were
Quoted 1 line
some changes to tun/tap for virtio lately which could make a
difference.
Host:
Fedora 8 x86_64
2.6.25.14-69.fc8 SMP
kvm-72
Guest:
Fedora 9 i386
2.6.25.11-97.fc9.i686
Greetings
- fabian
Quoted 29 lines · On Sat, 06 Sep 2008 14:04:30 +0200 Fabian Deutsch <[email protected]> wrote:
Am Samstag, den 06.09.2008, 09:40 +0200 schrieb Christian Borntraeger:
Quoted 6 lines
Am Montag, 1. September 2008 schrieb Fabian Deutsch:
Quoted 5 lines
Hey.
I'm observing some network problems when using virtio and a tap if. The trouble is a drop of network troughput from abot 90Mbit/s to
about
Quoted 10 lines
90kbit/s,
Quoted 9 lines
after about 1.7 TB of transferred data.
Setup
Host:
- Fedora 8, nearly all updates
- Intel(R) Xeon(R) CPU X3210 @ 2.13GHz
- 8GB RAM
- 2 GuestsQuoted 7 lines
Quoted 4 lines
Guest:
- Fedora 9 all updates
- virtio nic using tap
- samba sharing a mountpoint using an iscsi targetCan you tell us what host kernel and what guest kernel you are using?
The were
Quoted 1 line
some changes to tun/tap for virtio lately which could make a
difference.
Quoted 8 lines
Host:
Fedora 8x86_64
2.6.25.14-69.fc8 SMP
kvm-72Guest:
Fedora 9 i386
2.6.25.11-97.fc9.i686
I noticed the same behaviour using a rtl8139 and a tap device. After some MB's (!) I had to invoke a in the guest:
ifdown eth0; ifup eth0
to get the network back again. I replaced the rtl-8139 by a virtio device and the problem seems to have gone, but I haven't tried yet with a large amount of data.
kvm-73 (kvm74 gives compile errors)
host: 64bit ubuntu-8.04-LTS with vanilla kernel 2.6.26.3 SMP
guest: ubuntu-8.04-LTS with ubuntu kernel 32-bit
On the same host runs a 32bit w2k3 (virtio NIC) and I haven't noticed such problems, even when this host ran with an rtl-8139 device.
Sorry if I missed some postings concerning the issue, I just (re)subscribed to the list.
R.
Signature · 9 lines
___________________________________________________________________ It is better to remain silent and be thought a fool, than to speak aloud and remove all doubt. +------------------------------------------------------------------+ | Richard Lucassen, Utrecht | | Public key and email address: | | http://www.lucassen.org/mail-pubkey.html | +------------------------------------------------------------------+
Am Samstag, den 06.09.2008, 14:26 +0200 schrieb richard lucassen:
Quoted 37 lines
Quoted 34 lines · On Sat, 06 Sep 2008 14:04:30 +0200 Fabian Deutsch <[email protected]> wrote:
Am Samstag, den 06.09.2008, 09:40 +0200 schrieb Christian Borntraeger:
Quoted 6 lines
Am Montag, 1. September 2008 schrieb Fabian Deutsch:
Quoted 5 lines
Hey.
I'm observing some network problems when using virtio and a tap if. The trouble is a drop of network troughput from abot 90Mbit/s to
about
Quoted 10 lines
90kbit/s,
Quoted 9 lines
after about 1.7 TB of transferred data.
Setup
Host:
- Fedora 8, nearly all updates
- Intel(R) Xeon(R) CPU X3210 @ 2.13GHz
- 8GB RAM
- 2 GuestsQuoted 7 lines
Quoted 4 lines
Guest:
- Fedora 9 all updates
- virtio nic using tap
- samba sharing a mountpoint using an iscsi targetCan you tell us what host kernel and what guest kernel you are using?
The were
Quoted 1 line
some changes to tun/tap for virtio lately which could make a
difference.
Host:
Fedora 8x86_64
2.6.25.14-69.fc8 SMP
kvm-72
Quoted 8 lines
Quoted 3 lines
Guest:
Fedora 9 i386
2.6.25.11-97.fc9.i686I noticed the same behaviour using a rtl8139 and a tap device. After some MB's (!) I had to invoke a in the guest:
ifdown eth0; ifup eth0
ifdown ; ifup doesn't help in this case.
Even a guest-restart "init 6" doesn't help.
Just shutting it completely down "init 0" and powering it on again helps.
Quoted 4 lines
to get the network back again. I replaced the rtl-8139 by a virtio device and the problem seems to have gone, but I haven't tried yet with a large amount of data.
Virtio net works fast and withouth trouble - up to some point, then packet loss occurs.
Quoted 13 lines
kvm-73 (kvm74 gives compile errors)
host: 64bit ubuntu-8.04-LTS with vanilla kernel 2.6.26.3 SMP
guest: ubuntu-8.04-LTS with ubuntu kernel 32-bitOn the same host runs a 32bit w2k3 (virtio NIC) and I haven't noticed such problems, even when this host ran with an rtl-8139 device.
Sorry if I missed some postings concerning the issue, I just (re)subscribed to the list.
R.
Quoted 9 lines · On Sat, 06 Sep 2008 15:47:23 +0200 Fabian Deutsch <[email protected]> wrote:
Quoted 4 lines
I noticed the same behaviour using a rtl8139 and a tap device. After some MB's (!) I had to invoke a in the guest:
ifdown eth0; ifup eth0
ifdown ; ifup doesn't help in this case.
Even a guest-restart "init 6" doesn't help.
Just shutting it completely down "init 0" and powering it on again helps.
No, for me no need for shutdown.
Quoted 6 lines
Quoted 3 lines
to get the network back again. I replaced the rtl-8139 by a virtio device and the problem seems to have gone, but I haven't tried yet with a large amount of data.
Virtio net works fast and withouth trouble - up to some point, then packet loss occurs.
I'll try to transfer /dev/urandom through ssh to another host (vdisk too small)
But not now, I'm too busy at the moment :-(
Signature · 9 lines
___________________________________________________________________ It is better to remain silent and be thought a fool, than to speak aloud and remove all doubt. +------------------------------------------------------------------+ | Richard Lucassen, Utrecht | | Public key and email address: | | http://www.lucassen.org/mail-pubkey.html | +------------------------------------------------------------------+
Quoted 2 lines · On Sat, 06 Sep 2008 15:47:23 +0200 Fabian Deutsch <[email protected]> wrote:
Virtio net works fast and withouth trouble - up to some point, then packet loss occurs.
FYI, after some time I got this:
Received disconnect from 10.1.255.2: 2: Bad packet length 1367736372.
I was still able to relogon using ssh.
Signature · 9 lines
___________________________________________________________________ It is better to remain silent and be thought a fool, than to speak aloud and remove all doubt. +------------------------------------------------------------------+ | Richard Lucassen, Utrecht | | Public key and email address: | | http://www.lucassen.org/mail-pubkey.html | +------------------------------------------------------------------+