massive performance drop after a while when using virtio nics
Hey.
When running a guest in the following setup, I observ a "massive" performance decrease after a couple of gigabytes of transfrerred data.
other guests keep running fine (no regressions) and nothing unusual is observed on the host.
fedora 8 host, kvm-72 (user and module)
fedora 9 guest, all updates.
nic 0, vtio:
subnet a
samba listening
nic 1, vtio:
subnet b
iscsi initiator running
samba shares a mount point, pointing to an iscsi target.
now pulling/pushing about 80GB over samba results in a drop from about 60/70Mbit to about 1/2Mbit of troughput.
The problem is reproducible.
The problem is solved after a reboot of the guest. Just restarting networking doesn't help.
Guest's dmesg doesn't say anything about problems. No errors in ifconfig.
Has someone observed something similar?
Greetings
- fabian
Quoted 34 lines · Fabian Deutsch
Hey.
When running a guest in the following setup, I observ a "massive" performance decrease after a couple of gigabytes of transfrerred data.
other guests keep running fine (no regressions) and nothing unusual is observed on the host.
fedora 8 host, kvm-72 (user and module)
fedora 9 guest, all updates.
nic 0, vtio:
subnet a
samba listeningnic 1, vtio:
subnet b
iscsi initiator runningsamba shares a mount point, pointing to an iscsi target.
now pulling/pushing about 80GB over samba results in a drop from about 60/70Mbit to about 1/2Mbit of troughput.
The problem is reproducible.
The problem is solved after a reboot of the guest. Just restarting networking doesn't help.
Guest's dmesg doesn't say anything about problems. No errors in ifconfig.Has someone observed something similar?
Quoted 9 lines
Greetings
- fabian--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
We just saw the very same issue.
guest & host : 2.6.25.3, debian etch 64 bits
KVM 70 on this machine.
As fabian said, rebooting the kernel doesn't help, but rebooting the whole kvm guest does.
No error messages as far as I can tell. The kvm guest is a clonezilla server and send lots of data. It began to crawl after 2/3 weeks of usage.
Greetings,
Signature · 2 lines
Yann Dupont - Service IRTS, DSI Université de Nantes Tel : 02.51.12.53.91 - Mail/Jabber : [email protected]
Am Montag, den 08.09.2008, 14:07 +0200 schrieb Yann Dupont:
Quoted 47 lines
Quoted 34 lines · Fabian Deutsch
Hey.
When running a guest in the following setup, I observ a "massive" performance decrease after a couple of gigabytes of transfrerred data.
other guests keep running fine (no regressions) and nothing unusual is observed on the host.
fedora 8 host, kvm-72 (user and module)
fedora 9 guest, all updates.
nic 0, vtio:
subnet a
samba listeningnic 1, vtio:
subnet b
iscsi initiator runningsamba shares a mount point, pointing to an iscsi target.
now pulling/pushing about 80GB over samba results in a drop from about 60/70Mbit to about 1/2Mbit of troughput.
The problem is reproducible.
The problem is solved after a reboot of the guest. Just restarting networking doesn't help.
Guest's dmesg doesn't say anything about problems. No errors in ifconfig.Has someone observed something similar?
Quoted 9 lines
Greetings
- fabian--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.htmlWe just saw the very same issue.
guest & host : 2.6.25.3, debian etch 64 bits
Quoted 8 lines
KVM 70 on this machine.
As fabian said, rebooting the kernel doesn't help, but rebooting the whole kvm guest does.
No error messages as far as I can tell. The kvm guest is a clonezilla server and send lots of data.
It began to crawl after 2/3 weeks of usage.
Are you also using model=vortio?
If so, you might want to switch to e1000. Less performance but stable.
The issues seems to be related to virtio/tap.
And a little bit more to the tap itnerface/implementation, because of the reboot-observation.
fabian
Quoted 4 lines · Fabian Deutsch
Quoted 4 lines
We just saw the very same issue.
guest & host : 2.6.25.3, debian etch 64 bits
...
Quoted 11 lines
Are you also using model=vortio?
If so, you might want to switch to e1000. Less performance but stable.The issues seems to be related to virtio/tap.
And a little bit more to the tap itnerface/implementation, because of the reboot-observation.fabian
Yes it is virtio.
e1000 works but it is pure emulation, so :
Less performance
More CPU usage.
This server is in pre-prod right now, So I'd rather try to help debugging the issue for the moment.
greetings,
Signature · 2 lines
Yann Dupont - Service IRTS, DSI Université de Nantes Tel : 02.51.12.53.91 - Mail/Jabber : [email protected]
Hi,
Quoted 7 lines · Fabian Deutsch
Hey.
When running a guest in the following setup, I observ a "massive" performance decrease after a couple of gigabytes of transfrerred data.
other guests keep running fine (no regressions) and nothing unusual is observed on the host.
If you do "killall -ALRM qemu-kvm" after the slowdown has occurred, does it make things speed up again for a while?
If so, the issue is fixed by:
It also got applied in 2.6.26.3 and 2.6.25.16.
Cheers,
Mark.
Am Dienstag, den 09.09.2008, 11:22 +0100 schrieb Mark McLoughlin:
Quoted 5 lines
Hi,
Quoted 2 lines · Fabian Deutsch
Hey.
Quoted 9 lines
Quoted 5 lines
When running a guest in the following setup, I observ a "massive" performance decrease after a couple of gigabytes of transfrerred data.
other guests keep running fine (no regressions) and nothing unusual is observed on the host.
If you do "killall -ALRM qemu-kvm" after the slowdown has occurred, does it make things speed up again for a while?
It seems so!
Fantastic.
Quoted 6 lines
If so, the issue is fixed by:
It also got applied in 2.6.26.3 and 2.6.25.16.
So it was some signaling-bug on the .. client side?
Cheers!
fabian
Quoted 19 lines · Fabian Deutsch
Am Dienstag, den 09.09.2008, 11:22 +0100 schrieb Mark McLoughlin:
Quoted 5 lines
Hi,
Quoted 2 lines · Fabian Deutsch
Hey.
Quoted 9 lines
Quoted 5 lines
When running a guest in the following setup, I observ a "massive" performance decrease after a couple of gigabytes of transfrerred data.
other guests keep running fine (no regressions) and nothing unusual is observed on the host.
If you do "killall -ALRM qemu-kvm" after the slowdown has occurred, does it make things speed up again for a while?
It seems so!
Fantastic.
Quoted 8 lines
Quoted 6 lines
If so, the issue is fixed by:
It also got applied in 2.6.26.3 and 2.6.25.16.
So it was some signaling-bug on the .. client side?
No, it was a kernel race condition causing the qemu-kvm process to stop receiving SIGALRM signals.
The original thread might explain it a bit better:
http://thread.gmane.org/gmane.comp.emulators.kvm.devel/19582
Cheers,
Mark.
tis 2008-09-09 klockan 11:47 +0100 skrev Mark McLoughlin:
Quoted 24 lines · Fabian Deutsch
Quoted 23 lines
Am Dienstag, den 09.09.2008, 11:22 +0100 schrieb Mark McLoughlin:
Quoted 5 lines
Hi,
Quoted 2 lines · Fabian Deutsch
Hey.
Quoted 9 lines
Quoted 5 lines
When running a guest in the following setup, I observ a "massive" performance decrease after a couple of gigabytes of transfrerred data.
other guests keep running fine (no regressions) and nothing unusual is observed on the host.
If you do "killall -ALRM qemu-kvm" after the slowdown has occurred, does it make things speed up again for a while?
It seems so!
Fantastic.Quoted 4 lines
If so, the issue is fixed by:
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
It also got applied in 2.6.26.3 and 2.6.25.16.
So it was some signaling-bug on the .. client side?
No, it was a kernel race condition causing the qemu-kvm process to stop receiving SIGALRM signals.
So is it the host kernel that should be patched?
/Henrik Holst
Witsbits AB
Quoted 28 lines · Henrik Holst
tis 2008-09-09 klockan 11:47 +0100 skrev Mark McLoughlin:
Quoted 27 lines · Fabian Deutsch
Quoted 26 lines
Am Dienstag, den 09.09.2008, 11:22 +0100 schrieb Mark McLoughlin:
Quoted 5 lines
Hi,
Quoted 2 lines · Fabian Deutsch
Hey.
Quoted 9 lines
Quoted 5 lines
When running a guest in the following setup, I observ a "massive" performance decrease after a couple of gigabytes of transfrerred data.
other guests keep running fine (no regressions) and nothing unusual is observed on the host.
If you do "killall -ALRM qemu-kvm" after the slowdown has occurred, does it make things speed up again for a while?
It seems so!
Fantastic.Quoted 6 lines
If so, the issue is fixed by:
It also got applied in 2.6.26.3 and 2.6.25.16.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
So it was some signaling-bug on the .. client side?
No, it was a kernel race condition causing the qemu-kvm process to stop receiving SIGALRM signals.
So is it the host kernel that should be patched?
Yes.
(Or build qemu-kvm to not use signalfd)
Cheers,
Mark.
Quoted 9 lines · Mark McLoughlin
Hi,
Quoted 5 lines · Fabian Deutsch
Hey.
When running a guest in the following setup, I observ a "massive" performance decrease after a couple of gigabytes of transfrerred data.
Quoted 8 lines
Quoted 3 lines
other guests keep running fine (no regressions) and nothing unusual is observed on the host.
If you do "killall -ALRM qemu-kvm" after the slowdown has occurred, does it make things speed up again for a while?
Too soon to say if it works for us as it seems the problem is long to trigger in our case ; Anyway as the problem seems similar, it's good to know it is probably fixed.
Quoted 7 lines
If so, the issue is fixed by:
It also got applied in 2.6.26.3 and 2.6.25.16.
Ok, so upgrading the kernel is sufficient ?
Quoted 9 lines
Cheers,
Mark.--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Thanks for the answer,
greetings,
Signature · 2 lines
Yann Dupont - Service IRTS, DSI Université de Nantes Tel : 02.51.12.53.91 - Mail/Jabber : [email protected]