Loss of network connectivity with high load

DISCUSSION Daniel Bareiro 5 participants 9 messages Jun 15, 2010 – Jun 22, 2010 KVM

Share & tools

Daniel Bareiro

author

2010-06-15 21:01 UTC #0 lore raw

Hi all!

I'm using Linux 2.6.31.13 compiled with the kernel.org source code on KVM host with Debian GNU/Linux Lenny amd64. Also I'm using Debian GNU/Linux Lenny amd64 virtual machine with kernel 2.6.26-2 from Debian repositories. I'm using qemu-kvm 0.12.3.

These are the parameters I'm using to start the virtual machine:

escher:~# ps ax|grep kvm
 6299 ?        Rl   113:40 /usr/local/qemu-kvm/bin/qemu-system-x86_64
-drive file=/dev/vm/belvedere-raiz,cache=none,if=virtio,boot=on -drive
file=/dev/vm/belvedere-u01,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u02,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u03,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u04,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u05,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u06,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u07,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u08,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u09,cache=none,if=virtio -m 3072 -smp 2 -net
nic,model=virtio,macaddr=00:16:3e:00:00:56 -net tap -daemonize -vnc :2
-k es -localtime -monitor telnet:localhost:4002,server,nowait -serial
telnet:localhost:4042,server,nowait

When I try to copy directories of several gigabytes in this VM using rsync to other hosts, this virtual machine loses connectivity with the rest of the network. Checking, using a serial console connection, /var/log/syslog and /var/log/messages in search of the problem, I don't see information that might give a clue to the problem.

This problem is not with the network driver by default (rtl8139), so I think that should be something related to Virtio.

Thanks in advance for your reply.

Regards,
Daniel

Signature · 2 lines

Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

Brian Jackson

2010-06-15 21:13 UTC #1 lore raw

Quoted 27 lines · On Tuesday 15 June 2010 16:00:38 Daniel Bareiro wrote:

Hi all!

These are the parameters I'm using to start the virtual machine:

escher:~# ps ax|grep kvm
 6299 ?        Rl   113:40 /usr/local/qemu-kvm/bin/qemu-system-x86_64
-drive file=/dev/vm/belvedere-raiz,cache=none,if=virtio,boot=on -drive
file=/dev/vm/belvedere-u01,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u02,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u03,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u04,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u05,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u06,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u07,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u08,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u09,cache=none,if=virtio -m 3072 -smp 2 -net
nic,model=virtio,macaddr=00:16:3e:00:00:56 -net tap -daemonize -vnc :2
-k es -localtime -monitor telnet:localhost:4002,server,nowait -serial
telnet:localhost:4042,server,nowait

Quoted 8 lines

When I try to copy directories of several gigabytes in this VM using rsync to other hosts, this virtual machine loses connectivity with the rest of the network. Checking, using a serial console connection, /var/log/syslog and /var/log/messages in search of the problem, I don't see information that might give a clue to the problem.
This problem is not with the network driver by default (rtl8139), so I think that should be something related to Virtio.

There have been a few bugs similar to this reported that should be pretty easy to find. Basically, I'd say try 0.12.4, it should have a few fixes in this area over 0.12.3.

Quoted 5 lines

Thanks in advance for your reply.
Regards,
Daniel

Daniel Bareiro

author

2010-06-16 03:36 UTC #2 lore raw

Hi, Brian.

Quoted 32 lines · Brian Jackson

Quoted 32 lines
I'm using Linux 2.6.31.13 compiled with the kernel.org source code on KVM host with Debian GNU/Linux Lenny amd64. Also I'm using Debian GNU/Linux Lenny amd64 virtual machine with kernel 2.6.26-2 from Debian repositories. I'm using qemu-kvm 0.12.3.
These are the parameters I'm using to start the virtual machine:
escher:~# ps ax|grep kvm
 6299 ?        Rl   113:40 /usr/local/qemu-kvm/bin/qemu-system-x86_64
-drive file=/dev/vm/belvedere-raiz,cache=none,if=virtio,boot=on -drive
file=/dev/vm/belvedere-u01,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u02,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u03,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u04,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u05,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u06,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u07,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u08,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u09,cache=none,if=virtio -m 3072 -smp 2 -net
nic,model=virtio,macaddr=00:16:3e:00:00:56 -net tap -daemonize -vnc :2
-k es -localtime -monitor telnet:localhost:4002,server,nowait -serial
telnet:localhost:4042,server,nowait
When I try to copy directories of several gigabytes in this VM using rsync to other hosts, this virtual machine loses connectivity with the rest of the network. Checking, using a serial console connection, /var/log/syslog and /var/log/messages in search of the problem, I don't see information that might give a clue to the problem.

Quoted 6 lines

Quoted 2 lines
This problem is not with the network driver by default (rtl8139), so I think that should be something related to Virtio.
There have been a few bugs similar to this reported that should be pretty easy to find. Basically, I'd say try 0.12.4, it should have a few fixes in this area over 0.12.3.

I was testing with qemu-kvm 0.12.4 but I'm having the same problem. I re-read the announce about qemu-kvm 0.12.4 sent by Avi, but I didn't find in it some reference about fixes related to Virtio.

If there's anything else I can try, please tell me.

Regards,
Daniel

Signature · 2 lines

Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

Daniel Bareiro

author

2010-06-16 10:19 UTC #3 lore raw

Quoted 9 lines · Daniel Bareiro

Quoted 3 lines
There have been a few bugs similar to this reported that should be pretty easy to find. Basically, I'd say try 0.12.4, it should have a few fixes in this area over 0.12.3.
I was testing with qemu-kvm 0.12.4 but I'm having the same problem. I re-read the announce about qemu-kvm 0.12.4 sent by Avi, but I didn't find in it some reference about fixes related to Virtio.
If there's anything else I can try, please tell me.

I found some references to similar problems:

http://forum.proxmox.com/threads/3117-virtio-net-crashing-(stop-sending-traffic)
http://www.mail-archive.com/[email protected]/msg26033.html
http://marc.info/?l=kvm&m=126564542625725&w=2

In particular, I note the patches provided by Tom Lendacky and Herbert Xu. Are these patches applied to qemu-kvm 0.12.3 or 0.12.4? If so, it could be a regression?

Thanks in advance for your reply.

Regards,
Daniel

Signature · 2 lines

Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

Avi Kivity

2010-06-16 14:14 UTC #4 lore raw

In reply to Daniel Bareiro

Quoted 8 lines · Daniel Bareiro

Hi all!
I'm using Linux 2.6.31.13 compiled with the kernel.org source code on KVM host with Debian GNU/Linux Lenny amd64. Also I'm using Debian GNU/Linux Lenny amd64 virtual machine with kernel 2.6.26-2 from Debian repositories. I'm using qemu-kvm 0.12.3.

2.6.26 is really old. Can you try a new kernel in the guest? Recent kernels had many virtio fixes.

Signature · 1 line

error compiling committee.c: too many arguments to function

Daniel Bareiro

author

2010-06-16 18:10 UTC #5 lore raw

Hi, Avi.

Quoted 7 lines · Avi Kivity

Quoted 4 lines
I'm using Linux 2.6.31.13 compiled with the kernel.org source code on KVM host with Debian GNU/Linux Lenny amd64. Also I'm using Debian GNU/Linux Lenny amd64 virtual machine with kernel 2.6.26-2 from Debian repositories. I'm using qemu-kvm 0.12.3.
2.6.26 is really old. Can you try a new kernel in the guest? Recent kernels had many virtio fixes.

I tested with Linux 2.6.32-bpo.5-amd64 from backports repositories and so far I didn't have a problem. If the problem persists, I will comment to the list.

Thanks for your reply.

Regards,
Daniel

Signature · 2 lines

Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

Thomas Besser

2010-06-22 06:32 UTC #6 lore raw

Hi all,

Quoted 7 lines · Daniel Bareiro

Quoted 4 lines · Avi Kivity
Quoted 4 lines
I'm using Linux 2.6.31.13 compiled with the kernel.org source code on KVM host with Debian GNU/Linux Lenny amd64. Also I'm using Debian GNU/Linux Lenny amd64 virtual machine with kernel 2.6.26-2 from Debian repositories. I'm using qemu-kvm 0.12.3.

Quoted 6 lines

Quoted 2 lines
2.6.26 is really old. Can you try a new kernel in the guest? Recent kernels had many virtio fixes.
I tested with Linux 2.6.32-bpo.5-amd64 from backports repositories and so far I didn't have a problem. If the problem persists, I will comment to the list.

I think we have the same problem here: after about 3 GB of transfered data with rsync, network (virtio) in the guest stops working (restart of network solves the network stop).

Both host and guest Debian Lenny with 2.6.32. qemu-kvm is version 0.12.4. Will try 2.6.34 in guest and report, if the problem still exists.

Guest starting options:

/usr/bin/kvm -name sysimager -vnc :35 -usbdevice tablet -net nic,vlan=0,model=virtio,macaddr=00:08:15:00:09:95 -net tap,vlan=0,ifname=tap35,script=no -m 512 -monitor unix:/var/run/kvm/sysimager,server,nowait -drive index=0,media=disk,file=/vm/sysimager/sysimager.raw,if=virtio,boot=on -drive index=1,media=disk,file=/vm/sysimager/daten.raw,if=virtio -daemonize -vga std -k de -pidfile /var/run/kvm/sysimager.pid -tdf

No error messages in /var/log/syslog, but via vnc on console. Any hint how to copy and paste via vncviewer the message?

Regards
Thomas

Thomas Besser

2010-06-22 12:32 UTC #7 lore raw

Quoted 10 lines · Thomas Besser

Quoted 4 lines
Quoted 2 lines · Avi Kivity
2.6.26 is really old. Can you try a new kernel in the guest? Recent kernels had many virtio fixes.
I think we have the same problem here: after about 3 GB of transfered data with rsync, network (virtio) in the guest stops working (restart of network solves the network stop).
Both host and guest Debian Lenny with 2.6.32. qemu-kvm is version 0.12.4. Will try 2.6.34 in guest and report, if the problem still exists.

With a 2.6.34 guest kernel the problem seems to be gone (after about 6 GB transferred data).

Regards
Thomas

RW

2010-06-22 12:57 UTC #8 lore raw

I've had the same issues too. I'm using Gentoo. Kernel packages < 2.6.32-r7 (Release 7 not RC-7) have had this problem. Haven't tested -r8 to -r10. Kernel 2.6.32-r11 is currently working fine. 2.6.34 was working too but it crashed after two days with a clock syncing issue. I'm hopping that 2.6.32-r11 doesn't have this issue. It's currently running for 24 hours now. KVM is 0.12.4.

- Robert

Quoted 22 lines · Thomas Besser

Quoted 10 lines · Thomas Besser
Quoted 4 lines
Quoted 2 lines · Avi Kivity
2.6.26 is really old. Can you try a new kernel in the guest? Recent kernels had many virtio fixes.
I think we have the same problem here: after about 3 GB of transfered data with rsync, network (virtio) in the guest stops working (restart of network solves the network stop).
Both host and guest Debian Lenny with 2.6.32. qemu-kvm is version 0.12.4. Will try 2.6.34 in guest and report, if the problem still exists.
With a 2.6.34 guest kernel the problem seems to be gone (after about 6 GB transferred data).
Regards
Thomas
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html