Network Overruns
Hi all,
I'm running ~13 virtual machines in production on KVM 62 (unfortunately the best ubuntu-stable can provide for me :/) and i've noticed a lot of TX overruns, is this something i should be worried about? Are my machines losing data?
These machines are all bridged with options like this:
-net nic,macaddr=00:16:3e:00:00:05,vlan=0,model=e1000 -net tap,fd=17,script=,vlan=0 (where fd differs for each machine)
ifconfig output enclosed below:
br0 Link encap:Ethernet HWaddr 00:15:17:6b:2d:14
inet addr:my.ip.address Bcast:my.ip.addr.255 Mask:255.255.255.0
inet6 addr: my.ipv6 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:610572463 errors:0 dropped:0 overruns:0 frame:0
TX packets:108729730 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:824841768 (786.6 MB) TX bytes:3829964354 (3.5 GB)eth0 Link encap:Ethernet HWaddr 00:15:17:6b:2d:14
inet6 addr: my.ipv6 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1521528058 errors:0 dropped:108777021 overruns:0
frame:0
TX packets:276414465 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:417285897 (397.9 MB) TX bytes:3547313532 (3.3 GB)
Base address:0xef00 Memory:fdde0000-fde00000lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:64334 errors:0 dropped:0 overruns:0 frame:0
TX packets:64334 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:7516953 (7.1 MB) TX bytes:7516953 (7.1 MB)vnet0 Link encap:Ethernet HWaddr ca:c2:8b:75:ee:83
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0 inet6 addr: fe80::c8c2:8bff:fe75:ee83/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:468 (468.0 B)
vnet1 Link encap:Ethernet HWaddr 00:ff:c1:ca:e0:18
inet6 addr: fe80::2ff:c1ff:feca:e018/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:306211 errors:0 dropped:0 overruns:0 frame:0
TX packets:208417450 errors:0 dropped:0 overruns:292 carrier:0
collisions:0 txqueuelen:500
RX bytes:84842203 (80.9 MB) TX bytes:3341249731 (3.1 GB)vnet2 Link encap:Ethernet HWaddr 00:ff:55:94:72:9b
inet6 addr: fe80::2ff:55ff:fe94:729b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:243806 errors:0 dropped:0 overruns:0 frame:0
TX packets:227992073 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:58875698 (56.1 MB) TX bytes:1147433844 (1.0 GB)vnet3 Link encap:Ethernet HWaddr 00:ff:b1:b7:41:5f
inet6 addr: fe80::2ff:b1ff:feb7:415f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:813360 errors:0 dropped:0 overruns:0 frame:0
TX packets:228960883 errors:0 dropped:0 overruns:16 carrier:0
collisions:0 txqueuelen:500
RX bytes:64048107 (61.0 MB) TX bytes:1217812685 (1.1 GB)vnet4 Link encap:Ethernet HWaddr 00:ff:6e:87:17:7b
inet6 addr: fe80::2ff:6eff:fe87:177b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:24463 errors:0 dropped:0 overruns:0 frame:0
TX packets:227534964 errors:0 dropped:0 overruns:501 carrier:0
collisions:0 txqueuelen:500
RX bytes:4312887 (4.1 MB) TX bytes:1060554904 (1011.4 MB)vnet5 Link encap:Ethernet HWaddr 00:ff:50:6a:26:62
inet6 addr: fe80::2ff:50ff:fe6a:2662/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:57623 errors:0 dropped:0 overruns:0 frame:0
TX packets:203904515 errors:0 dropped:0 overruns:491 carrier:0
collisions:0 txqueuelen:500
RX bytes:9015385 (8.5 MB) TX bytes:2856817773 (2.6 GB)vnet6 Link encap:Ethernet HWaddr 00:ff:3f:ff:32:86
inet6 addr: fe80::2ff:3fff:feff:3286/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:377799 errors:0 dropped:0 overruns:0 frame:0
TX packets:233685442 errors:0 dropped:0 overruns:1 carrier:0
collisions:0 txqueuelen:500
RX bytes:67794919 (64.6 MB) TX bytes:2100174866 (1.9 GB)vnet7 Link encap:Ethernet HWaddr 00:ff:79:fd:64:94
inet6 addr: fe80::2ff:79ff:fefd:6494/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:306667 errors:0 dropped:0 overruns:0 frame:0
TX packets:228114579 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:42935219 (40.9 MB) TX bytes:1879470986 (1.7 GB)vnet9 Link encap:Ethernet HWaddr 00:ff:b4:9c:e6:31
inet6 addr: fe80::2ff:b4ff:fe9c:e631/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:227503853 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:0 (0.0 B) TX bytes:1057149624 (1008.1 MB)vnet10 Link encap:Ethernet HWaddr 00:ff:6a:2f:08:5e
inet6 addr: fe80::2ff:6aff:fe2f:85e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:29123 errors:0 dropped:0 overruns:0 frame:0
TX packets:227532228 errors:0 dropped:0 overruns:218 carrier:0
collisions:0 txqueuelen:500
RX bytes:1591885 (1.5 MB) TX bytes:1058977339 (1009.9 MB)vnet11 Link encap:Ethernet HWaddr 00:ff:d4:76:9d:de
inet6 addr: fe80::2ff:d4ff:fe76:9dde/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:218597 errors:0 dropped:0 overruns:0 frame:0
TX packets:227728885 errors:0 dropped:0 overruns:566 carrier:0
collisions:0 txqueuelen:500
RX bytes:31377899 (29.9 MB) TX bytes:1084583551 (1.0 GB)vnet12 Link encap:Ethernet HWaddr d2:11:1d:f4:46:c0
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0 inet6 addr: fe80::d011:1dff:fef4:46c0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:468 (468.0 B)
vnet13 Link encap:Ethernet HWaddr 00:ff:77:a1:0c:2a
inet6 addr: fe80::2ff:77ff:fea1:c2a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:30510393 errors:0 dropped:0 overruns:0 frame:0
TX packets:249244012 errors:0 dropped:0 overruns:2036 carrier:0
collisions:0 txqueuelen:500
RX bytes:3854788416 (3.5 GB) TX bytes:2761388091 (2.5 GB)Cheers,
Henri
Quoted 7 lines · Henri Cook
Hi all,
I'm running ~13 virtual machines in production on KVM 62 (unfortunately the best ubuntu-stable can provide for me :/) and i've noticed a lot of TX overruns, is this something i should be worried about? Are my machines losing data?
There shouldn't be any data loss, since the guest expects the network to be lossy. But this can cause performance drops due to retries.
Most likely the guest's rx queue length is greater than the host's. You might try
ifconfig vnet0 txqueuelen 1500
(and so on for every interface)
or perhaps reducing the guests' txqueuelen.
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
Quoted 9 lines
Most likely the guest's rx queue length is greater than the host's. You might try
ifconfig vnet0 txqueuelen 1500
(and so on for every interface)
or perhaps reducing the guests' txqueuelen.
I have the same issue (overruns) and I have stalled network (in my other report I can only unstall the network by putting the NIC in promisc back and forth).
The txqueuelen is indeed greater in the guest then the tap on the host, and I also notice that it's almost ways the output NIc which get stuck.
So changing the tap (on the host) to 1500, I can't not reproduce the hang yet and a ping -f -s 64000 (between guests) does not produce any packet loss.
27810 packets transmitted, 27809 received, 0% packet loss, time 259080ms rtt min/avg/max/mdev = 5.013/7.639/33.999/1.536 ms, pipe 3, ipg/ewma 9.316/7.284 ms
I hope this is the solution for my network stalls and packet loss in games.
Quoted 18 lines · xming
Quoted 10 lines
Most likely the guest's rx queue length is greater than the host's. You might try
ifconfig vnet0 txqueuelen 1500
(and so on for every interface)
or perhaps reducing the guests' txqueuelen.
I have the same issue (overruns) and I have stalled network (in my other report I can only unstall the network by putting the NIC in promisc back and forth).
The txqueuelen is indeed greater in the guest then the tap on the host, and I also notice that it's almost ways the output NIc which get stuck.
Quoted 9 lines
So changing the tap (on the host) to 1500, I can't not reproduce the hang yet and a ping -f -s 64000 (between guests) does not produce any packet loss.
27810 packets transmitted, 27809 received, 0% packet loss, time 259080ms rtt min/avg/max/mdev = 5.013/7.639/33.999/1.536 ms, pipe 3, ipg/ewma 9.316/7.284 ms
I hope this is the solution for my network stalls and packet loss in games.
It's not a fix, rather a workaround. Hopefully some networking guru will come up with a real fix.
What NIC model are you using in the guest?
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
A message that is not in the archive (sent off-list or not yet received): [email protected]
(re-adding list)
Quoted 7 lines · xming
Quoted 2 lines
What NIC model are you using in the guest?
I have tried 8319, e1000 and virtio both had stalls. I am using virtio with txqueuelen 1500 on the tap.
I didin't had this proble with kvm 69, 70.
What is the oldest version that doesn't work? (i.e. when was the regression introduced?)
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 2 lines
What is the oldest version that doesn't work? (i.e. when was the regression introduced?)
I can (almost) be sure the kvm-70 did not have stalls and that kvm 73 certainly has this problem.
Never tried 71 and I am not sure about 72 (haven't ran it for long time and upgraded to 73)
Quoted 9 lines · xming
Quoted 2 lines
What is the oldest version that doesn't work? (i.e. when was the regression introduced?)
I can (almost) be sure the kvm-70 did not have stalls and that kvm 73 certainly has this problem.
Never tried 71 and I am not sure about 72 (haven't ran it for long time and upgraded to 73)
Ok I tried kvm 69 to 75, it does not seems to be kvm version related. It seems to be host kernel version realted.
- with 2.6.26.x I have very bad network, stalled nfs, stalled https, packet loss
- with 2.6.25.x I don't have stalled (or at least much less frequent, haven't seen one yet) and much less packet loss
- with 2.6.27-rcX network seems to be fine but can't test for long because the virtio block hangs is back :(
Quoted 10 lines
Quoted 9 lines
Quoted 2 lines
What is the oldest version that doesn't work? (i.e. when was the regression introduced?)
I can (almost) be sure the kvm-70 did not have stalls and that kvm 73 certainly has this problem.
Never tried 71 and I am not sure about 72 (haven't ran it for long time and upgraded to 73)
Quoted 8 lines
Ok I tried kvm 69 to 75, it does not seems to be kvm version related. It seems to be host kernel version realted.
- with 2.6.26.x I have very bad network, stalled nfs, stalled https, packet loss
- with 2.6.25.x I don't have stalled (or at least much less frequent, haven't seen one yet) and much less packet loss
- with 2.6.27-rcX network seems to be fine but can't test for long because the virtio block hangs is back :(
An other user on the irc came to the same conclusion
< OpenTokix> I have just shy of 3000 new connections per second, and I get strange timeouts on a server
running 2.6.26 (inside kvm) and not as many on
2.6.25 - both using virtio-interface