strange guest slowness after some time

I have a strange slowness which affects some guests after they are running for some time. "Slowness" can happen a few hours after guest start, or, a couple of days after guest start.

What do I mean by "slowness"?

This is how long it takes to log in via SSH to an unaffected guest - below a second:

$ time ssh backupuser@normal_guest exit
0.02user 0.01system 0:00.67elapsed 4%CPU (0avgtext+0avgdata 0maxresident)

Now, let's try to log in to the affected guest running on the same host
- more than 12 seconds:

$ time ssh backupuser@slow_guest exit
0.02user 0.01system 0:12.56elapsed 0%CPU (0avgtext+0avgdata 0maxresident)

If I log in via SSH to the affected guest, any key presses lag a second or two.

This is actually weird - if I run something IO intensive on the guest, the login is much faster (running CPU-intensive tasks makes no difference):

guest# dd if=/dev/vda of=/dev/null

$ time ssh backupuser@slow_guest exit
0.02user 0.00system 0:00.70elapsed 2%CPU (0avgtext+0avgdata 0maxresident)

Also, running "ping -f <slow_guest>" helps a lot and SSH logins are fast.

Look at the difference here - 7470ms vs 139183ms (and packet losses):

# ping -f -c 10000 normal_guest

10000 packets transmitted, 10000 received, 0% packet loss, time 7470ms rtt min/avg/max/mdev = 0.443/0.709/6.487/0.112 ms, ipg/ewma 0.747/0.716 ms

# ping -f -c 10000 slow_guest

10000 packets transmitted, 9934 received, 0% packet loss, time 139183ms rtt min/avg/max/mdev = 0.470/14.337/50.455/5.409 ms, pipe 4, ipg/ewma 13.919/14.788 ms

CPU-intensive tasks are as fast as on unaffected guests. Reading from /dev/vda is as fast as on unaffected guests.

So the only thing broken seems to be the network.

Rebooting the guest does not help - it is still slow.
The only thing that helps is stopping the guest and starting it again (i.e., stopping kvm process and starting a new one).

Is there an explanation to this phenomenon? Looks like a problem with virtio drivers somewhere, or?

The host is running kvm-83.
Affected guests are running 2.6.27.14 kernels and use virtio drivers. The problem happens only _sometimes_. Out of 9 guests I have running on this host, I saw this problem only on 3 guests. I never saw this happening on more than one guest at a time.
All three have 512 MB memory assigned, other guests have less memory.

are your nameservers ok?
ssh is reveres checking your ip, if your nameserver is not available login may take some time.

The host is running kvm-83.
Affected guests are running 2.6.27.14 kernels and use virtio drivers.
The problem happens only _sometimes_. Out of 9 guests I have running on this host, I saw this problem only on 3 guests. I never saw this happening on more than one guest at a time. All three have 512 MB memory assigned, other guests have less memory.

I upgraded ~2 days ago to kvm-84 and the same just happened for a guest with 256 MB memory.

Note how _time_ is different (similar timings are to other unaffected guests):

I have a strange slowness which affects some guests after they are running for some time. "Slowness" can happen a few hours after guest start, or, a couple of days after guest start.

What do I mean by "slowness"?

This is how long it takes to log in via SSH to an unaffected guest - below a second:

$ time ssh backupuser@normal_guest exit
0.02user 0.01system 0:00.67elapsed 4%CPU (0avgtext+0avgdata 0maxresident)

Now, let's try to log in to the affected guest running on the same host - more than 12 seconds:

$ time ssh backupuser@slow_guest exit
0.02user 0.01system 0:12.56elapsed 0%CPU (0avgtext+0avgdata 0maxresident)

If I log in via SSH to the affected guest, any key presses lag a second or two.

This is actually weird - if I run something IO intensive on the guest, the login is much faster (running CPU-intensive tasks makes no difference):

guest# dd if=/dev/vda of=/dev/null

$ time ssh backupuser@slow_guest exit
0.02user 0.00system 0:00.70elapsed 2%CPU (0avgtext+0avgdata 0maxresident)

Also, running "ping -f <slow_guest>" helps a lot and SSH logins are fast.

Look at the difference here - 7470ms vs 139183ms (and packet losses):

# ping -f -c 10000 normal_guest

10000 packets transmitted, 10000 received, 0% packet loss, time 7470ms rtt min/avg/max/mdev = 0.443/0.709/6.487/0.112 ms, ipg/ewma 0.747/0.716 ms

# ping -f -c 10000 slow_guest

10000 packets transmitted, 9934 received, 0% packet loss, time 139183ms rtt min/avg/max/mdev = 0.470/14.337/50.455/5.409 ms, pipe 4, ipg/ewma 13.919/14.788 ms

CPU-intensive tasks are as fast as on unaffected guests. Reading from /dev/vda is as fast as on unaffected guests.

So the only thing broken seems to be the network.

Rebooting the guest does not help - it is still slow.
The only thing that helps is stopping the guest and starting it again (i.e., stopping kvm process and starting a new one).

Is there an explanation to this phenomenon? Looks like a problem with virtio drivers somewhere, or?

The host is running kvm-83.
Affected guests are running 2.6.27.14 kernels and use virtio drivers.
The problem happens only _sometimes_. Out of 9 guests I have running on this host, I saw this problem only on 3 guests. I never saw this happening on more than one guest at a time. All three have 512 MB memory assigned, other guests have less memory.

I'm guessing there's a problem with timers or timer interrupts.

What is the host cpu?

Does the problem occur if you pin a guest to a cpu with taskset?

Quoted 3 lines · Avi Kivity

I'm guessing there's a problem with timers or timer interrupts.

What is the host cpu?

4 entries like this in /proc/cpuinfo:

processor : 3
vendor_id : AuthenticAMD
cpu family : 15
model : 65
model name : Dual-Core AMD Opteron(tm) Processor 2212

Quoted 1 line

Does the problem occur if you pin a guest to a cpu with taskset?

Like this?

# taskset -p 01 22906

Quoted 8 lines · Tomasz Chmielewski

Quoted 3 lines · Avi Kivity

I'm guessing there's a problem with timers or timer interrupts.

What is the host cpu?

4 entries like this in /proc/cpuinfo:

Quoted 6 lines

processor : 3
vendor_id : AuthenticAMD
cpu family : 15
model : 65
model name : Dual-Core AMD Opteron(tm) Processor 2212

That's probably the kvmclock issue that hit older AMDs. It was fixed in kvm-84, please try that.

Quoted 6 lines

Quoted 1 line

Does the problem occur if you pin a guest to a cpu with taskset?

Like this?

# taskset -p 01 22906

I meant 'taskset 01 qemu ...' but it wouldn't have helped if it's kvmclock.

Quoted 16 lines · Avi Kivity

Quoted 14 lines · Tomasz Chmielewski

Quoted 3 lines · Avi Kivity

I'm guessing there's a problem with timers or timer interrupts.

What is the host cpu?

4 entries like this in /proc/cpuinfo:

processor : 3
vendor_id : AuthenticAMD
cpu family : 15
model : 65
model name : Dual-Core AMD Opteron(tm) Processor 2212

Quoted 1 line

That's probably the kvmclock issue that hit older AMDs. It was fixed in kvm-84, please try that.

It is kvm-84, I have it running since Saturday (but I had this issue with kvm-83 as well).

Quoted 2 lines

I meant 'taskset 01 qemu ...' but it wouldn't have helped if it's kvmclock.

It can be done on a running process as well (22906 is the PID of the affected gue

Quoted 15 lines

Quoted 14 lines

Quoted 4 lines

Quoted 3 lines

I'm guessing there's a problem with timers or timer interrupts.

What is the host cpu?

Quoted 8 lines

4 entries like this in /proc/cpuinfo:

processor : 3
vendor_id : AuthenticAMD
cpu family : 15
model : 65
model name : Dual-Core AMD Opteron(tm) Processor 2212

That's probably the kvmclock issue that hit older AMDs. It was fixed in kvm-84, please try that.

Quoted 2 lines

It is kvm-84, I have it running since Saturday (but I had this issue with kvm-83 as well).

And the problem continues?

What's your current clocksource (in the guest)? Does changing it help?

See /sys/devices/system/clocksource/clocksource0/*.

Quoted 3 lines

Quoted 1 line

I meant 'taskset 01 qemu ...' but it wouldn't have helped if it's kvmclock.

It can be done on a running process as well (22906 is the PID of the affected gue

Right, but if the guest is poisoned somehow, this won't help.

Quoted 18 lines · Avi Kivity

Quoted 17 lines

Quoted 6 lines

Quoted 6 lines

Quoted 3 lines

I'm guessing there's a problem with timers or timer interrupts.

What is the host cpu?

4 entries like this in /proc/cpuinfo:

Quoted 8 lines

Quoted 6 lines

processor : 3
vendor_id : AuthenticAMD
cpu family : 15
model : 65
model name : Dual-Core AMD Opteron(tm) Processor 2212

That's probably the kvmclock issue that hit older AMDs. It was fixed in kvm-84, please try that.

It is kvm-84, I have it running since Saturday (but I had this issue with kvm-83 as well).

Quoted 5 lines

And the problem continues?

What's your current clocksource (in the guest)? Does changing it help?

See /sys/devices/system/clocksource/clocksource0/*.

It was kvm-clock.
I tried changing it to acpi_pm, jiffies, tsc, but it made no difference.

Quoted 3 lines

It was kvm-clock.
I tried changing it to acpi_pm, jiffies, tsc, but it made no difference.

Actually, I don't think that I checked tsc, because when I changed to jiffies, the time has stopped:

# echo jiffies > /sys/devices/system/clocksource/clocksource0/current_clocksource
# date
Mon Mar  9 12:29:00 CET 2009
# date
Mon Mar  9 12:29:00 CET 2009
# date
Mon Mar  9 12:29:00 CET 2009

Quoted 6 lines · Tomasz Chmielewski

Quoted 3 lines

It was kvm-clock.
I tried changing it to acpi_pm, jiffies, tsc, but it made no difference.

Actually, I don't think that I checked tsc, because when I changed to jiffies, the time has stopped:

Quoted 7 lines

# echo jiffies > /sys/devices/system/clocksource/clocksource0/current_clocksource
# date
Mon Mar  9 12:29:00 CET 2009
# date
Mon Mar  9 12:29:00 CET 2009
# date
Mon Mar  9 12:29:00 CET 2009

can you post some /proc/interrupt dumps from the guest? I guess the timer interrupt isn't working.

Does -no-kvm-irqchip help?

Quoted 6 lines

Quoted 3 lines

I'm guessing there's a problem with timers or timer interrupts.

What is the host cpu?

4 entries like this in /proc/cpuinfo:

Quoted 6 lines

processor : 3
vendor_id : AuthenticAMD
cpu family : 15
model : 65
model name : Dual-Core AMD Opteron(tm) Processor 2212

That's probably the kvmclock issue that hit older AMDs. It was fixed in kvm-84, please try that.

Quoted 2 lines

Does -no-kvm-irqchip help?

Nope, it doesn't - with jiffies, time always stops.

Quoted 6 lines · Tomasz Chmielewski

Quoted 2 lines

Does -no-kvm-irqchip help?

Nope, it doesn't - with jiffies, time always stops.

Here, too. This is strange. On bare metal it works as expected.

Quoted 13 lines · Avi Kivity

Quoted 12 lines

Quoted 3 lines

I'm guessing there's a problem with timers or timer interrupts.

What is the host cpu?

4 entries like this in /proc/cpuinfo:

processor : 3
vendor_id : AuthenticAMD
cpu family : 15
model : 65
model name : Dual-Core AMD Opteron(tm) Processor 2212

Quoted 1 line

That's probably the kvmclock issue that hit older AMDs. It was fixed in kvm-84, please try that.

I've been running it for about a week now with kvm-84 and no guest got slow.

Can it be related to using cpufreq and ondemand governor?

After a week or so, network in one guest got slow with kvm-84 and no cpufreq.

Quoted 3 lines · Tomasz Chmielewski

After a week or so, network in one guest got slow with kvm-84 and no cpufreq.

This is virtio, right? What about e1000?

(I realize it takes a week to reproduce, but maybe you have some more experience)

Quoted 5 lines · Avi Kivity

Quoted 1 line · Tomasz Chmielewski

After a week or so, network in one guest got slow with kvm-84 and no cpufreq.

This is virtio, right? What about e1000?

(I realize it takes a week to reproduce, but maybe you have some more experience)

Yes, all affected had virtio. Probably because I didn't have many guests with e1000 interface.

After a guest gets slow, I stop it and add another interface, e1000.

If it gets slow again, I'll check if e1000 interface is slow as well.

Will keep you updated.

Quoted 3 lines

If it gets slow again, I'll check if e1000 interface is slow as well.

Will keep you updated.

I see similar behavior: After a week one of my guests' network totally stops to respond. Only guests using virtio networking get hit. Both windows and linux guests are affected. My guests in production use e1000 and have never been hit.
While that can be a coincidence it seems very unlikely: Out of 3 virtio guests 2 have been hit, one repeatedly.
Out of 3 e1000 guests none has ever been hit.

Observed with kvm-83 and kvm-84 with the host running in-kernel KVM code (linux 2.6.25.7)

I see similar behavior: After a week one of my guests' network totally stops to respond. Only guests using virtio networking get hit. Both windows and linux guests are affected. My guests in production use e1000 and have never been hit.
While that can be a coincidence it seems very unlikely: Out of 3 virtio guests 2 have been hit, one repeatedly.
Out of 3 e1000 guests none has ever been hit.

Observed with kvm-83 and kvm-84 with the host running in-kernel KVM code (linux 2.6.25.7)

Quoted 6 lines · Felix Leimbach

I see similar behavior: After a week one of my guests' network totally stops to respond. Only guests using virtio networking get hit. Both windows and linux guests are affected. My guests in production use e1000 and have never been hit.
While that can be a coincidence it seems very unlikely: Out of 3 virtio guests 2 have been hit, one repeatedly.
Out of 3 e1000 guests none has ever been hit.

Observed with kvm-83 and kvm-84 with the host running in-kernel KVM code (linux 2.6.25.7)

Might it be that some counter overflowed? What are the packet counts on long running guests?

(output of ifconfig, even on an unaffected e1000 guest, might help)

Quoted 8 lines

Yes, all affected had virtio. Probably because I didn't have many guests with e1000 interface.

After a guest gets slow, I stop it and add another interface, e1000.

If it gets slow again, I'll check if e1000 interface is slow as well.

Will keep you updated.

I see similar behavior: After a week one of my guests' network totally stops to respond. Only guests using virtio networking get hit. Both windows and linux guests are affected.

Quoted 3 lines · Felix Leimbach

Out of 3 e1000 guests none has ever been hit.

Observed with kvm-83 and kvm-84 with the host running in-kernel KVM code (linux 2.6.25.7)

Could you add a (unused) e1000 interface to your virtio guests?
As this issue happens rarely for me, maybe you could help to reproduce it as well (i.e. if network gets slow on virtio interface, give e1000 a IP address, and try if network is also slow on e1000 on the very same guest).

BTW, what CPU do you have?

Quoted 2 lines

BTW, what CPU do you have?

One dual core Opteron 2212

Quoted 1 line · Avi Kivity

Might it be that some counter overflowed? What are the packet counts on long running guests?

I don't think so.

I just made both counters (TX, RX) of ifconfig for virtio interfaces overflow several times and everything is still as fast as it should be.

Quoted 1 line

(output of ifconfig, even on an unaffected e1000 guest, might help)

Quoted 5 lines · Tomasz Chmielewski

Quoted 1 line · Avi Kivity

Might it be that some counter overflowed? What are the packet counts on long running guests?

I don't think so.

I just made both counters (TX, RX) of ifconfig for virtio interfaces overflow several times and everything is still as fast as it should be.

I had overflows on the counters as well (32 bit guests) without an problem. Here is the current ifconfig output of a machine which suffered the problem before:

eth0 Link encap:Ethernet HWaddr 52:54:00:74:01:01

         inet addr:10.75.13.1  Bcast:10.75.255.255  Mask:255.255.0.0
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:3542104 errors:0 dropped:0 overruns:0 frame:0
         TX packets:412546 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:682285568 (650.6 MiB)  TX bytes:2907586796 (2.7 GiB)

currently I have e1000 only on windows guests. Is there a way to gather relevant statistics there too?

Quoted 1 line · Felix Leimbach

I see similar behavior: After a week one of my guests' network totally stops to respond. Only guests using virtio networking get hit. Both windows and linux guests are affected.

Also, does guest reboot help for you (for me, it doesn't)?

Or, you have to halt the guest and start it again (i.e. stop kvm/qemu process and start a new one) to make the network working properly again?

I noticed that you Tomasz are also running kvm-83. Maybe kvm-84 fixed the issue already?

I have not tried rebooting; always stopped and restarted the qemu instance. Will try on the next occasion.

Before I wrote that I tested on kvm-83 and 84 but it turns out the kvm-84 part was wrong: Since the upgrade 4 days ago I have not yet had a hang. I noticed that you Tomasz are also running kvm-83. Maybe kvm-84 fixed the issue already?

Quoted 1 line

BTW, what CPU do you have?

One dual core Opteron 2212
Note: I will upgrade to two Shanghai Quad-Cores in 2 weeks and test with those as well.

processor : 1
vendor_id : AuthenticAMD
cpu family : 15
model : 65
model name : Dual-Core AMD Opteron(tm) Processor 2212
stepping : 2
cpu MHz : 1994.996
cache size : 1024 KB

Quoted 6 lines · Felix Leimbach

Quoted 2 lines · Tomasz Chmielewski

Quoted 1 line · Avi Kivity

Might it be that some counter overflowed? What are the packet counts on long running guests?

Here is the current ifconfig output of a machine which suffered the problem before:

Quoted 7 lines

eth0 Link encap:Ethernet HWaddr 52:54:00:74:01:01

         inet addr:10.75.13.1  Bcast:10.75.255.255  Mask:255.255.0.0
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:3542104 errors:0 dropped:0 overruns:0 frame:0
         TX packets:412546 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:682285568 (650.6 MiB)  TX bytes:2907586796 (2.7 GiB)

packet counters are will within 32-bit limits. byte counters not so interesting.

Does idle=poll help things? It can cause tsc breakage similar to cpufreq.

Quoted 15 lines · Avi Kivity

Quoted 13 lines · Felix Leimbach

Quoted 2 lines · Tomasz Chmielewski

Quoted 1 line · Avi Kivity

Might it be that some counter overflowed? What are the packet counts on long running guests?

Here is the current ifconfig output of a machine which suffered the problem before:

eth0 Link encap:Ethernet HWaddr 52:54:00:74:01:01

         inet addr:10.75.13.1  Bcast:10.75.255.255  Mask:255.255.0.0
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:3542104 errors:0 dropped:0 overruns:0 frame:0
         TX packets:412546 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:682285568 (650.6 MiB)  TX bytes:2907586796 (2.7 GiB)

Quoted 1 line

packet counters are will within 32-bit limits. byte counters not so interesting.

Ah OK.
I did only byte overflow.

Packet overflow will take much longer. It's one of these very rare cases where setting very small MTU is useful...

Quoted 7 lines · Felix Leimbach

eth0 Link encap:Ethernet HWaddr 52:54:00:74:01:01

         inet addr:10.75.13.1  Bcast:10.75.255.255  Mask:255.255.0.0
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:3542104 errors:0 dropped:0 overruns:0 frame:0
         TX packets:412546 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:682285568 (650.6 MiB)  TX bytes:2907586796 (2.7 GiB)

packet counters are will within 32-bit limits. byte counters not so interesting.

Do you experience the slowdown on Windows guests?

Quoted 7 lines · Tomasz Chmielewski

Quoted 1 line · Avi Kivity

packet counters are will within 32-bit limits. byte counters not so interesting.

Ah OK.
I did only byte overflow.

Packet overflow will take much longer. It's one of these very rare cases where setting very small MTU is useful...

OK, another bug found.

Set your MTU to 100.

On two hosts, do:

HOST1_MTU1500# dd if=/dev/zero | ssh manager@HOST2 dd of=/dev/null HOST2_MTU100# dd if=/dev/zero | ssh manager@HOST1 dd of=/dev/null

HOST2 with MTU 100 will crash after 10-15 minutes (with packet count still not overflown).

Quoted 11 lines

OK, another bug found.

Set your MTU to 100.

On two hosts, do:

HOST1_MTU1500# dd if=/dev/zero | ssh manager@HOST2 dd of=/dev/null HOST2_MTU100# dd if=/dev/zero | ssh manager@HOST1 dd of=/dev/null

HOST2 with MTU 100 will crash after 10-15 minutes (with packet count still not overflown).

Intersting. What are the packet counter at crash time (roughly)?

My - currently running - test is:

Guest 1 (Linux):
MTU 150

# cat /dev/zero | nc <guest2ip> 7777

Guest 2 (Windows 2003 Server):
MTU: 1500

# nc -l -p 7777 > NUL

My packet are currently at 63 million without a problem - yet.

See this screenshot:

http://www1.wpkg.org/lockup.png

Guest that locks up is running Debian Lenny with 2.6.26 kernel. Guest that does not lock up runs Mandriva 2009.0 with 2.6.27.x kernel. (data being transferred both side to/from each of these hosts).

Sorry, both machines run Debian Lenny and 2.6.26 kernel.
The only difference is that machine which crashes (with MTU=100) or locks up (with MTU=1500) runs a "2.6.26-1-686" kernel and the one which doesn't lock up runs "2.6.26-1-486" kernel (both are Debian's kernels).

Quoted 2 lines · Tomasz Chmielewski

Sorry, both machines run Debian Lenny and 2.6.26 kernel.
The only difference is that machine which crashes (with MTU=100) or locks up (with MTU=1500) runs a "2.6.26-1-686" kernel and the one which doesn't lock up runs "2.6.26-1-486" kernel (both are Debian's kernels).

Some more tries and I got this one. Serial console died, but SSH is still working.

Note the "S" tainted flag.
According to Documentation/oops-tracing.txt, it means:

3: 'S' if the oops occurred on an SMP kernel running on hardware that
hasn't been certified as safe to run multiprocessor. Currently this occurs only on various Athlons that are not SMP capable.

And this is a difference between "2.6.26-1-686" and "2.6.26-1-486" kernels.

# grep -i smp /boot/config-2.6.26-1-686
CONFIG_X86_SMP=y
CONFIG_X86_32_SMP=y
CONFIG_SMP=y

# grep -i smp /boot/config-2.6.26-1-486
CONFIG_BROKEN_ON_SMP=y
# CONFIG_SMP is not set

packet counters are will within 32-bit limits. byte counters not so interesting.

Quoted 17 lines · Felix Leimbach

Quoted 11 lines

OK, another bug found.

Set your MTU to 100.

On two hosts, do:

HOST1_MTU1500# dd if=/dev/zero | ssh manager@HOST2 dd of=/dev/null HOST2_MTU100# dd if=/dev/zero | ssh manager@HOST1 dd of=/dev/null

HOST2 with MTU 100 will crash after 10-15 minutes (with packet count still not overflown).

Intersting. What are the packet counter at crash time (roughly)?

My - currently running - test is:

Guest 1 (Linux):
MTU 150

Quoted 7 lines

# cat /dev/zero | nc <guest2ip> 7777

Guest 2 (Windows 2003 Server):
MTU: 1500

# nc -l -p 7777 > NUL

My packet are currently at 63 million without a problem - yet.

I have it running with MTU 1500. And one of the guests (the one which was crashing with MTU=100) froze.

On a VNC console I can see:

virtio_net virtio0: id 64 is not a head!

BUG: soft lockup - CPU#0 stuck for 61s! [ssh:2265]

And "soft lockup" is being printed periodically. VNC and serial console do not react to any key press. Guest do not react on ACPI events (shutdown). kvm/qemu process is using 100% CPU.

See this screenshot:

http://www1.wpkg.org/lockup.png

Guest that locks up is running Debian Lenny with 2.6.26 kernel. Guest that does not lock up runs Mandriva 2009.0 with 2.6.27.x kernel. (data being transferred both side to/from each of these hosts).

Quoted 1 line · Avi Kivity

Does idle=poll help things? It can cause tsc breakage similar to cpufreq.

On the host, right? Can't test that as I cannot reboot the server.
Is tsc breakage still s.th. to watch out after I've upgraded to the Shanghai quadcores?

Quoted 1 line · Felix Leimbach

Is tsc breakage still s.th. to watch out after I've upgraded to the Shanghai quadcores?

No, should be gone.

Will you have the old server around so we can test things?

Quoted 5 lines · Avi Kivity

Quoted 1 line · Felix Leimbach

Is tsc breakage still s.th. to watch out after I've upgraded to the Shanghai quadcores?

No, should be gone.

Quoted 1 line

Will you have the old server around so we can test things?

No, I'll be upgrading the existing server. If you have specific tests in mind I can perform them in the next two weeks before the upgrade. But I cannot restart the server because a few VMs are in production use.

If a developer is interested in the old CPU (Opteron 2212) then I can have it mailed to him/her.

Since there are now at least two reports, maybe it won't be that difficult. If you can figure out a way to reliably reproduce this, that would be most helpful.

Quoted 1 line · Tomasz Chmielewski

virtio_net virtio0: id 64 is not a head!

Quoted 2 lines · On Wednesday 18 March 2009 16:59:36 Avi Kivity wrote:

Quoted 1 line · Tomasz Chmielewski

virtio_net virtio0: id 64 is not a head!

This means that qemu said "I've finished with buffer 64" and the guest didn't know anything about buffer 64.

We should not lock up, tho networking is toast: I think that qemu got upset and that caused this as well as it to chew 100% cpu.

I'll see if I can reproduce with kvm-84 userspace and 2.6.27 guests, 32-bit guests on a 64-bit AMD host. What's your kvm/qemu command line?

Thanks,
Rusty.
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 12 lines · Rusty Russell

Quoted 2 lines · On Wednesday 18 March 2009 16:59:36 Avi Kivity wrote:

Quoted 1 line · Tomasz Chmielewski

virtio_net virtio0: id 64 is not a head!

This means that qemu said "I've finished with buffer 64" and the guest didn't know anything about buffer 64.

We should not lock up, tho networking is toast: I think that qemu got upset and that caused this as well as it to chew 100% cpu.

I'll see if I can reproduce with kvm-84 userspace and 2.6.27 guests, 32-bit guests on a 64-bit AMD host. What's your kvm/qemu command line?

I've hit this as well.

Intel host, running RHEL5.3, x86_64 with KVM-81.

Guest is RHEL4.7, 32-bit, with the virtio drivers from RHEL4.8 beta.

Happens pretty darn quickly for me.

david

Quoted 8 lines

Thanks,
Rusty.
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 17 lines · David S. Ahern

Quoted 12 lines · Rusty Russell

Quoted 2 lines · On Wednesday 18 March 2009 16:59:36 Avi Kivity wrote:

Quoted 1 line · Tomasz Chmielewski

virtio_net virtio0: id 64 is not a head!

This means that qemu said "I've finished with buffer 64" and the guest didn't know anything about buffer 64.

We should not lock up, tho networking is toast: I think that qemu got upset and that caused this as well as it to chew 100% cpu.

I'll see if I can reproduce with kvm-84 userspace and 2.6.27 guests, 32-bit guests on a 64-bit AMD host. What's your kvm/qemu command line?

I've hit this as well.

Intel host, running RHEL5.3, x86_64 with KVM-81.

Quoted 6 lines

Guest is RHEL4.7, 32-bit, with the virtio drivers from RHEL4.8 beta.

Happens pretty darn quickly for me.

david

Like I said, pretty darn quickly.

Quoted 1 line

Note how _time_ is different (similar timings are to other unaffected guests):

This is also pretty interesting:

# ping -c 10 <unaffected guest>
PING 192.168.4.4 (192.168.4.4) 56(84) bytes of data.
64 bytes from 192.168.4.4: icmp_seq=1 ttl=64 time=1.25 ms
64 bytes from 192.168.4.4: icmp_seq=2 ttl=64 time=1.58 ms

--- 192.168.4.4 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9091ms rtt min/avg/max/mdev = 1.031/2.059/3.894/1.045 ms

How probable it is so many pings returned with exactly 1000 ms?

# ping -c 10 <affected_guest>
PING 192.168.4.5 (192.168.4.5) 56(84) bytes of data.
64 bytes from 192.168.4.5: icmp_seq=1 ttl=64 time=1009 ms
64 bytes from 192.168.4.5: icmp_seq=2 ttl=64 time=9.61 ms
64 bytes from 192.168.4.5: icmp_seq=3 ttl=64 time=1000 ms
64 bytes from 192.168.4.5: icmp_seq=4 ttl=64 time=1000 ms

Quoted 24 lines · David S. Ahern

Quoted 22 lines · David S. Ahern

Quoted 7 lines · Rusty Russell

Quoted 2 lines · On Wednesday 18 March 2009 16:59:36 Avi Kivity wrote:

Quoted 1 line · Tomasz Chmielewski

virtio_net virtio0: id 64 is not a head!

This means that qemu said "I've finished with buffer 64" and the guest didn't
know anything about buffer 64.

Quoted 8 lines

We should not lock up, tho networking is toast: I think that qemu got upset
and that caused this as well as it to chew 100% cpu.

I'll see if I can reproduce with kvm-84 userspace and 2.6.27 guests, 32-bit
guests on a 64-bit AMD host. What's your kvm/qemu command line?

I've hit this as well.

Intel host, running RHEL5.3, x86_64 with KVM-81.

Guest is RHEL4.7, 32-bit, with the virtio drivers from RHEL4.8 beta.

Quoted 6 lines

Quoted 4 lines

Happens pretty darn quickly for me.

david

Like I said, pretty darn quickly.

Can you reproduce it also with e1000 instead of virtio?

Quoted 3 lines · Tomasz Chmielewski

Quoted 2 lines · Felix Leimbach

Out of 3 e1000 guests none has ever been hit.

Quoted 3 lines

Quoted 1 line

Observed with kvm-83 and kvm-84 with the host running in-kernel KVM code (linux 2.6.25.7)

Could you add a (unused) e1000 interface to your virtio guests?
As this issue happens rarely for me, maybe you could help to reproduce it as well (i.e. if network gets slow on virtio interface, give e1000 a IP address, and try if network is also slow on e1000 on the very same guest).

Will do and report

Quoted 2 lines

BTW, what CPU do you have?

One dual core Opteron 2212
Note: I will upgrade to two Shanghai Quad-Cores in 2 weeks and test with those as well.

Quoted 11 lines · Felix Leimbach

Quoted 10 lines · Tomasz Chmielewski

Quoted 4 lines · Felix Leimbach

Out of 3 e1000 guests none has ever been hit.

Observed with kvm-83 and kvm-84 with the host running in-kernel KVM code (linux 2.6.25.7)

Could you add a (unused) e1000 interface to your virtio guests? As this issue happens rarely for me, maybe you could help to reproduce it as well (i.e. if network gets slow on virtio interface, give e1000 a IP address, and try if network is also slow on e1000 on the very same guest).

Quoted 6 lines

Will do and report

Quoted 2 lines

BTW, what CPU do you have?

One dual core Opteron 2212
Note: I will upgrade to two Shanghai Quad-Cores in 2 weeks and test with those as well.

I have this "slowness" on an Intel CPU as well, after about 10 days of guest uptime (using virtio net):

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Xeon(R) CPU 3050 @ 2.13GHz
stepping : 6
cpu MHz : 2133.410
cache size : 2048 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge

mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr lahf_lm
bogomips : 4266.87
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

Quoted 10 lines

Quoted 10 lines

Quoted 5 lines

Could you add a (unused) e1000 interface to your virtio guests? As this issue happens rarely for me, maybe you could help to reproduce it as well (i.e. if network gets slow on virtio interface, give e1000 a IP address, and try if network is also slow on e1000 on the very same guest).

Will do and report

Quoted 1 line

BTW, what CPU do you have?

One dual core Opteron 2212
Note: I will upgrade to two Shanghai Quad-Cores in 2 weeks and test with those as well.

Quoted 8 lines

I have this "slowness" on an Intel CPU as well, after about 10 days of guest uptime (using virtio net):

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Xeon(R) CPU 3050 @ 2.13GHz

For the Intel server, the guest is using the e1000 NIC or virtio or other? I have a few DL320G5s with this processor; I have not hit this problem running rhel3 and rhel4 guests using e1000/scsi devices.

As I mentioned, it was using virtio net.

Guests running with e1000 (and virtio_blk) don't have this problem.

Quoted 3 lines · Tomasz Chmielewski

As I mentioned, it was using virtio net.

Guests running with e1000 (and virtio_blk) don't have this problem.

Also, virtio_console seem to be affected by this "slowness" issue.

Quoted 6 lines · On Tuesday 07 April 2009 00:49:17 Tomasz Chmielewski wrote:

Quoted 3 lines · Tomasz Chmielewski

As I mentioned, it was using virtio net.

Guests running with e1000 (and virtio_blk) don't have this problem.

Also, virtio_console seem to be affected by this "slowness" issue.

I'm pretty sure this is different. Older virtio_console code ignored interrupts and polled, and use a heuristic to back off on polling (this was because we used the generic "hvc" infrastructure which hacked support).

You'll find a delay on the first keystroke after idle, but none on the second.

Quoted 5 lines · On Tuesday 07 April 2009 00:49:17 Tomasz Chmielewski wrote:

Quoted 3 lines · Tomasz Chmielewski

As I mentioned, it was using virtio net.

Guests running with e1000 (and virtio_blk) don't have this problem.

Quoted 1 line

Also, virtio_console seem to be affected by this "slowness" issue.

I'm pretty sure this is different. Older virtio_console code ignored interrupts and polled, and use a heuristic to back off on polling (this was because we used the generic "hvc" infrastructure which hacked support).

You'll find a delay on the first keystroke after idle, but none on the second.

I still observe this "slowness" with kvm-86 after the guest is running for some time (virtio_net and virtio_console seem to be affected; guest restart doesn't fix it).

Quoted 3 lines · Tomasz Chmielewski

I still observe this "slowness" with kvm-86 after the guest is running for some time (virtio_net and virtio_console seem to be affected; guest restart doesn't fix it).

Anything in guest dmesg?

Quoted 5 lines · Avi Kivity

Quoted 3 lines · Tomasz Chmielewski

I still observe this "slowness" with kvm-86 after the guest is running for some time (virtio_net and virtio_console seem to be affected; guest restart doesn't fix it).

Quoted 1 line

Anything in guest dmesg?

No.
No hints in syslog, dmesg...

Can it be that this is more likely to happens on "busy" hosts?

It happens for me on a host where I have 16 guests running.

Also, as I booted the host almost 2 days ago, 2 or 3 guests didn't start properly (16 guests were starting at the same time), with their kernel saying:

    Kernel panic - not syncing: IO-APIC + timer doesn't work!

Can it be related?

After I restarted these failed guests, they started properly.

Quoted 6 lines · Tomasz Chmielewski

Quoted 5 lines · Avi Kivity

Quoted 3 lines · Tomasz Chmielewski

I still observe this "slowness" with kvm-86 after the guest is running for some time (virtio_net and virtio_console seem to be affected; guest restart doesn't fix it).

Quoted 8 lines

Quoted 1 line

Anything in guest dmesg?

No.
No hints in syslog, dmesg...

Can it be that this is more likely to happens on "busy" hosts?

We'll only know once we fix it...

Maybe virtio is racy and a loaded host exposes the race.

Quoted 1 line

Maybe virtio is racy and a loaded host exposes the race.

I see it happening with virtio on 2.6.29.x guests as well.

So, what would you do if you saw it on your systems as well? ;)

Add some debug routines into virtio_* modules?

Quoted 8 lines · Tomasz Chmielewski

Quoted 1 line

Maybe virtio is racy and a loaded host exposes the race.

I see it happening with virtio on 2.6.29.x guests as well.

So, what would you do if you saw it on your systems as well? ;)

Add some debug routines into virtio_* modules?

I'm no virtio expert. Maybe I'd insert tracepoints to record interrupts and kicks.

Accidentally, I made some "interesting" discovery.

This ~2 MB video shows a kvm-86 guest being rebooted and GRUB started:

http://syneticon.net/kvm/kvm-slowness.ogg

GRUB has its timeout set to 50 seconds, and is supposed to show it on the screen by decreasing the number of seconds shown, every second.

Here, GRUB decreases the second counter very fast by 2 seconds, then waits 2 seconds, then again decreases the number of sends by 2 seconds very fast, and so on.

Perhaps my wording does not describe it very well though, so just try to download the video and open it i.e. in mplayer.

Quoted 4 lines · Felix Leimbach

Quoted 1 line

BTW, what CPU do you have?

One dual core Opteron 2212
Note: I will upgrade to two Shanghai Quad-Cores in 2 weeks and test with those as well.

Quoted 8 lines

processor : 1
vendor_id : AuthenticAMD
cpu family : 15
model : 65
model name : Dual-Core AMD Opteron(tm) Processor 2212
stepping : 2
cpu MHz : 1994.996
cache size : 1024 KB

It's exactly the same CPU I have.

Quoted 1 line

It's exactly the same CPU I have.

Interesting: Since two months I'm running on 2 Shanghai Quad-Cores instead and the problem is definitely gone. The rest of the hardware as well as the whole software-stack remained unchanged.

That should confirm what we assumed already.