RHEL5.5, 32-bit VM repeatedly locks up due to kvmclock
After a few days of debugging I think kvmclock is the source of lockups for a RHEL5.5-based VM. The VM works fine on one host, but repeatedly locks up on another.
Server 1 - VM locks up repeatedly
-- DL580 G5
-- 4 quad-core X7350 processors at 2.93GHz
-- 48GB RAM
Server 2 - VM works just fine
-- DL380 G6
-- 2 quad-core E5540 processors at 2.53GHz
-- 24GB RAM
Both host servers are running Fedora Core 12, 2.6.32.11-99.fc12.x86_64 kernel. I have tried various versions of qemu-kvm -- the version in FC-12 and the version for FC-12 in virt-preview. In both cases the qemu-kvm command line is identical.
VM
- RHEL5.5, PAE kernel (also tried standard 32-bit)
- 2 vcpus
- 3GB RAM
- virtio network and disk
When the VM locks up both vcpu threads are spinning at 100%. Changing the clocksource to jiffies appears to have addressed the problem.
David
Quoted 19 lines · On Friday 23 April 2010 12:08:22 David S. Ahern wrote:
After a few days of debugging I think kvmclock is the source of lockups for a RHEL5.5-based VM. The VM works fine on one host, but repeatedly locks up on another.
Server 1 - VM locks up repeatedly
-- DL580 G5
-- 4 quad-core X7350 processors at 2.93GHz
-- 48GB RAMServer 2 - VM works just fine
-- DL380 G6
-- 2 quad-core E5540 processors at 2.53GHz
-- 24GB RAMBoth host servers are running Fedora Core 12, 2.6.32.11-99.fc12.x86_64 kernel. I have tried various versions of qemu-kvm -- the version in FC-12 and the version for FC-12 in virt-preview. In both cases the qemu-kvm command line is identical.
Quoted 8 lines
VM
- RHEL5.5, PAE kernel (also tried standard 32-bit)
- 2 vcpus
- 3GB RAM
- virtio network and diskWhen the VM locks up both vcpu threads are spinning at 100%. Changing the clocksource to jiffies appears to have addressed the problem.
Does changing the guest to -smp 1 help?
Quoted 6 lines
David
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 27 lines · Brian Jackson
Quoted 25 lines · On Friday 23 April 2010 12:08:22 David S. Ahern wrote:
After a few days of debugging I think kvmclock is the source of lockups for a RHEL5.5-based VM. The VM works fine on one host, but repeatedly locks up on another.
Server 1 - VM locks up repeatedly
-- DL580 G5
-- 4 quad-core X7350 processors at 2.93GHz
-- 48GB RAMServer 2 - VM works just fine
-- DL380 G6
-- 2 quad-core E5540 processors at 2.53GHz
-- 24GB RAMBoth host servers are running Fedora Core 12, 2.6.32.11-99.fc12.x86_64 kernel. I have tried various versions of qemu-kvm -- the version in FC-12 and the version for FC-12 in virt-preview. In both cases the qemu-kvm command line is identical.
VM
- RHEL5.5, PAE kernel (also tried standard 32-bit)
- 2 vcpus
- 3GB RAM
- virtio network and disk
Quoted 7 lines
Quoted 3 lines
When the VM locks up both vcpu threads are spinning at 100%. Changing the clocksource to jiffies appears to have addressed the problem.
Does changing the guest to -smp 1 help?
Based on our current understanding of the problem, it should help, but it may not prevent the problem entirely.
There are three issues with kvmclock due to sampling:
1) smp clock alignment may be slightly off due to timing conditions
2) kvmclock is resampled at each switch of vcpu to another pcpu
3) kvmclock granularity exceeds that of kernel timespec, which means sampling errors may show even on UP
Recommend using a different clocksource (tsc is great if you have stable TSC and don't migrate across different-speed machines) until we have all the fixes in place.
Zach
Quoted 41 lines · Zachary Amsden
Quoted 27 lines · Brian Jackson
Quoted 25 lines · On Friday 23 April 2010 12:08:22 David S. Ahern wrote:
After a few days of debugging I think kvmclock is the source of lockups for a RHEL5.5-based VM. The VM works fine on one host, but repeatedly locks up on another.
Server 1 - VM locks up repeatedly
-- DL580 G5
-- 4 quad-core X7350 processors at 2.93GHz
-- 48GB RAMServer 2 - VM works just fine
-- DL380 G6
-- 2 quad-core E5540 processors at 2.53GHz
-- 24GB RAMBoth host servers are running Fedora Core 12, 2.6.32.11-99.fc12.x86_64 kernel. I have tried various versions of qemu-kvm -- the version in FC-12 and the version for FC-12 in virt-preview. In both cases the qemu-kvm command line is identical.
VM
- RHEL5.5, PAE kernel (also tried standard 32-bit)
- 2 vcpus
- 3GB RAM
- virtio network and diskQuoted 7 lines
Quoted 3 lines
When the VM locks up both vcpu threads are spinning at 100%. Changing the clocksource to jiffies appears to have addressed the problem.
Does changing the guest to -smp 1 help?
Based on our current understanding of the problem, it should help, but it may not prevent the problem entirely.
There are three issues with kvmclock due to sampling:
Quoted 8 lines
1) smp clock alignment may be slightly off due to timing conditions
2) kvmclock is resampled at each switch of vcpu to another pcpu
3) kvmclock granularity exceeds that of kernel timespec, which means sampling errors may show even on UPRecommend using a different clocksource (tsc is great if you have stable TSC and don't migrate across different-speed machines) until we have all the fixes in place.
That's my plan for now. As I recall jiffies was the default in early RHEL5 versions. Not sure what that means hardware wise.
The biggest problem for me is that RHEL5.5 defaults to kvmclock; I'll find some workaround for it.
David
Quoted 2 lines
Zach
Quoted 53 lines · David S. Ahern
Quoted 41 lines · Zachary Amsden
Quoted 27 lines · Brian Jackson
Quoted 25 lines · On Friday 23 April 2010 12:08:22 David S. Ahern wrote:
After a few days of debugging I think kvmclock is the source of lockups for a RHEL5.5-based VM. The VM works fine on one host, but repeatedly locks up on another.
Server 1 - VM locks up repeatedly
-- DL580 G5
-- 4 quad-core X7350 processors at 2.93GHz
-- 48GB RAMServer 2 - VM works just fine
-- DL380 G6
-- 2 quad-core E5540 processors at 2.53GHz
-- 24GB RAMBoth host servers are running Fedora Core 12, 2.6.32.11-99.fc12.x86_64 kernel. I have tried various versions of qemu-kvm -- the version in FC-12 and the version for FC-12 in virt-preview. In both cases the qemu-kvm command line is identical.
VM
- RHEL5.5, PAE kernel (also tried standard 32-bit)
- 2 vcpus
- 3GB RAM
- virtio network and diskQuoted 7 lines
Quoted 3 lines
When the VM locks up both vcpu threads are spinning at 100%. Changing the clocksource to jiffies appears to have addressed the problem.
Does changing the guest to -smp 1 help?
Based on our current understanding of the problem, it should help, but it may not prevent the problem entirely.
There are three issues with kvmclock due to sampling:
Quoted 8 lines
1) smp clock alignment may be slightly off due to timing conditions
2) kvmclock is resampled at each switch of vcpu to another pcpu
3) kvmclock granularity exceeds that of kernel timespec, which means sampling errors may show even on UPRecommend using a different clocksource (tsc is great if you have stable TSC and don't migrate across different-speed machines) until we have all the fixes in place.
Quoted 5 lines
That's my plan for now. As I recall jiffies was the default in early RHEL5 versions. Not sure what that means hardware wise.
The biggest problem for me is that RHEL5.5 defaults to kvmclock; I'll find some workaround for it.
Could you try hpet? I had similar problem with multicore and multiCPU (per mother board) [even with constant_tsc].
Since I changed the guest to hpet i had no more problems.
Quoted 9 lines
David
Quoted 2 lines
Zach
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 3 lines
Bruno Ribas - [email protected] http://www.inf.ufpr.br/ribas C3SL: http://www.c3sl.ufpr.br
Quoted 5 lines · BRUNO CESAR RIBAS
Could you try hpet? I had similar problem with multicore and multiCPU (per mother board) [even with
constant_tsc].Since I changed the guest to hpet i had no more problems.
It's stable in the sense of no lockups yet, but is a much slower time source from a gettimeofday perspective compared to tsc and jiffies (based on speed jiffies appears to be tsc-based).
David
Quoted 11 lines
Quoted 10 lines
David
Quoted 2 lines
Zach
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 12 lines · David S. Ahern
Quoted 5 lines · BRUNO CESAR RIBAS
Could you try hpet? I had similar problem with multicore and multiCPU (per mother board) [even with
constant_tsc].Since I changed the guest to hpet i had no more problems.
It's stable in the sense of no lockups yet, but is a much slower time source from a gettimeofday perspective compared to tsc and jiffies (based on speed jiffies appears to be tsc-based).
Jiffies doesn't sample any hardware; instead, a timer interrupt causes a counter to be incremented, and that counter is sampled. The downside is that clock granularity is very low - you can't use it for accurate timing.
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.