Skip to contentSkip to conversation

setting time in guest with ntpdate results in VM hang

DISCUSSION Dale Swanston 3 participants 5 messages Aug 24, 2012 – Aug 27, 2012 KVM

lore
Share & tools
lore
#0 lore raw

Hello.

We are running a guest OS of CentOS 4.4 (kernel 2.6.12) for legacy reasons, upgrading is not an option. NTP is running on the host and synching with a local GPS NTP server. But due to frequency drift in the guest it restarts itself periodically and upon start up performs an ntpdate to force a time jump on the guest.

I have seen 2 occasions now (over 2 months) where the VM hangs right as the ntpdate command alters the guest clock (based on output in /var/log/messages).

From the host's perspective the VM is still running but it appears to be using very high CPU percentage (more than typical). The only recovery option is to force shutdown of the VM and restart it.

1. Are there any known issues with ntpdate and VMs hanging? Any workarounds?
2. Are there any debugging tools further characterise the problem?

Thanks in advance,
Dale

Further specifics, let me know if more details are needed:

Host:

# uname -a
Linux s1pscu01host 2.6.32-220.el6.x86_64 #1 SMP Sat Dec 10 17:04:11 CST 2011 x86_64 x86_64 x86_64 GNU/Linux

on Intel(R) Xeon(R) CPU E5620 @ 2.40GHz

with qemu-kvm-0.12.1.2-2.209.el6.x86_64

Guest:
I don't use any kernel time modifiers in the guest ... kernel /vmlinuz-2.6.12 ro root=/dev/VolGroup00/LogVol00 rhgb quiet

CentOS 4.4

#1 lore raw
Quoted 8 lines · Dale Swanston

Hello.

We are running a guest OS of CentOS 4.4 (kernel 2.6.12) for legacy
reasons, upgrading is not an option. NTP is running on the host
and synching with a local GPS NTP server. But due to frequency drift in the guest it restarts itself periodically and upon start up performs an ntpdate to force a time jump on the guest.

Quoted 7 lines

I have seen 2 occasions now (over 2 months) where the VM hangs right as the ntpdate command alters the guest clock (based on output in /var/log/messages).

From the host's perspective the VM is still running but it appears to be using very high CPU percentage (more than typical). The only recovery option is to force shutdown of the VM and restart it.

This should not happen.

Quoted 3 lines

1. Are there any known issues with ntpdate and VMs hanging? Any workarounds?
2. Are there any debugging tools further characterise the problem?

Upgrading the guest kernel is not an option? At least install recent kernel in guest to confirm that its not an already fixed bug.

Quoted 24 lines

Thanks in advance,
Dale

Further specifics, let me know if more details are needed:

Host:

# uname -a
Linux s1pscu01host 2.6.32-220.el6.x86_64 #1 SMP Sat Dec 10 17:04:11
CST 2011 x86_64 x86_64 x86_64 GNU/Linux

on Intel(R) Xeon(R) CPU E5620 @ 2.40GHz

with qemu-kvm-0.12.1.2-2.209.el6.x86_64

Guest:
I don't use any kernel time modifiers in the guest ... kernel /vmlinuz-2.6.12 ro root=/dev/VolGroup00/LogVol00 rhgb quiet

CentOS 4.4
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw
Quoted 16 lines · Marcelo Tosatti
Quoted 8 lines · Dale Swanston

Hello.

We are running a guest OS of CentOS 4.4 (kernel 2.6.12) for legacy
reasons, upgrading is not an option. NTP is running on the host
and synching with a local GPS NTP server. But due to frequency drift in the guest it restarts itself periodically and upon start up performs an ntpdate to force a time jump on the guest.

Quoted 7 lines

I have seen 2 occasions now (over 2 months) where the VM hangs right as the ntpdate command alters the guest clock (based on output in /var/log/messages).

From the host's perspective the VM is still running but it appears
to be using very high CPU percentage (more than typical). The only recovery option is to force shutdown of the VM and restart it.

Quoted 7 lines

This should not happen.

Quoted 3 lines

1. Are there any known issues with ntpdate and VMs hanging? Any workarounds?
2. Are there any debugging tools further characterise the problem?

Upgrading the guest kernel is not an option? At least install recent kernel in guest to confirm that its not an already fixed bug.

Good idea. I'll try that.

But are there any tools available to determine what the VM is doing when it appears hung? I've looked but haven't found much on debug or diagnostics on a running VM. Any links?

Is it possible the guest kernel is panicking? What would the VM do if that happened? Would it shutdown?

Thanks again.

#3 lore raw
Quoted 5 lines · Dale Swanston

Good idea. I'll try that.

But are there any tools available to determine what the VM is doing when it appears hung? I've looked but haven't found much on debug or diagnostics on a running VM. Any links?

If you have the vmlinux, enable the gdbserver stub via Qemu's monitor. Then use 'gdb vmlinux', connect to the VM 'target remote host:port' and look at the backtrace.

I have seen something similar using kvm-clock in a guest running 2.6.27.

David

#4 lore raw
Quoted 10 lines · David Ahern
Quoted 5 lines · Dale Swanston

Good idea. I'll try that.

But are there any tools available to determine what the VM is doing when it appears hung? I've looked but haven't found much on debug or diagnostics on a running VM. Any links?

If you have the vmlinux, enable the gdbserver stub via Qemu's monitor. Then use 'gdb vmlinux', connect to the VM 'target remote host:port' and look at the backtrace.

Another option is to boot the host with profile=kvm, wait for the guest to hang, then do:

readprofile -r ; readprofile -m System-map-of-guest.map

Quoted 3 lines

I have seen something similar using kvm-clock in a guest running 2.6.27.

David