gettimeofday "slow" in RHEL4 guests

I noticed that gettimeofday in RHEL4.6 guests is taking much longer than with RHEL3.8 guests. I wrote a simple program (see below) to call gettimeofday in a loop 1,000,000 times and then used time to measure how long it took.

For the RHEL3.8 guest:
time -p ./timeofday_bench
real 0.99
user 0.12
sys 0.24

For the RHEL4.6 guest with the default clock source (pmtmr):
time -p ./timeofday_bench
real 15.65
user 0.18
sys 15.46

and RHEL4.6 guest with PIT as the clock source (clock=pit kernel parameter):
time -p ./timeofday_bench
real 13.67
user 0.21
sys 13.45

So, basically gettimeofday() takes about 50 times as long on a RHEL4 guest.

Host is a DL380G5, 2 dual-core Xeon 5140 processors, 4 GB of RAM. It's running kvm.git tree as of 11/18/08 with kvm-75 userspace. Guest in both RHEL3 and RHEL4 cases has 4 vcpus, 3.5GB of RAM.

david

----------

timeofday_bench.c:

#include <sys/time.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[])

{
	int rc = 0, n;
	struct timeval tv;
	int iter = 1000000;  /* number of times to call gettimeofday */

	if (argc > 1)
		iter = atoi(argv[1]);

	if (iter == 0) {
		fprintf(stderr, "invalid number of iterations\n");
		return 1;
	}

	printf("starting.... ");
	for (n = 0; n < iter; ++n) {
		if (gettimeofday(&tv, NULL) != 0) {
			fprintf(stderr, "\ngettimeofday failed\n");
			rc = 1;
			break;
		}
	}

	if (!rc)
		printf("done\n");

	return rc;
}

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Any chance the pit or pmtmr options can be optimized a bit?

Quoted 2 lines · David S. Ahern

Any chance the pit or pmtmr options can be optimized a bit?

They both will require vmexits and never be really fast. Same for HPET. If you want fast gtod you really need to make TSC work.

-Andi

Signature · 5 lines

[email protected]
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Why does hpet need to be slow? Can't you just 1:1 pass through one of the hpet timers if you only have a limited amount of vms?

If done cleverly this might even work if #hpet > #cpu.

Quoted 2 lines

Why does hpet need to be slow? Can't you just 1:1 pass through one of the hpet timers if you only have a limited amount of vms?

HPET is not a truly virtualizable device, it's all the counters in one block that cannot be really mapped to different people.

Also most systems have very little counters and Linux typically needs two at least (system timer and /dev/hpet)

But yeah - the remapping of HPET timers to virtual HPET timers sounds pretty tough. I wonder if one could overcome that with a little hardware support though ...

RHEL3 (which is based on the 2.4.21 kernel) gets microsecond resolutions
by reading the TSC. Reading the TSC from within a guest is very fast on kvm.

RHEL4 (which is basd on the 2.6.9 kernel) allows multiple time sources:
pmtmr (ACPI power management timer which is the default), pit, hpet and TSC.

The pmtmr and pit both do ioport reads to get microsecond resolutions (see read_pmtmr and get_offset_pit, respectively). For the tsc as the timer source gettimeofday is *very* lightweight, but time drifts very badly and ntpd cannot acquire a sync.

Quoted 10 lines · David S. Ahern

RHEL3 (which is based on the 2.4.21 kernel) gets microsecond resolutions
by reading the TSC. Reading the TSC from within a guest is very fast on kvm.

RHEL4 (which is basd on the 2.6.9 kernel) allows multiple time sources:
pmtmr (ACPI power management timer which is the default), pit, hpet and TSC.

Quoted 4 lines

The pmtmr and pit both do ioport reads to get microsecond resolutions (see read_pmtmr and get_offset_pit, respectively). For the tsc as the timer source gettimeofday is *very* lightweight, but time drifts very badly and ntpd cannot acquire a sync.

Why aren't you seeing severe time drift when using RHEL3 guests with the TSC time source?

Quoted 3 lines

But yeah - the remapping of HPET timers to virtual HPET timers sounds pretty tough. I wonder if one could overcome that with a little hardware support though ...

For gettimeofday better make TSC work. Even in the best case (no virtualization) it is much faster than HPET because it sits in the CPU, while HPET is far away on the external south bridge.

Quoted 7 lines · Andi Kleen

Quoted 3 lines

But yeah - the remapping of HPET timers to virtual HPET timers sounds pretty tough. I wonder if one could overcome that with a little hardware support though ...

For gettimeofday better make TSC work. Even in the best case (no virtualization) it is much faster than HPET because it sits in the CPU, while HPET is far away on the external south bridge.

The tsc clock on older Linux 2.6 kernels compensates for lost ticks. The algorithm uses the PIT count (latched) to measure the delay between interrupt generation and handling, and sums that value, on the next interrupt, to the TSC delta.

Sheng investigated this problem in the discussions before in-kernel PIT was merged:

http://www.mail-archive.com/[email protected]/msg13873.html

The algorithm overcompensates for lost ticks and the guest time runs faster than the hosts.

There are two issues:

1) A bug in the in-kernel PIT which miscalculates the count value.

2) For the case where more than one interrupt is lost, and later reinjected, the value read from PIT count is meaningless for the purpose of the tsc algorithm. The count is interpreted as the delay until the next interrupt, which is not the case with reinjection.

As Sheng mentioned in the thread above, Xen pulls back the TSC value when reinjecting interrupts. VMWare ESX has a notion of "virtual TSC", which I believe is similar in this context.

For KVM I believe the best immediate solution (for now) is to provide an option to disable reinjection, behaving similarly to real hardware. The advantage is simplicity compared to virtualizing the time sources.

The QEMU PIT emulation has a limit on the rate of interrupt reinjection, perhaps something similar should be investigated in the future.

The following patch (which contains the bugfix for 1) and disabled reinjection) fixes the severe time drift on RHEL4 with "clock=tsc". What I'm proposing is to condition reinjection with an option (-kvm-pit-no-reinject or something).

Comments or better ideas?

arch/x86/kvm/i8254.c +4 -1

index e665d1c..608af7b 100644
@@ -201,13 +201,16 @@ static int __pit_timer_fn(struct kvm_kpit_state *ps)
 	if (!atomic_inc_and_test(&pt->pending))
 		set_bit(KVM_REQ_PENDING_TIMER, &vcpu0->requests);
 
+	if (atomic_read(&pt->pending) > 1)
+		atomic_set(&pt->pending, 1);
+
 	if (vcpu0 && waitqueue_active(&vcpu0->wq))
 		wake_up_interruptible(&vcpu0->wq);
 
 	hrtimer_add_expires_ns(&pt->timer, pt->period);
 	pt->scheduled = hrtimer_get_expires_ns(&pt->timer);
 	if (pt->period)
-		ps->channels[0].count_load_time = hrtimer_get_expires(&pt->timer);
+		ps->channels[0].count_load_time = ktime_get();
 
 	return (pt->period == 0 ? 0 : 1);
 }

The tsc clock on older Linux 2.6 kernels compensates for lost ticks. The algorithm uses the PIT count (latched) to measure the delay between interrupt generation and handling, and sums that value, on the next interrupt, to the TSC delta.

Sheng investigated this problem in the discussions before in-kernel PIT was merged:

http://www.mail-archive.com/[email protected]/msg13873.html

The algorithm overcompensates for lost ticks and the guest time runs faster than the hosts.

There are two issues:

1) A bug in the in-kernel PIT which miscalculates the count value.

2) For the case where more than one interrupt is lost, and later reinjected, the value read from PIT count is meaningless for the purpose of the tsc algorithm. The count is interpreted as the delay until the next interrupt, which is not the case with reinjection.

As Sheng mentioned in the thread above, Xen pulls back the TSC value when reinjecting interrupts. VMWare ESX has a notion of "virtual TSC", which I believe is similar in this context.

For KVM I believe the best immediate solution (for now) is to provide an option to disable reinjection, behaving similarly to real hardware. The advantage is simplicity compared to virtualizing the time sources.

The QEMU PIT emulation has a limit on the rate of interrupt reinjection, perhaps something similar should be investigated in the future.

The following patch (which contains the bugfix for 1) and disabled reinjection) fixes the severe time drift on RHEL4 with "clock=tsc". What I'm proposing is to condition reinjection with an option (-kvm-pit-no-reinject or something).

Comments or better ideas?

arch/x86/kvm/i8254.c +0 -0

index e665d1c..608af7b 100644

arch/x86/kvm/i8254.c +1 -1

 	if (vcpu0 && waitqueue_active(&vcpu0->wq))
 		wake_up_interruptible(&vcpu0->wq);
 
 	hrtimer_add_expires_ns(&pt->timer, pt->period);
 	pt->scheduled = hrtimer_get_expires_ns(&pt->timer);
 	if (pt->period)
-		ps->channels[0].count_load_time = hrtimer_get_expires(&pt->timer);
+		ps->channels[0].count_load_time = ktime_get();
 
 	return (pt->period == 0 ? 0 : 1);

}

Quoted 51 lines · Marcelo Tosatti

The tsc clock on older Linux 2.6 kernels compensates for lost ticks. The algorithm uses the PIT count (latched) to measure the delay between interrupt generation and handling, and sums that value, on the next interrupt, to the TSC delta.

Sheng investigated this problem in the discussions before in-kernel PIT was merged:

http://www.mail-archive.com/[email protected]/msg13873.html

The algorithm overcompensates for lost ticks and the guest time runs faster than the hosts.

There are two issues:

1) A bug in the in-kernel PIT which miscalculates the count value.

2) For the case where more than one interrupt is lost, and later reinjected, the value read from PIT count is meaningless for the purpose of the tsc algorithm. The count is interpreted as the delay until the next interrupt, which is not the case with reinjection.

As Sheng mentioned in the thread above, Xen pulls back the TSC value when reinjecting interrupts. VMWare ESX has a notion of "virtual TSC", which I believe is similar in this context.

For KVM I believe the best immediate solution (for now) is to provide an option to disable reinjection, behaving similarly to real hardware. The advantage is simplicity compared to virtualizing the time sources.

The QEMU PIT emulation has a limit on the rate of interrupt reinjection, perhaps something similar should be investigated in the future.

The following patch (which contains the bugfix for 1) and disabled reinjection) fixes the severe time drift on RHEL4 with "clock=tsc". What I'm proposing is to condition reinjection with an option (-kvm-pit-no-reinject or something).

Comments or better ideas?

arch/x86/kvm/i8254.c +0 -0

index e665d1c..608af7b 100644
@@ -201,13 +201,16 @@ static int __pit_timer_fn(struct kvm_kpit_state 
 *ps)
     if (!atomic_inc_and_test(&pt->pending))
         set_bit(KVM_REQ_PENDING_TIMER, &vcpu0->requests);
 

arch/x86/kvm/i8254.c static int __pit_timer_fn(struct kvm_kpit_state *ps) · line 209

Quoted 4 lines

arch/x86/kvm/i8254.c +3 -0

+    if (atomic_read(&pt->pending) > 1)
+        atomic_set(&pt->pending, 1);
+
 

Replace the atomic_inc() with atomic_set(, 1) instead? One less test, and more important, the logic is scattered less around the source.

Quoted 11 lines

arch/x86/kvm/i8254.c +1 -1

     if (vcpu0 && waitqueue_active(&vcpu0->wq))
         wake_up_interruptible(&vcpu0->wq);
 
     hrtimer_add_expires_ns(&pt->timer, pt->period);
     pt->scheduled = hrtimer_get_expires_ns(&pt->timer);
     if (pt->period)
-        ps->channels[0].count_load_time = 
 hrtimer_get_expires(&pt->timer);
+        ps->channels[0].count_load_time = ktime_get();

Quoted 3 lines

     return (pt->period == 0 ? 0 : 1);

}

I don't like the idea of punting to the user but looks like we don't have a choice. Hopefully vendors will port kvmclock to these kernels and release them as updates -- time simply doesn't work will with virtualization, especially Linux guests.

Quoted 7 lines

arch/x86/kvm/i8254.c static int __pit_timer_fn(struct kvm_kpit_state *ps) · line 206

Quoted 5 lines

arch/x86/kvm/i8254.c +3 -0

+    if (atomic_read(&pt->pending) > 1)
+        atomic_set(&pt->pending, 1);
+

Replace the atomic_inc() with atomic_set(, 1) instead? One less test, and more important, the logic is scattered less around the source.

But having only a pending bit instead of a counter will cause kvm to drop pit irqs on rare high load situations. The disable reinjection option is better.

Except for these 'tsc compensate' guest, what are the occasions where the guest writes his tsc? If this is the only case we can disable reinjection once we trap tsc writes.

Quoted 7 lines

arch/x86/kvm/i8254.c static int __pit_timer_fn(struct kvm_kpit_state *ps) · line 206

Quoted 5 lines

arch/x86/kvm/i8254.c +3 -0

+    if (atomic_read(&pt->pending) > 1)
+        atomic_set(&pt->pending, 1);
+

Replace the atomic_inc() with atomic_set(, 1) instead? One less test, and more important, the logic is scattered less around the source.

But having only a pending bit instead of a counter will cause kvm to drop pit irqs on rare high load situations. The disable reinjection option is better.

Except for these 'tsc compensate' guest, what are the occasions where the guest writes his tsc? If this is the only case we can disable reinjection once we trap tsc writes.