Improve hpet accuracy

1. Introduction

This patch improves the hpet based guest clock in terms of drift and monotonicity.
Prior to this work the drift with hpet was greater than 2%, far above the .05% limit
for ntp to synchronize. With this code, the drift ranges from .001% to .0033% depending
on guest and physical platform.

Using hpet allows guest operating systems to provide monotonic time to their applications. Time sources other than hpet are not monotonic because of their reliance on tsc, which is not synchronized across physical processors.

Windows 2k864 and many Linux guests are supported with two policies, one for guests
that handle missed clock interrupts and the other for guests that require the correct number of interrupts.

Guests may use hpet for the timing source even if the physical platform has no visible
hpet. Migration is supported between physical machines which differ in physical
hpet visibility.

Most of the changes are in hpet.c. Two general facilities are added to track interrupt
progress. The ideas here and the facilities would be useful in vpt.c, for other time
sources, though no attempt is made here to improve vpt.c.

The following sections discuss hpet dependencies, interrupt delivery policies, live migration,
test results, and relation to recent work with monotonic time.

2. Virtual Hpet dependencies

The virtual hpet depends on the ability to read the physical or simulated (see discussion below) hpet. For timekeeping, the virtual hpet also depends on two new interrupt notification facilities to implement its policies for interrupt delivery.

2.1. Two modes of low-level hpet main counter reads.

In this implementation, the virtual hpet reads with read_64_main_counter(), exported by
time.c, either the real physical hpet main counter register directly or a "simulated"
hpet main counter.

The simulated mode uses a monotonic version of get_s_time() (NOW()), where the last
time value is returned whenever the current time value is less than the last time
value. In simulated mode, since it is layered on s_time, the underlying hardware
can be hpet or some other device. The frequency of the main counter in simulated
mode is the same as the standard physical hpet frequency, allowing live migration
between nodes that are configured differently.

If the physical platform does not have an hpet device, or if xen is configured not
to use the device, then the simulated method is used. If there is a physical hpet device,
and xen has initialized it, then either simulated or physical mode can be used.
This is governed by a boot time option, hpet-avoid. Setting this option to 1 gives the
simulated mode and 0 the physical mode. The default is physical mode.

A disadvantage of the physical mode is that may take longer to read the device than in simulated mode. On some platforms the cost is about the same (less than 250 nsec) for
physical and simulated modes, while on others physical cost is much higher than simulated.
A disadvantage of the simulated mode is that it can return the same value for the counter in consecutive calls.

2.2. Interrupt notification facilities.

Two interrupt notification facilities are introduced, one is hvm_isa_irq_assert_cb()
and the other hvm_register_intr_en_notif().

The vhpet uses hvm_isa_irq_assert_cb to deliver interrupts to the vioapic. hvm_isa_irq_assert_cb allows a callback to be passed along to vioapic_deliver()
and this callback is called with a mask of the vcpus which will get the interrupt. This callback is made before any vcpus receive an interrupt.

Vhpet uses hvm_register_intr_en_notif() to register a handler for a particular vector that will be called when that vector is injected in [vmx,svm]_intr_assist()
and also when the guest finishes handling the interrupt. Here finished is defined
as the point when the guest re-enables interrupts or lowers the tpr value. EOI is not used as the end of interrupt as this is sometimes returned before the interrupt handler has done its work. A flag is passed to the handler indicating
whether this is the injection point (post = 1) or the interrupt finished (post = 0) point.
The need for the finished point callback is discussed in the missed ticks policy section.

To prevent a possible early trigger of the finished callback, intr_en_notif logic
has a two stage arm, the first at injection (hvm_intr_en_notif_arm()) and the second when
interrupts are seen to be disabled (hvm_intr_en_notif_disarm()). Once fully armed, re-enabling
interrupts will cause hvm_intr_en_notif_disarm() to make the end of interrupt callback. hvm_intr_en_notif_arm() and hvm_intr_en_notif_disarm() are called by [vmx,svm]_intr_assist().

3. Interrupt delivery policies

The existing hpet interrupt delivery is preserved. This includes vcpu round robin delivery used by Linux and broadcast delivery used by Windows.

There are two policies for interrupt delivery, one for Windows 2k8-64 and the other
for Linux. The Linux policy takes advantage of the (guest) Linux missed tick and offset
calculations and does not attempt to deliver the right number of interrupts. The Windows policy delivers the correct number of interrupts, even if sometimes much
closer to each other than the period. The policies are similar to those in vpt.c, though
there are some important differences.

Policies are selected with an HVMOP_set_param hypercall with index HVM_PARAM_TIMER_MODE.
Two new values are added, HVM_HPET_guest_computes_missed_ticks and HVM_HPET_guest_does_not_compute_missed_ticks. The reason that two new ones are added is that
in some guests (32bit Linux) a no-missed policy is needed for clock sources other than hpet
and a missed ticks policy for hpet. It was felt that there would be less confusion by simply
introducing the two hpet policies.

3.1. The missed ticks policy

The Linux clock interrupt handler for hpet calculates missed ticks and offset using the hpet
main counter. The algorithm works well when the time since the last interrupt is greater than
or equal to a period and poorly otherwise.

The missed ticks policy ensures that no two clock interrupts are delivered to the guest at
a time interval less than a period. A time stamp (hpet main counter value) is recorded (by a
callback registered with hvm_register_intr_en_notif) when Linux finishes handling the clock
interrupt. Then, ensuing interrupts are delivered to the vioapic only if the current main
counter value is a period greater than when the last interrupt was handled.

Tests showed a significant improvement in clock drift with end of interrupt time stamps
versus beginning of interrupt[1]. It is believed that the reason for the improvement
is that the clock interrupt handler goes for a spinlock and can be therefore delayed in its
processing. Furthermore, the main counter is read by the guest under the lock. The net
effect is that if we time stamp injection, we can get the difference in time between successive interrupt handler lock acquisitions to be less than the period.

3.2. The no-missed ticks policy

Windows 2k864 keeps very poor time with the missed ticks policy. So the no-missed ticks policy
was developed. In the no-missed ticks policy we deliver the correct number of interrupts,
even if they are spaced less than a period apart (when catching up).

Windows 2k864 uses a broadcast mode in the interrupt routing such that all vcpus get the clock interrupt. The best Windows drift performance was achieved when the
policy code ensured that all the previous interrupts (on the various vcpus) had been injected
before injecting the next interrupt to the vioapic..

The policy code works as follows. It uses the hvm_isa_irq_assert_cb() to record
the vcpus to be interrupted in h->hpet.pending_mask. Then, in the callback registered
with hvm_register_intr_en_notif() at post=1 time it clears the current vcpu in the pending_mask.
When the pending_mask is clear it decrements hpet.intr_pending_nr and if intr_pending_nr is still
non-zero posts another interrupt to the ioapic with hvm_isa_irq_assert_cb(). Intr_pending_nr is incremented in hpet_route_decision_not_missed_ticks().

The missed ticks policy intr_en_notif callback also uses the pending_mask method. So even though
Linux does not broadcast its interrupts, the code could handle it if it did. In this case the end of interrupt time stamp is made when the pending_mask is clear.

4. Live Migration

Live migration with hpet preserves the current offset of the guest clock with respect
to ntp. This is accomplished by migrating all of the state in the h->hpet data structure
in the usual way. The hp->mc_offset is recalculated on the receiving node so that the
guest sees a continuous hpet main counter.

Code as been added to xc_domain_save.c to send a small message after the domain context is sent. The contents of the message is the physical tsc timestamp, last_tsc,
read just before the message is sent. When the last_tsc message is received in xc_domain_restore.c,
another physical tsc timestamp, cur_tsc, is read. The two timestamps are loaded into the domain
structure as last_tsc_sender and first_tsc_receiver with hypercalls. Then xc_domain_hvm_setcontext
is called so that hpet_load has access to these time stamps. Hpet_load uses the timestamps
to account for the time spent saving and loading the domain context. With this technique,
the only neglected time is the time spent sending a small network message.

5. Test Results

Some recent test results are:

5.1 Linux 4u664 and Windows 2k864 load test.
Duration: 70 hours.
Test date: 6/2/08
Loads: usex -b48 on Linux; burn-in on Windows Guest vcpus: 8 for Linux; 2 for Windows
Hardware: 8 physical cpu AMD
Clock drift : Linux: .0012% Windows: .009%

5.2 Linux 4u664, Linux 4u464 , and Windows 2k864 no-load test
Duration: 23 hours.
Test date: 6/3/08
Loads: none
Guest vcpus: 8 for each Linux; 2 for Windows
Hardware: 4 physical cpu AMD
Clock drift : Linux: .033% Windows: .019%

6. Relation to recent work in xen-unstable

There is a similarity between hvm_get_guest_time() in xen-unstable and read_64_main_counter()
in this code. However, read_64_main_counter() is more tuned to the needs of hpet.c. It has no
"set" operation, only the get. It isolates the mode, physical or simulated, in read_64_main_counter()
itself. It uses no vcpu or domain state as it is a physical entity, in either mode. And it provides a real
physical mode for every read for those applications that desire this.

7. Conclusion

The virtual hpet is improved by this patch in terms of accuracy and monotonicity.
Tests performed to date verify this and more testing is under way.

8. Future Work

Testing with Windows Vista will be performed soon. The reason for accuracy variations
on different platforms using the physical hpet device will be investigated. Additional overhead measurements on simulated vs physical hpet mode will be made.

Footnotes:

1. I don't recall the accuracy improvement with end of interrupt stamping, but it was
significant, perhaps better than two to one improvement. It would be a very simple matter
to re-measure the improvement as the facility can call back at injection time as well.

Signed-off-by Dave Winchell <[email protected]>

<mailto:[email protected]>

Signed-off-by Ben Guthro <[email protected]>

<mailto:[email protected]>

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

1. Introduction

This patch improves the hpet based guest clock in terms of drift and monotonicity.
Prior to this work the drift with hpet was greater than 2%, far above the .05% limit
for ntp to synchronize. With this code, the drift ranges from .001% to .0033% depending
on guest and physical platform.

Using hpet allows guest operating systems to provide monotonic time to their applications. Time sources other than hpet are not monotonic because of their reliance on tsc, which is not synchronized across physical processors.

Windows 2k864 and many Linux guests are supported with two policies, one for guests
that handle missed clock interrupts and the other for guests that require the correct number of interrupts.

Guests may use hpet for the timing source even if the physical platform has no visible
hpet. Migration is supported between physical machines which differ in physical
hpet visibility.

Most of the changes are in hpet.c. Two general facilities are added to track interrupt
progress. The ideas here and the facilities would be useful in vpt.c, for other time
sources, though no attempt is made here to improve vpt.c.

The following sections discuss hpet dependencies, interrupt delivery policies, live migration,
test results, and relation to recent work with monotonic time.

2. Virtual Hpet dependencies

The virtual hpet depends on the ability to read the physical or simulated (see discussion below) hpet. For timekeeping, the virtual hpet also depends on two new interrupt notification facilities to implement its policies for interrupt delivery.

2.1. Two modes of low-level hpet main counter reads.

In this implementation, the virtual hpet reads with read_64_main_counter(), exported by
time.c, either the real physical hpet main counter register directly or a "simulated"
hpet main counter.

The simulated mode uses a monotonic version of get_s_time() (NOW()), where the last
time value is returned whenever the current time value is less than the last time
value. In simulated mode, since it is layered on s_time, the underlying hardware
can be hpet or some other device. The frequency of the main counter in simulated
mode is the same as the standard physical hpet frequency, allowing live migration
between nodes that are configured differently.

If the physical platform does not have an hpet device, or if xen is configured not
to use the device, then the simulated method is used. If there is a physical hpet device,
and xen has initialized it, then either simulated or physical mode can be used.
This is governed by a boot time option, hpet-avoid. Setting this option to 1 gives the
simulated mode and 0 the physical mode. The default is physical mode.

A disadvantage of the physical mode is that may take longer to read the device than in simulated mode. On some platforms the cost is about the same (less than 250 nsec) for
physical and simulated modes, while on others physical cost is much higher than simulated.
A disadvantage of the simulated mode is that it can return the same value for the counter in consecutive calls.

2.2. Interrupt notification facilities.

Two interrupt notification facilities are introduced, one is hvm_isa_irq_assert_cb()
and the other hvm_register_intr_en_notif().

The vhpet uses hvm_isa_irq_assert_cb to deliver interrupts to the vioapic. hvm_isa_irq_assert_cb allows a callback to be passed along to vioapic_deliver()
and this callback is called with a mask of the vcpus which will get the interrupt. This callback is made before any vcpus receive an interrupt.

Vhpet uses hvm_register_intr_en_notif() to register a handler for a particular vector that will be called when that vector is injected in [vmx,svm]_intr_assist()
and also when the guest finishes handling the interrupt. Here finished is defined
as the point when the guest re-enables interrupts or lowers the tpr value. EOI is not used as the end of interrupt as this is sometimes returned before the interrupt handler has done its work. A flag is passed to the handler indicating
whether this is the injection point (post = 1) or the interrupt finished (post = 0) point.
The need for the finished point callback is discussed in the missed ticks policy section.

To prevent a possible early trigger of the finished callback, intr_en_notif logic
has a two stage arm, the first at injection (hvm_intr_en_notif_arm()) and the second when
interrupts are seen to be disabled (hvm_intr_en_notif_disarm()). Once fully armed, re-enabling
interrupts will cause hvm_intr_en_notif_disarm() to make the end of interrupt callback. hvm_intr_en_notif_arm() and hvm_intr_en_notif_disarm() are called by [vmx,svm]_intr_assist().

3. Interrupt delivery policies

The existing hpet interrupt delivery is preserved. This includes vcpu round robin delivery used by Linux and broadcast delivery used by Windows.

There are two policies for interrupt delivery, one for Windows 2k8-64 and the other
for Linux. The Linux policy takes advantage of the (guest) Linux missed tick and offset
calculations and does not attempt to deliver the right number of interrupts. The Windows policy delivers the correct number of interrupts, even if sometimes much
closer to each other than the period. The policies are similar to those in vpt.c, though
there are some important differences.

Policies are selected with an HVMOP_set_param hypercall with index HVM_PARAM_TIMER_MODE.
Two new values are added, HVM_HPET_guest_computes_missed_ticks and HVM_HPET_guest_does_not_compute_missed_ticks. The reason that two new ones are added is that
in some guests (32bit Linux) a no-missed policy is needed for clock sources other than hpet
and a missed ticks policy for hpet. It was felt that there would be less confusion by simply
introducing the two hpet policies.

3.1. The missed ticks policy

The Linux clock interrupt handler for hpet calculates missed ticks and offset using the hpet
main counter. The algorithm works well when the time since the last interrupt is greater than
or equal to a period and poorly otherwise.

The missed ticks policy ensures that no two clock interrupts are delivered to the guest at
a time interval less than a period. A time stamp (hpet main counter value) is recorded (by a
callback registered with hvm_register_intr_en_notif) when Linux finishes handling the clock
interrupt. Then, ensuing interrupts are delivered to the vioapic only if the current main
counter value is a period greater than when the last interrupt was handled.

Tests showed a significant improvement in clock drift with end of interrupt time stamps
versus beginning of interrupt[1]. It is believed that the reason for the improvement
is that the clock interrupt handler goes for a spinlock and can be therefore delayed in its
processing. Furthermore, the main counter is read by the guest under the lock. The net
effect is that if we time stamp injection, we can get the difference in time between successive interrupt handler lock acquisitions to be less than the period.

3.2. The no-missed ticks policy

Windows 2k864 keeps very poor time with the missed ticks policy. So the no-missed ticks policy
was developed. In the no-missed ticks policy we deliver the correct number of interrupts,
even if they are spaced less than a period apart (when catching up).

Windows 2k864 uses a broadcast mode in the interrupt routing such that all vcpus get the clock interrupt. The best Windows drift performance was achieved when the
policy code ensured that all the previous interrupts (on the various vcpus) had been injected
before injecting the next interrupt to the vioapic..

The policy code works as follows. It uses the hvm_isa_irq_assert_cb() to record
the vcpus to be interrupted in h->hpet.pending_mask. Then, in the callback registered
with hvm_register_intr_en_notif() at post=1 time it clears the current vcpu in the pending_mask.
When the pending_mask is clear it decrements hpet.intr_pending_nr and if intr_pending_nr is still
non-zero posts another interrupt to the ioapic with hvm_isa_irq_assert_cb(). Intr_pending_nr is incremented in hpet_route_decision_not_missed_ticks().

The missed ticks policy intr_en_notif callback also uses the pending_mask method. So even though
Linux does not broadcast its interrupts, the code could handle it if it did. In this case the end of interrupt time stamp is made when the pending_mask is clear.

4. Live Migration

Live migration with hpet preserves the current offset of the guest clock with respect
to ntp. This is accomplished by migrating all of the state in the h->hpet data structure
in the usual way. The hp->mc_offset is recalculated on the receiving node so that the
guest sees a continuous hpet main counter.

Code as been added to xc_domain_save.c to send a small message after the domain context is sent. The contents of the message is the physical tsc timestamp, last_tsc,
read just before the message is sent. When the last_tsc message is received in xc_domain_restore.c,
another physical tsc timestamp, cur_tsc, is read. The two timestamps are loaded into the domain
structure as last_tsc_sender and first_tsc_receiver with hypercalls. Then xc_domain_hvm_setcontext
is called so that hpet_load has access to these time stamps. Hpet_load uses the timestamps
to account for the time spent saving and loading the domain context. With this technique,
the only neglected time is the time spent sending a small network message.

5. Test Results

Some recent test results are:

5.1 Linux 4u664 and Windows 2k864 load test.
Duration: 70 hours.
Test date: 6/2/08
Loads: usex -b48 on Linux; burn-in on Windows Guest vcpus: 8 for Linux; 2 for Windows
Hardware: 8 physical cpu AMD
Clock drift : Linux: .0012% Windows: .009%

5.2 Linux 4u664, Linux 4u464 , and Windows 2k864 no-load test
Duration: 23 hours.
Test date: 6/3/08
Loads: none
Guest vcpus: 8 for each Linux; 2 for Windows
Hardware: 4 physical cpu AMD
Clock drift : Linux: .033% Windows: .019%

6. Relation to recent work in xen-unstable

There is a similarity between hvm_get_guest_time() in xen-unstable and read_64_main_counter()
in this code. However, read_64_main_counter() is more tuned to the needs of hpet.c. It has no
"set" operation, only the get. It isolates the mode, physical or simulated, in read_64_main_counter()
itself. It uses no vcpu or domain state as it is a physical entity, in either mode. And it provides a real
physical mode for every read for those applications that desire this.

7. Conclusion

The virtual hpet is improved by this patch in terms of accuracy and monotonicity.
Tests performed to date verify this and more testing is under way.

8. Future Work

Testing with Windows Vista will be performed soon. The reason for accuracy variations
on different platforms using the physical hpet device will be investigated. Additional overhead measurements on simulated vs physical hpet mode will be made.

Footnotes:

1. I don't recall the accuracy improvement with end of interrupt stamping, but it was
significant, perhaps better than two to one improvement. It would be a very simple matter
to re-measure the improvement as the facility can call back at injection time as well.

Signed-off-by Dave Winchell <[email protected]>

<mailto:[email protected]>

Signed-off-by Ben Guthro <[email protected]>

<mailto:[email protected]>

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

1. Introduction

This patch improves the hpet based guest clock in terms of drift and monotonicity.
Prior to this work the drift with hpet was greater than 2%, far above the .05% limit
for ntp to synchronize. With this code, the drift ranges from .001% to .0033% depending
on guest and physical platform.

Using hpet allows guest operating systems to provide monotonic time to their applications. Time sources other than hpet are not monotonic because of their reliance on tsc, which is not synchronized across physical processors.

Windows 2k864 and many Linux guests are supported with two policies, one for guests
that handle missed clock interrupts and the other for guests that require the correct number of interrupts.

Guests may use hpet for the timing source even if the physical platform has no visible
hpet. Migration is supported between physical machines which differ in physical
hpet visibility.

Most of the changes are in hpet.c. Two general facilities are added to track interrupt
progress. The ideas here and the facilities would be useful in vpt.c, for other time
sources, though no attempt is made here to improve vpt.c.

The following sections discuss hpet dependencies, interrupt delivery policies, live migration,
test results, and relation to recent work with monotonic time.

2. Virtual Hpet dependencies

The virtual hpet depends on the ability to read the physical or simulated (see discussion below) hpet. For timekeeping, the virtual hpet also depends on two new interrupt notification facilities to implement its policies for interrupt delivery.

2.1. Two modes of low-level hpet main counter reads.

In this implementation, the virtual hpet reads with read_64_main_counter(), exported by
time.c, either the real physical hpet main counter register directly or a "simulated"
hpet main counter.

The simulated mode uses a monotonic version of get_s_time() (NOW()), where the last
time value is returned whenever the current time value is less than the last time
value. In simulated mode, since it is layered on s_time, the underlying hardware
can be hpet or some other device. The frequency of the main counter in simulated
mode is the same as the standard physical hpet frequency, allowing live migration
between nodes that are configured differently.

If the physical platform does not have an hpet device, or if xen is configured not
to use the device, then the simulated method is used. If there is a physical hpet device,
and xen has initialized it, then either simulated or physical mode can be used.
This is governed by a boot time option, hpet-avoid. Setting this option to 1 gives the
simulated mode and 0 the physical mode. The default is physical mode.

A disadvantage of the physical mode is that may take longer to read the device than in simulated mode. On some platforms the cost is about the same (less than 250 nsec) for
physical and simulated modes, while on others physical cost is much higher than simulated.
A disadvantage of the simulated mode is that it can return the same value for the counter in consecutive calls.

2.2. Interrupt notification facilities.

Two interrupt notification facilities are introduced, one is hvm_isa_irq_assert_cb()
and the other hvm_register_intr_en_notif().

The vhpet uses hvm_isa_irq_assert_cb to deliver interrupts to the vioapic. hvm_isa_irq_assert_cb allows a callback to be passed along to vioapic_deliver()
and this callback is called with a mask of the vcpus which will get the interrupt. This callback is made before any vcpus receive an interrupt.

Vhpet uses hvm_register_intr_en_notif() to register a handler for a particular vector that will be called when that vector is injected in [vmx,svm]_intr_assist()
and also when the guest finishes handling the interrupt. Here finished is defined
as the point when the guest re-enables interrupts or lowers the tpr value. EOI is not used as the end of interrupt as this is sometimes returned before the interrupt handler has done its work. A flag is passed to the handler indicating
whether this is the injection point (post = 1) or the interrupt finished (post = 0) point.
The need for the finished point callback is discussed in the missed ticks policy section.

To prevent a possible early trigger of the finished callback, intr_en_notif logic
has a two stage arm, the first at injection (hvm_intr_en_notif_arm()) and the second when
interrupts are seen to be disabled (hvm_intr_en_notif_disarm()). Once fully armed, re-enabling
interrupts will cause hvm_intr_en_notif_disarm() to make the end of interrupt callback. hvm_intr_en_notif_arm() and hvm_intr_en_notif_disarm() are called by [vmx,svm]_intr_assist().

3. Interrupt delivery policies

The existing hpet interrupt delivery is preserved. This includes vcpu round robin delivery used by Linux and broadcast delivery used by Windows.

There are two policies for interrupt delivery, one for Windows 2k8-64 and the other
for Linux. The Linux policy takes advantage of the (guest) Linux missed tick and offset
calculations and does not attempt to deliver the right number of interrupts. The Windows policy delivers the correct number of interrupts, even if sometimes much
closer to each other than the period. The policies are similar to those in vpt.c, though
there are some important differences.

Policies are selected with an HVMOP_set_param hypercall with index HVM_PARAM_TIMER_MODE.
Two new values are added, HVM_HPET_guest_computes_missed_ticks and HVM_HPET_guest_does_not_compute_missed_ticks. The reason that two new ones are added is that
in some guests (32bit Linux) a no-missed policy is needed for clock sources other than hpet
and a missed ticks policy for hpet. It was felt that there would be less confusion by simply
introducing the two hpet policies.

3.1. The missed ticks policy

The Linux clock interrupt handler for hpet calculates missed ticks and offset using the hpet
main counter. The algorithm works well when the time since the last interrupt is greater than
or equal to a period and poorly otherwise.

The missed ticks policy ensures that no two clock interrupts are delivered to the guest at
a time interval less than a period. A time stamp (hpet main counter value) is recorded (by a
callback registered with hvm_register_intr_en_notif) when Linux finishes handling the clock
interrupt. Then, ensuing interrupts are delivered to the vioapic only if the current main
counter value is a period greater than when the last interrupt was handled.

Tests showed a significant improvement in clock drift with end of interrupt time stamps
versus beginning of interrupt[1]. It is believed that the reason for the improvement
is that the clock interrupt handler goes for a spinlock and can be therefore delayed in its
processing. Furthermore, the main counter is read by the guest under the lock. The net
effect is that if we time stamp injection, we can get the difference in time between successive interrupt handler lock acquisitions to be less than the period.

3.2. The no-missed ticks policy

Windows 2k864 keeps very poor time with the missed ticks policy. So the no-missed ticks policy
was developed. In the no-missed ticks policy we deliver the correct number of interrupts,
even if they are spaced less than a period apart (when catching up).

Windows 2k864 uses a broadcast mode in the interrupt routing such that all vcpus get the clock interrupt. The best Windows drift performance was achieved when the
policy code ensured that all the previous interrupts (on the various vcpus) had been injected
before injecting the next interrupt to the vioapic..

The policy code works as follows. It uses the hvm_isa_irq_assert_cb() to record
the vcpus to be interrupted in h->hpet.pending_mask. Then, in the callback registered
with hvm_register_intr_en_notif() at post=1 time it clears the current vcpu in the pending_mask.
When the pending_mask is clear it decrements hpet.intr_pending_nr and if intr_pending_nr is still
non-zero posts another interrupt to the ioapic with hvm_isa_irq_assert_cb(). Intr_pending_nr is incremented in hpet_route_decision_not_missed_ticks().

The missed ticks policy intr_en_notif callback also uses the pending_mask method. So even though
Linux does not broadcast its interrupts, the code could handle it if it did. In this case the end of interrupt time stamp is made when the pending_mask is clear.

4. Live Migration

Live migration with hpet preserves the current offset of the guest clock with respect
to ntp. This is accomplished by migrating all of the state in the h->hpet data structure
in the usual way. The hp->mc_offset is recalculated on the receiving node so that the
guest sees a continuous hpet main counter.

Code as been added to xc_domain_save.c to send a small message after the domain context is sent. The contents of the message is the physical tsc timestamp, last_tsc,
read just before the message is sent. When the last_tsc message is received in xc_domain_restore.c,
another physical tsc timestamp, cur_tsc, is read. The two timestamps are loaded into the domain
structure as last_tsc_sender and first_tsc_receiver with hypercalls. Then xc_domain_hvm_setcontext
is called so that hpet_load has access to these time stamps. Hpet_load uses the timestamps
to account for the time spent saving and loading the domain context. With this technique,
the only neglected time is the time spent sending a small network message.

5. Test Results

Some recent test results are:

5.1 Linux 4u664 and Windows 2k864 load test.
Duration: 70 hours.
Test date: 6/2/08
Loads: usex -b48 on Linux; burn-in on Windows Guest vcpus: 8 for Linux; 2 for Windows
Hardware: 8 physical cpu AMD
Clock drift : Linux: .0012% Windows: .009%

5.2 Linux 4u664, Linux 4u464 , and Windows 2k864 no-load test
Duration: 23 hours.
Test date: 6/3/08
Loads: none
Guest vcpus: 8 for each Linux; 2 for Windows
Hardware: 4 physical cpu AMD
Clock drift : Linux: .033% Windows: .019%

6. Relation to recent work in xen-unstable

There is a similarity between hvm_get_guest_time() in xen-unstable and read_64_main_counter()
in this code. However, read_64_main_counter() is more tuned to the needs of hpet.c. It has no
"set" operation, only the get. It isolates the mode, physical or simulated, in read_64_main_counter()
itself. It uses no vcpu or domain state as it is a physical entity, in either mode. And it provides a real
physical mode for every read for those applications that desire this.

7. Conclusion

The virtual hpet is improved by this patch in terms of accuracy and monotonicity.
Tests performed to date verify this and more testing is under way.

8. Future Work

Testing with Windows Vista will be performed soon. The reason for accuracy variations
on different platforms using the physical hpet device will be investigated. Additional overhead measurements on simulated vs physical hpet mode will be made.

Footnotes:

1. I don't recall the accuracy improvement with end of interrupt stamping, but it was
significant, perhaps better than two to one improvement. It would be a very simple matter
to re-measure the improvement as the facility can call back at injection time as well.

Signed-off-by Dave Winchell <[email protected]>

<mailto:[email protected]>

Signed-off-by Ben Guthro <[email protected]>

<mailto:[email protected]>

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

This seems to break the save/restore format (in at least two places)...

S.

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Hi Dave and Ben --

When running tests on xen-unstable (without your patch), please ensure that hpet=1 is set in the hvm config and also I think that when hpet is the clocksource on RHEL4-32, the clock IS resilient to missed ticks so timer_mode should be 2 (vs when pit is the clocksource on RHEL4-32, all clock ticks must be delivered and so timer_mode should be 0).

Per http://lists.xensource.com/archives/html/xen-devel/2008-06/msg00098.html it's my intent to clean this up, but I won't get to it until next week.

Thanks,
Dan

-----Original Message-----
*From:* [email protected] [mailto:[email protected]]*On Behalf Of *Dave Winchell
*Sent:* Friday, June 06, 2008 4:46 AM
*To:* Keir Fraser; Ben Guthro; xen-devel
*Cc:* [email protected]; Dave Winchell *Subject:* RE: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Keir,

I think the changes are required. We'll run some tests today today so that we have some data to talk about.

-Dave

-----Original Message-----
From: [email protected] on behalf of Keir Fraser
Sent: Fri 6/6/2008 4:58 AM
To: Ben Guthro; xen-devel
Cc: [email protected]
Subject: Re: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Are these patches needed now the timers are built on Xen system time rather
than host TSC? Dan has reported much better time-keeping with his patch
checked in, and it¹s for sure a lot less invasive than this patchset.

-- Keir

On 5/6/08 15:59, "Ben Guthro" <[email protected]> wrote:

>
> 1. Introduction
>
> This patch improves the hpet based guest clock in terms of drift and > monotonicity.
> Prior to this work the drift with hpet was greater than 2%, far above the .05%
> limit
> for ntp to synchronize. With this code, the drift ranges from .001% to .0033%
> depending
> on guest and physical platform.
>
> Using hpet allows guest operating systems to provide monotonic time to their
> applications. Time sources other than hpet are not monotonic because > of their reliance on tsc, which is not synchronized across physical > processors.
>
> Windows 2k864 and many Linux guests are supported with two policies, one for
> guests
> that handle missed clock interrupts and the other for guests that require the
> correct number of interrupts.
>
> Guests may use hpet for the timing source even if the physical platform has no
> visible
> hpet. Migration is supported between physical machines which differ in
> physical
> hpet visibility.
>
> Most of the changes are in hpet.c. Two general facilities are added to track
> interrupt
> progress. The ideas here and the facilities would be useful in vpt.c, for
> other time
> sources, though no attempt is made here to improve vpt.c. >
> The following sections discuss hpet dependencies, interrupt delivery policies,
> live migration,
> test results, and relation to recent work with monotonic time. >
>
> 2. Virtual Hpet dependencies
>
> The virtual hpet depends on the ability to read the physical or simulated
> (see discussion below) hpet. For timekeeping, the virtual hpet also depends
> on two new interrupt notification facilities to implement its policies for
> interrupt delivery.
>
> 2.1. Two modes of low-level hpet main counter reads.
>
> In this implementation, the virtual hpet reads with read_64_main_counter(),
> exported by
> time.c, either the real physical hpet main counter register directly or a
> "simulated"
> hpet main counter.
>
> The simulated mode uses a monotonic version of get_s_time() (NOW()), where the
> last
> time value is returned whenever the current time value is less than the last
> time
> value. In simulated mode, since it is layered on s_time, the underlying
> hardware
> can be hpet or some other device. The frequency of the main counter in
> simulated
> mode is the same as the standard physical hpet frequency, allowing live
> migration
> between nodes that are configured differently.
>
> If the physical platform does not have an hpet device, or if xen is configured
> not
> to use the device, then the simulated method is used. If there is a physical
> hpet device,
> and xen has initialized it, then either simulated or physical mode can be
> used.
> This is governed by a boot time option, hpet-avoid. Setting this option to 1
> gives the
> simulated mode and 0 the physical mode. The default is physical mode.
>
> A disadvantage of the physical mode is that may take longer to read the device
> than in simulated mode. On some platforms the cost is about the same (less
> than 250 nsec) for
> physical and simulated modes, while on others physical cost is much higher
> than simulated.
> A disadvantage of the simulated mode is that it can return the same value
> for the counter in consecutive calls.
>
> 2.2. Interrupt notification facilities.
>
> Two interrupt notification facilities are introduced, one is > hvm_isa_irq_assert_cb()
> and the other hvm_register_intr_en_notif().
>
> The vhpet uses hvm_isa_irq_assert_cb to deliver interrupts to the vioapic.
> hvm_isa_irq_assert_cb allows a callback to be passed along to > vioapic_deliver()
> and this callback is called with a mask of the vcpus which will get the
> interrupt. This callback is made before any vcpus receive an interrupt.
>
> Vhpet uses hvm_register_intr_en_notif() to register a handler for a particular
> vector that will be called when that vector is injected in > [vmx,svm]_intr_assist()
> and also when the guest finishes handling the interrupt. Here finished is
> defined
> as the point when the guest re-enables interrupts or lowers the tpr value.
> EOI is not used as the end of interrupt as this is sometimes returned before
> the interrupt handler has done its work. A flag is passed to the handler
> indicating
> whether this is the injection point (post = 1) or the interrupt finished (post
> = 0) point.
> The need for the finished point callback is discussed in the missed ticks
> policy section.
>
> To prevent a possible early trigger of the finished callback, intr_en_notif
> logic
> has a two stage arm, the first at injection (hvm_intr_en_notif_arm()) and the
> second when
> interrupts are seen to be disabled (hvm_intr_en_notif_disarm()). Once fully
> armed, re-enabling
> interrupts will cause hvm_intr_en_notif_disarm() to make the end of interrupt
> callback. hvm_intr_en_notif_arm() and hvm_intr_en_notif_disarm() are called by
> [vmx,svm]_intr_assist().
>
> 3. Interrupt delivery policies
>
> The existing hpet interrupt delivery is preserved. This includes > vcpu round robin delivery used by Linux and broadcast delivery used by
> Windows.
>
> There are two policies for interrupt delivery, one for Windows 2k8-64 and the
> other
> for Linux. The Linux policy takes advantage of the (guest) Linux missed tick
> and offset
> calculations and does not attempt to deliver the right number of interrupts.
> The Windows policy delivers the correct number of interrupts, even if
> sometimes much
> closer to each other than the period. The policies are similar to those in
> vpt.c, though
> there are some important differences.
>
> Policies are selected with an HVMOP_set_param hypercall with index > HVM_PARAM_TIMER_MODE.
> Two new values are added, HVM_HPET_guest_computes_missed_ticks and > HVM_HPET_guest_does_not_compute_missed_ticks. The reason that two new ones
> are added is that
> in some guests (32bit Linux) a no-missed policy is needed for clock sources
> other than hpet
> and a missed ticks policy for hpet. It was felt that there would be less
> confusion by simply
> introducing the two hpet policies.
>
> 3.1. The missed ticks policy
>
> The Linux clock interrupt handler for hpet calculates missed ticks and offset
> using the hpet
> main counter. The algorithm works well when the time since the last interrupt
> is greater than
> or equal to a period and poorly otherwise.
>
> The missed ticks policy ensures that no two clock interrupts are delivered to
> the guest at
> a time interval less than a period. A time stamp (hpet main counter value) is
> recorded (by a
> callback registered with hvm_register_intr_en_notif) when Linux finishes
> handling the clock
> interrupt. Then, ensuing interrupts are delivered to the vioapic only if the
> current main
> counter value is a period greater than when the last interrupt was handled.
>
> Tests showed a significant improvement in clock drift with end of interrupt
> time stamps
> versus beginning of interrupt[1]. It is believed that the reason for the
> improvement
> is that the clock interrupt handler goes for a spinlock and can be therefore
> delayed in its
> processing. Furthermore, the main counter is read by the guest under the lock.
> The net
> effect is that if we time stamp injection, we can get the difference in time
> between successive interrupt handler lock acquisitions to be less than the
> period.
>
> 3.2. The no-missed ticks policy
>
> Windows 2k864 keeps very poor time with the missed ticks policy. So the
> no-missed ticks policy
> was developed. In the no-missed ticks policy we deliver the correct number of
> interrupts,
> even if they are spaced less than a period apart (when catching up). >
> Windows 2k864 uses a broadcast mode in the interrupt routing such that
> all vcpus get the clock interrupt. The best Windows drift performance was
> achieved when the
> policy code ensured that all the previous interrupts (on the various vcpus)
> had been injected
> before injecting the next interrupt to the vioapic..
>
> The policy code works as follows. It uses the hvm_isa_irq_assert_cb() to
> record
> the vcpus to be interrupted in h->hpet.pending_mask. Then, in the callback
> registered
> with hvm_register_intr_en_notif() at post=1 time it clears the current vcpu in
> the pending_mask.
> When the pending_mask is clear it decrements hpet.intr_pending_nr and if
> intr_pending_nr is still
> non-zero posts another interrupt to the ioapic with hvm_isa_irq_assert_cb().
> Intr_pending_nr is incremented in hpet_route_decision_not_missed_ticks().
>
> The missed ticks policy intr_en_notif callback also uses the pending_mask
> method. So even though
> Linux does not broadcast its interrupts, the code could handle it if it did.
> In this case the end of interrupt time stamp is made when the pending_mask is
> clear.
>
> 4. Live Migration
>
> Live migration with hpet preserves the current offset of the guest clock with
> respect
> to ntp. This is accomplished by migrating all of the state in the h->hpet data
> structure
> in the usual way. The hp->mc_offset is recalculated on the receiving node so
> that the
> guest sees a continuous hpet main counter.
>
> Code as been added to xc_domain_save.c to send a small message after the
> domain context is sent. The contents of the message is the physical tsc
> timestamp, last_tsc,
> read just before the message is sent. When the last_tsc message is received in
> xc_domain_restore.c,
> another physical tsc timestamp, cur_tsc, is read. The two timestamps are
> loaded into the domain
> structure as last_tsc_sender and first_tsc_receiver with hypercalls. Then
> xc_domain_hvm_setcontext
> is called so that hpet_load has access to these time stamps. Hpet_load uses
> the timestamps
> to account for the time spent saving and loading the domain context. With this
> technique,
> the only neglected time is the time spent sending a small network message.
>
> 5. Test Results
>
> Some recent test results are:
>
> 5.1 Linux 4u664 and Windows 2k864 load test.
> Duration: 70 hours.
> Test date: 6/2/08
> Loads: usex -b48 on Linux; burn-in on Windows
> Guest vcpus: 8 for Linux; 2 for Windows
> Hardware: 8 physical cpu AMD
> Clock drift : Linux: .0012% Windows: .009%
>
> 5.2 Linux 4u664, Linux 4u464 , and Windows 2k864 no-load test
> Duration: 23 hours.
> Test date: 6/3/08
> Loads: none
> Guest vcpus: 8 for each Linux; 2 for Windows
> Hardware: 4 physical cpu AMD
> Clock drift : Linux: .033% Windows: .019%
>
> 6. Relation to recent work in xen-unstable
>
> There is a similarity between hvm_get_guest_time() in xen-unstable and
> read_64_main_counter()
> in this code. However, read_64_main_counter() is more tuned to the needs of
> hpet.c. It has no
> "set" operation, only the get. It isolates the mode, physical or simulated, in
> read_64_main_counter()
> itself. It uses no vcpu or domain state as it is a physical entity, in either
> mode. And it provides a real
> physical mode for every read for those applications that desire this.
>
> 7. Conclusion
>
> The virtual hpet is improved by this patch in terms of accuracy and > monotonicity.
> Tests performed to date verify this and more testing is under way. >
> 8. Future Work
>
> Testing with Windows Vista will be performed soon. The reason for accuracy
> variations
> on different platforms using the physical hpet device will be investigated.
> Additional overhead measurements on simulated vs physical hpet mode will be
> made.
>
> Footnotes:
>
> 1. I don't recall the accuracy improvement with end of interrupt stamping, but
> it was
> significant, perhaps better than two to one improvement. It would be a very
> simple matter
> to re-measure the improvement as the facility can call back at injection time
> as well.
>
>
> Signed-off-by: Dave Winchell <[email protected]> > <mailto:[email protected]>
> Signed-off-by: Ben Guthro <[email protected]>
> <mailto:[email protected]>
>
>
> _______________________________________________
> Xen-devel mailing list
> [email protected]
> http://lists.xensource.com/xen-devel

Hi Dave and Ben --

When running tests on xen-unstable (without your patch), please ensure that hpet=1 is set in the hvm config and also I think that when hpet is the clocksource on RHEL4-32, the clock IS resilient to missed ticks so timer_mode should be 2 (vs when pit is the clocksource on RHEL4-32, all clock ticks must be delivered and so timer_mode should be 0).

Per http://lists.xensource.com/archives/html/xen-devel/2008-06/msg00098.html it's my intent to clean this up, but I won't get to it until next week.

Thanks,
Dan

-----Original Message-----
*From:* [email protected] [mailto:[email protected]]*On Behalf Of *Dave Winchell
*Sent:* Friday, June 06, 2008 4:46 AM
*To:* Keir Fraser; Ben Guthro; xen-devel
*Cc:* [email protected]; Dave Winchell *Subject:* RE: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Keir,

I think the changes are required. We'll run some tests today today so that we have some data to talk about.

-Dave

-----Original Message-----
From: [email protected] on behalf of Keir Fraser
Sent: Fri 6/6/2008 4:58 AM
To: Ben Guthro; xen-devel
Cc: [email protected]
Subject: Re: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Are these patches needed now the timers are built on Xen system time rather
than host TSC? Dan has reported much better time-keeping with his patch
checked in, and it¹s for sure a lot less invasive than this patchset.

-- Keir

On 5/6/08 15:59, "Ben Guthro" <[email protected]> wrote:

>
> 1. Introduction
>
> This patch improves the hpet based guest clock in terms of drift and > monotonicity.
> Prior to this work the drift with hpet was greater than 2%, far above the .05%
> limit
> for ntp to synchronize. With this code, the drift ranges from .001% to .0033%
> depending
> on guest and physical platform.
>
> Using hpet allows guest operating systems to provide monotonic time to their
> applications. Time sources other than hpet are not monotonic because > of their reliance on tsc, which is not synchronized across physical > processors.
>
> Windows 2k864 and many Linux guests are supported with two policies, one for
> guests
> that handle missed clock interrupts and the other for guests that require the
> correct number of interrupts.
>
> Guests may use hpet for the timing source even if the physical platform has no
> visible
> hpet. Migration is supported between physical machines which differ in
> physical
> hpet visibility.
>
> Most of the changes are in hpet.c. Two general facilities are added to track
> interrupt
> progress. The ideas here and the facilities would be useful in vpt.c, for
> other time
> sources, though no attempt is made here to improve vpt.c. >
> The following sections discuss hpet dependencies, interrupt delivery policies,
> live migration,
> test results, and relation to recent work with monotonic time. >
>
> 2. Virtual Hpet dependencies
>
> The virtual hpet depends on the ability to read the physical or simulated
> (see discussion below) hpet. For timekeeping, the virtual hpet also depends
> on two new interrupt notification facilities to implement its policies for
> interrupt delivery.
>
> 2.1. Two modes of low-level hpet main counter reads.
>
> In this implementation, the virtual hpet reads with read_64_main_counter(),
> exported by
> time.c, either the real physical hpet main counter register directly or a
> "simulated"
> hpet main counter.
>
> The simulated mode uses a monotonic version of get_s_time() (NOW()), where the
> last
> time value is returned whenever the current time value is less than the last
> time
> value. In simulated mode, since it is layered on s_time, the underlying
> hardware
> can be hpet or some other device. The frequency of the main counter in
> simulated
> mode is the same as the standard physical hpet frequency, allowing live
> migration
> between nodes that are configured differently.
>
> If the physical platform does not have an hpet device, or if xen is configured
> not
> to use the device, then the simulated method is used. If there is a physical
> hpet device,
> and xen has initialized it, then either simulated or physical mode can be
> used.
> This is governed by a boot time option, hpet-avoid. Setting this option to 1
> gives the
> simulated mode and 0 the physical mode. The default is physical mode.
>
> A disadvantage of the physical mode is that may take longer to read the device
> than in simulated mode. On some platforms the cost is about the same (less
> than 250 nsec) for
> physical and simulated modes, while on others physical cost is much higher
> than simulated.
> A disadvantage of the simulated mode is that it can return the same value
> for the counter in consecutive calls.
>
> 2.2. Interrupt notification facilities.
>
> Two interrupt notification facilities are introduced, one is > hvm_isa_irq_assert_cb()
> and the other hvm_register_intr_en_notif().
>
> The vhpet uses hvm_isa_irq_assert_cb to deliver interrupts to the vioapic.
> hvm_isa_irq_assert_cb allows a callback to be passed along to > vioapic_deliver()
> and this callback is called with a mask of the vcpus which will get the
> interrupt. This callback is made before any vcpus receive an interrupt.
>
> Vhpet uses hvm_register_intr_en_notif() to register a handler for a particular
> vector that will be called when that vector is injected in > [vmx,svm]_intr_assist()
> and also when the guest finishes handling the interrupt. Here finished is
> defined
> as the point when the guest re-enables interrupts or lowers the tpr value.
> EOI is not used as the end of interrupt as this is sometimes returned before
> the interrupt handler has done its work. A flag is passed to the handler
> indicating
> whether this is the injection point (post = 1) or the interrupt finished (post
> = 0) point.
> The need for the finished point callback is discussed in the missed ticks
> policy section.
>
> To prevent a possible early trigger of the finished callback, intr_en_notif
> logic
> has a two stage arm, the first at injection (hvm_intr_en_notif_arm()) and the
> second when
> interrupts are seen to be disabled (hvm_intr_en_notif_disarm()). Once fully
> armed, re-enabling
> interrupts will cause hvm_intr_en_notif_disarm() to make the end of interrupt
> callback. hvm_intr_en_notif_arm() and hvm_intr_en_notif_disarm() are called by
> [vmx,svm]_intr_assist().
>
> 3. Interrupt delivery policies
>
> The existing hpet interrupt delivery is preserved. This includes > vcpu round robin delivery used by Linux and broadcast delivery used by
> Windows.
>
> There are two policies for interrupt delivery, one for Windows 2k8-64 and the
> other
> for Linux. The Linux policy takes advantage of the (guest) Linux missed tick
> and offset
> calculations and does not attempt to deliver the right number of interrupts.
> The Windows policy delivers the correct number of interrupts, even if
> sometimes much
> closer to each other than the period. The policies are similar to those in
> vpt.c, though
> there are some important differences.
>
> Policies are selected with an HVMOP_set_param hypercall with index > HVM_PARAM_TIMER_MODE.
> Two new values are added, HVM_HPET_guest_computes_missed_ticks and > HVM_HPET_guest_does_not_compute_missed_ticks. The reason that two new ones
> are added is that
> in some guests (32bit Linux) a no-missed policy is needed for clock sources
> other than hpet
> and a missed ticks policy for hpet. It was felt that there would be less
> confusion by simply
> introducing the two hpet policies.
>
> 3.1. The missed ticks policy
>
> The Linux clock interrupt handler for hpet calculates missed ticks and offset
> using the hpet
> main counter. The algorithm works well when the time since the last interrupt
> is greater than
> or equal to a period and poorly otherwise.
>
> The missed ticks policy ensures that no two clock interrupts are delivered to
> the guest at
> a time interval less than a period. A time stamp (hpet main counter value) is
> recorded (by a
> callback registered with hvm_register_intr_en_notif) when Linux finishes
> handling the clock
> interrupt. Then, ensuing interrupts are delivered to the vioapic only if the
> current main
> counter value is a period greater than when the last interrupt was handled.
>
> Tests showed a significant improvement in clock drift with end of interrupt
> time stamps
> versus beginning of interrupt[1]. It is believed that the reason for the
> improvement
> is that the clock interrupt handler goes for a spinlock and can be therefore
> delayed in its
> processing. Furthermore, the main counter is read by the guest under the lock.
> The net
> effect is that if we time stamp injection, we can get the difference in time
> between successive interrupt handler lock acquisitions to be less than the
> period.
>
> 3.2. The no-missed ticks policy
>
> Windows 2k864 keeps very poor time with the missed ticks policy. So the
> no-missed ticks policy
> was developed. In the no-missed ticks policy we deliver the correct number of
> interrupts,
> even if they are spaced less than a period apart (when catching up). >
> Windows 2k864 uses a broadcast mode in the interrupt routing such that
> all vcpus get the clock interrupt. The best Windows drift performance was
> achieved when the
> policy code ensured that all the previous interrupts (on the various vcpus)
> had been injected
> before injecting the next interrupt to the vioapic..
>
> The policy code works as follows. It uses the hvm_isa_irq_assert_cb() to
> record
> the vcpus to be interrupted in h->hpet.pending_mask. Then, in the callback
> registered
> with hvm_register_intr_en_notif() at post=1 time it clears the current vcpu in
> the pending_mask.
> When the pending_mask is clear it decrements hpet.intr_pending_nr and if
> intr_pending_nr is still
> non-zero posts another interrupt to the ioapic with hvm_isa_irq_assert_cb().
> Intr_pending_nr is incremented in hpet_route_decision_not_missed_ticks().
>
> The missed ticks policy intr_en_notif callback also uses the pending_mask
> method. So even though
> Linux does not broadcast its interrupts, the code could handle it if it did.
> In this case the end of interrupt time stamp is made when the pending_mask is
> clear.
>
> 4. Live Migration
>
> Live migration with hpet preserves the current offset of the guest clock with
> respect
> to ntp. This is accomplished by migrating all of the state in the h->hpet data
> structure
> in the usual way. The hp->mc_offset is recalculated on the receiving node so
> that the
> guest sees a continuous hpet main counter.
>
> Code as been added to xc_domain_save.c to send a small message after the
> domain context is sent. The contents of the message is the physical tsc
> timestamp, last_tsc,
> read just before the message is sent. When the last_tsc message is received in
> xc_domain_restore.c,
> another physical tsc timestamp, cur_tsc, is read. The two timestamps are
> loaded into the domain
> structure as last_tsc_sender and first_tsc_receiver with hypercalls. Then
> xc_domain_hvm_setcontext
> is called so that hpet_load has access to these time stamps. Hpet_load uses
> the timestamps
> to account for the time spent saving and loading the domain context. With this
> technique,
> the only neglected time is the time spent sending a small network message.
>
> 5. Test Results
>
> Some recent test results are:
>
> 5.1 Linux 4u664 and Windows 2k864 load test.
> Duration: 70 hours.
> Test date: 6/2/08
> Loads: usex -b48 on Linux; burn-in on Windows
> Guest vcpus: 8 for Linux; 2 for Windows
> Hardware: 8 physical cpu AMD
> Clock drift : Linux: .0012% Windows: .009%
>
> 5.2 Linux 4u664, Linux 4u464 , and Windows 2k864 no-load test
> Duration: 23 hours.
> Test date: 6/3/08
> Loads: none
> Guest vcpus: 8 for each Linux; 2 for Windows
> Hardware: 4 physical cpu AMD
> Clock drift : Linux: .033% Windows: .019%
>
> 6. Relation to recent work in xen-unstable
>
> There is a similarity between hvm_get_guest_time() in xen-unstable and
> read_64_main_counter()
> in this code. However, read_64_main_counter() is more tuned to the needs of
> hpet.c. It has no
> "set" operation, only the get. It isolates the mode, physical or simulated, in
> read_64_main_counter()
> itself. It uses no vcpu or domain state as it is a physical entity, in either
> mode. And it provides a real
> physical mode for every read for those applications that desire this.
>
> 7. Conclusion
>
> The virtual hpet is improved by this patch in terms of accuracy and > monotonicity.
> Tests performed to date verify this and more testing is under way. >
> 8. Future Work
>
> Testing with Windows Vista will be performed soon. The reason for accuracy
> variations
> on different platforms using the physical hpet device will be investigated.
> Additional overhead measurements on simulated vs physical hpet mode will be
> made.
>
> Footnotes:
>
> 1. I don't recall the accuracy improvement with end of interrupt stamping, but
> it was
> significant, perhaps better than two to one improvement. It would be a very
> simple matter
> to re-measure the improvement as the facility can call back at injection time
> as well.
>
>
> Signed-off-by: Dave Winchell <[email protected]> > <mailto:[email protected]>
> Signed-off-by: Ben Guthro <[email protected]>
> <mailto:[email protected]>
>
>
> _______________________________________________
> Xen-devel mailing list
> [email protected]
> http://lists.xensource.com/xen-devel

-----Original Message-----
From: Dave Winchell [mailto:[email protected]]
Sent: Friday, June 06, 2008 1:33 PM
To: [email protected]; Keir Fraser
Cc: Ben Guthro; xen-devel; Dave Winchell
Subject: Re: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Dan, Keir:

Preliminary tests results indicate an error of .1% for Linux 64 bit guests configured
for hpet with xen-unstable as is. As we have discussed many times, the ntp requirement is .05%.
Tests on the patch we just submitted for hpet have indicated errors of .0012%
on this platform under similar test conditions and .03% on other platforms.

Windows vista64 has an error of 11% using hpet with the xen-unstable bits.
In an overnight test with our hpet patch, the Windows vista error was .008%.

The tests are with two or three guests on a physical node, all under load, and with
the ratio of vcpus to phys cpus > 1.

I will continue to run tests over the next few days.

thanks,
Dave

Quoted 3 lines · Dan Magenheimer

Hi Dave and Ben --

When running tests on xen-unstable (without your patch),

please ensure

Quoted 2 lines

that hpet=1 is set in the hvm config and also I think that when hpet is the clocksource on RHEL4-32, the clock IS resilient to

missed ticks

Quoted 1 line

so timer_mode should be 2 (vs when pit is the clocksource

on RHEL4-32,

Quoted 4 lines

all clock ticks must be delivered and so timer_mode should be 0).

Per

http://lists.xensource.com/archives/html/xen-devel/2008-06/msg
00098.html it's

Quoted 8 lines

my intent to clean this up, but I won't get to it until next week.

Thanks,
Dan

    -----Original Message-----
    *From:* [email protected]
    [mailto:[email protected]]*On 

Behalf Of *Dave

Quoted 9 lines

Winchell
*Sent:* Friday, June 06, 2008 4:46 AM
*To:* Keir Fraser; Ben Guthro; xen-devel
*Cc:* [email protected]; Dave Winchell *Subject:* RE: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Keir,

I think the changes are required. We'll run some tests

today today so

Quoted 7 lines

that we have some data to talk about.

-Dave

-----Original Message-----
From: [email protected] on behalf

of Keir Fraser

Quoted 8 lines

Sent: Fri 6/6/2008 4:58 AM
To: Ben Guthro; xen-devel
Cc: [email protected]
Subject: Re: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Are these patches needed now the timers are built on Xen system time rather
than host TSC? Dan has reported much better

time-keeping with his

Quoted 2 lines

patch
checked in, and it¹s for sure a lot less invasive than

this patchset.

Quoted 10 lines

     -- Keir

    On 5/6/08 15:59, "Ben Guthro" <[email protected]> wrote:

    >
    > 1. Introduction
    >
    > This patch improves the hpet based guest clock in 

terms of drift and

Quoted 2 lines

Quoted 2 lines

monotonicity.
Prior to this work the drift with hpet was greater

than 2%, far

Quoted 10 lines

above the .05%
> limit
> for ntp to synchronize. With this code, the drift ranges from .001% to .0033%
> depending
> on guest and physical platform.
>
> Using hpet allows guest operating systems to provide monotonic time to their
> applications. Time sources other than hpet are not

monotonic because

Quoted 1 line

Quoted 1 line

of their reliance on tsc, which is not synchronized

across physical

Quoted 31 lines

Quoted 3 lines

processors.

Windows 2k864 and many Linux guests are supported with two

policies, one for
> guests
> that handle missed clock interrupts and the other for guests that require the
> correct number of interrupts.
>
> Guests may use hpet for the timing source even if the physical platform has no
> visible
> hpet. Migration is supported between physical machines which differ in
> physical
> hpet visibility.
>
> Most of the changes are in hpet.c. Two general facilities are added to track
> interrupt
> progress. The ideas here and the facilities would be useful in vpt.c, for
> other time
> sources, though no attempt is made here to improve vpt.c. >
> The following sections discuss hpet dependencies, interrupt delivery policies,
> live migration,
> test results, and relation to recent work with monotonic time. >
>

Quoted 3 lines

> 2. Virtual Hpet dependencies
>
> The virtual hpet depends on the ability to read the

physical or

Quoted 2 lines

simulated
> (see discussion below) hpet. For timekeeping, the

virtual hpet

Quoted 32 lines

also depends
> on two new interrupt notification facilities to implement its policies for
> interrupt delivery.
>
> 2.1. Two modes of low-level hpet main counter reads. >
> In this implementation, the virtual hpet reads with read_64_main_counter(),
> exported by
> time.c, either the real physical hpet main counter register directly or a
> "simulated"
> hpet main counter.
>
> The simulated mode uses a monotonic version of get_s_time() (NOW()), where the
> last
> time value is returned whenever the current time value is less than the last
> time
> value. In simulated mode, since it is layered on s_time, the underlying
> hardware
> can be hpet or some other device. The frequency of the main counter in
> simulated
> mode is the same as the standard physical hpet frequency, allowing live
> migration
> between nodes that are configured differently.
>

Quoted 1 line

> If the physical platform does not have an hpet

device, or if xen

Quoted 9 lines

is configured
> not
> to use the device, then the simulated method is used. If there is a physical
> hpet device,
> and xen has initialized it, then either simulated or physical mode can be
> used.
> This is governed by a boot time option, hpet-avoid.

Setting this

Quoted 3 lines

option to 1
> gives the
> simulated mode and 0 the physical mode. The default

is physical

Quoted 5 lines

mode.
>
> A disadvantage of the physical mode is that may take longer to read the device
> than in simulated mode. On some platforms the cost is

about the

Quoted 15 lines

same (less
> than 250 nsec) for
> physical and simulated modes, while on others physical cost is much higher
> than simulated.
> A disadvantage of the simulated mode is that it can return the same value
> for the counter in consecutive calls.
>
> 2.2. Interrupt notification facilities.
>
> Two interrupt notification facilities are introduced, one is > hvm_isa_irq_assert_cb()
> and the other hvm_register_intr_en_notif().
>

Quoted 5 lines

> The vhpet uses hvm_isa_irq_assert_cb to deliver interrupts to the vioapic.
> hvm_isa_irq_assert_cb allows a callback to be passed along to > vioapic_deliver()
> and this callback is called with a mask of the vcpus

which will

Quoted 4 lines

get the
> interrupt. This callback is made before any vcpus receive an interrupt.
>

Quoted 8 lines

> Vhpet uses hvm_register_intr_en_notif() to register a handler for a particular
> vector that will be called when that vector is injected in > [vmx,svm]_intr_assist()
> and also when the guest finishes handling the interrupt. Here finished is
> defined
> as the point when the guest re-enables interrupts or

lowers the

Quoted 4 lines

tpr value.
> EOI is not used as the end of interrupt as this is sometimes returned before
> the interrupt handler has done its work. A flag is

passed to the

Quoted 3 lines

handler
> indicating
> whether this is the injection point (post = 1) or the

interrupt

Quoted 6 lines

finished (post
> = 0) point.
> The need for the finished point callback is discussed in the missed ticks
> policy section.
>

Quoted 7 lines

> To prevent a possible early trigger of the finished callback, intr_en_notif
> logic
> has a two stage arm, the first at injection (hvm_intr_en_notif_arm()) and the
> second when
> interrupts are seen to be disabled

(hvm_intr_en_notif_disarm()).

Quoted 3 lines

Once fully
> armed, re-enabling
> interrupts will cause hvm_intr_en_notif_disarm() to

make the end

Quoted 2 lines

of interrupt
> callback. hvm_intr_en_notif_arm() and

hvm_intr_en_notif_disarm()

Quoted 6 lines

are called by
> [vmx,svm]_intr_assist().
>
> 3. Interrupt delivery policies
>
> The existing hpet interrupt delivery is preserved.

This includes

Quoted 8 lines

Quoted 1 line

vcpu round robin delivery used by Linux and broadcast delivery

used by
> Windows.
>
> There are two policies for interrupt delivery, one for Windows 2k8-64 and the
> other
> for Linux. The Linux policy takes advantage of the

(guest) Linux

Quoted 3 lines

missed tick
> and offset
> calculations and does not attempt to deliver the

right number of

Quoted 10 lines

interrupts.
> The Windows policy delivers the correct number of interrupts, even if
> sometimes much
> closer to each other than the period. The policies are similar to those in
> vpt.c, though
> there are some important differences.
>
> Policies are selected with an HVMOP_set_param

hypercall with index

Quoted 2 lines

Quoted 2 lines

HVM_PARAM_TIMER_MODE.
Two new values are added,

HVM_HPET_guest_computes_missed_ticks and

Quoted 7 lines

Quoted 1 line

HVM_HPET_guest_does_not_compute_missed_ticks. The reason that

two new ones
> are added is that
> in some guests (32bit Linux) a no-missed policy is needed for clock sources
> other than hpet
> and a missed ticks policy for hpet. It was felt that

there would

Quoted 14 lines

be less
> confusion by simply
> introducing the two hpet policies.
>
> 3.1. The missed ticks policy
>
> The Linux clock interrupt handler for hpet calculates missed ticks and offset
> using the hpet
> main counter. The algorithm works well when the time since the last interrupt
> is greater than
> or equal to a period and poorly otherwise.
>

Quoted 1 line

> The missed ticks policy ensures that no two clock

interrupts are

Quoted 6 lines

delivered to
> the guest at
> a time interval less than a period. A time stamp (hpet main counter value) is
> recorded (by a
> callback registered with hvm_register_intr_en_notif)

when Linux

Quoted 3 lines

finishes
> handling the clock
> interrupt. Then, ensuing interrupts are delivered to

the vioapic

Quoted 9 lines

only if the
> current main
> counter value is a period greater than when the last interrupt was handled.
>
> Tests showed a significant improvement in clock drift with end of interrupt
> time stamps
> versus beginning of interrupt[1]. It is believed that

the reason

Quoted 3 lines

for the
> improvement
> is that the clock interrupt handler goes for a

spinlock and can

Quoted 11 lines

be therefore
> delayed in its
> processing. Furthermore, the main counter is read by the guest under the lock.
> The net
> effect is that if we time stamp injection, we can get the difference in time
> between successive interrupt handler lock acquisitions to be less than the
> period.
>

Quoted 3 lines

> 3.2. The no-missed ticks policy
>
> Windows 2k864 keeps very poor time with the missed

ticks policy.

Quoted 6 lines

So the
> no-missed ticks policy
> was developed. In the no-missed ticks policy we deliver the correct number of
> interrupts,
> even if they are spaced less than a period apart

(when catching up).

Quoted 50 lines

Quoted 2 lines

Windows 2k864 uses a broadcast mode in the interrupt routing

such that
> all vcpus get the clock interrupt. The best Windows drift performance was
> achieved when the
> policy code ensured that all the previous interrupts (on the various vcpus)
> had been injected
> before injecting the next interrupt to the vioapic.. >
> The policy code works as follows. It uses the hvm_isa_irq_assert_cb() to
> record
> the vcpus to be interrupted in h->hpet.pending_mask. Then, in the callback
> registered
> with hvm_register_intr_en_notif() at post=1 time it clears the current vcpu in
> the pending_mask.
> When the pending_mask is clear it decrements hpet.intr_pending_nr and if
> intr_pending_nr is still
> non-zero posts another interrupt to the ioapic with hvm_isa_irq_assert_cb().
> Intr_pending_nr is incremented in hpet_route_decision_not_missed_ticks().
>
> The missed ticks policy intr_en_notif callback also uses the pending_mask
> method. So even though
> Linux does not broadcast its interrupts, the code could handle it if it did.
> In this case the end of interrupt time stamp is made when the pending_mask is
> clear.
>
> 4. Live Migration
>
> Live migration with hpet preserves the current offset of the guest clock with
> respect
> to ntp. This is accomplished by migrating all of the state in the h->hpet data
> structure
> in the usual way. The hp->mc_offset is recalculated on the receiving node so
> that the
> guest sees a continuous hpet main counter.
>

Quoted 6 lines

> Code as been added to xc_domain_save.c to send a small message after the
> domain context is sent. The contents of the message is the physical tsc
> timestamp, last_tsc,
> read just before the message is sent. When the

last_tsc message

Quoted 39 lines

is received in
> xc_domain_restore.c,
> another physical tsc timestamp, cur_tsc, is read. The two timestamps are
> loaded into the domain
> structure as last_tsc_sender and first_tsc_receiver with hypercalls. Then
> xc_domain_hvm_setcontext
> is called so that hpet_load has access to these time stamps. Hpet_load uses
> the timestamps
> to account for the time spent saving and loading the domain context. With this
> technique,
> the only neglected time is the time spent sending a small network message.
>
> 5. Test Results
>
> Some recent test results are:
>
> 5.1 Linux 4u664 and Windows 2k864 load test.
> Duration: 70 hours.
> Test date: 6/2/08
> Loads: usex -b48 on Linux; burn-in on Windows
> Guest vcpus: 8 for Linux; 2 for Windows
> Hardware: 8 physical cpu AMD
> Clock drift : Linux: .0012% Windows: .009%
>
> 5.2 Linux 4u664, Linux 4u464 , and Windows 2k864 no-load test
> Duration: 23 hours.
> Test date: 6/3/08
> Loads: none
> Guest vcpus: 8 for each Linux; 2 for Windows
> Hardware: 4 physical cpu AMD
> Clock drift : Linux: .033% Windows: .019%
>
> 6. Relation to recent work in xen-unstable
>

Quoted 7 lines

> There is a similarity between hvm_get_guest_time() in xen-unstable and
> read_64_main_counter()
> in this code. However, read_64_main_counter() is more tuned to the needs of
> hpet.c. It has no
> "set" operation, only the get. It isolates the mode,

physical or

Quoted 6 lines

simulated, in
> read_64_main_counter()
> itself. It uses no vcpu or domain state as it is a physical entity, in either
> mode. And it provides a real
> physical mode for every read for those applications

that desire

Quoted 5 lines

this.
>
> 7. Conclusion
>
> The virtual hpet is improved by this patch in terms

of accuracy and

Quoted 2 lines

Quoted 2 lines

monotonicity.
Tests performed to date verify this and more testing

is under way.

Quoted 12 lines

Quoted 4 lines

8. Future Work

Testing with Windows Vista will be performed soon. The reason

for accuracy
> variations
> on different platforms using the physical hpet device will be investigated.
> Additional overhead measurements on simulated vs physical hpet mode will be
> made.
>

Quoted 3 lines

> Footnotes:
>
> 1. I don't recall the accuracy improvement with end

of interrupt

Quoted 23 lines

stamping, but
> it was
> significant, perhaps better than two to one improvement. It would be a very
> simple matter
> to re-measure the improvement as the facility can call back at injection time
> as well.
>
>
> Signed-off-by: Dave Winchell <[email protected]> > <mailto:[email protected]>
> Signed-off-by: Ben Guthro <[email protected]> > <mailto:[email protected]>
>
>
> _______________________________________________
> Xen-devel mailing list
> [email protected]
> http://lists.xensource.com/xen-devel

Although hwhpet=1 is a fine alternative in many cases, it may be unavailable on some systems and may cause significant performance issues on others. So I think we will still need to track down the poor accuracy when hwhpet=0. And if for some reason Xen system time can't be made accurate enough (< 0.05%), then I think we should consider building Xen system time itself on top of hardware hpet instead of TSC... at least when Xen discovers a capable hpet.

Or maybe there's a computation error somewhere in the hvm hpet scaling code? Hmmm...

Possibly there are bugs in the hpet device model which are fixed by Dave's patch. If this is actually the case, it would be nice to break those out as separate patches, as I think an 11% drift must largely be due to device-model bugs rather than relatively insignificant differences between hvm_get_guest_time() and physical HPET main counter.

Although hwhpet=1 is a fine alternative in many cases, it may be unavailable on some systems and may cause significant performance issues on others. So I think we will still need to track down the poor accuracy when hwhpet=0. And if for some reason Xen system time can't be made accurate enough (< 0.05%), then I think we should consider building Xen system time itself on top of hardware hpet instead of TSC... at least when Xen discovers a capable hpet.

Or maybe there's a computation error somewhere in the hvm hpet scaling code? Hmmm...

While I am fully supportive of offering hardware hpet as an option for hvm guests (let's call it hwhpet=1 for shorthand), I am very surprised by your preliminary results; the most obvious conclusion is that Xen system time is losing time at the rate of 1000 PPM though its possible there's a bug somewhere else in the "time stack". Your Windows result is jaw-dropping and inexplicable, though I have to admit ignorance of how Windows manages time.

I think with my recent patch and hpet=1 (essentially the same as your emulated hpet), hvm guest time should track Xen system time. I wonder if domain0 (which if I understand correctly is directly using Xen system time) is also seeing an error of .1%? Also I wonder for the skew you are seeing (in both hvm guests and domain0) is time moving too fast or two slow?

Although hwhpet=1 is a fine alternative in many cases, it may be unavailable on some systems and may cause significant performance issues on others. So I think we will still need to track down the poor accuracy when hwhpet=0.

And if for some reason
Xen system time can't be made accurate enough (< 0.05%), then I think we should consider building Xen system time itself on top of hardware hpet instead of TSC... at least when Xen discovers a capable hpet.

One more thought... do you know the accuracy of the TSC crystals on your test systems? I posted a patch awhile ago that was intended to test that, though I guess it was only testing skew of different TSCs on the same system, not TSCs against an external time source.

Or maybe there's a computation error somewhere in the hvm hpet scaling code? Hmmm...

-----Original Message-----
From: Dave Winchell [mailto:[email protected]]
Sent: Friday, June 06, 2008 1:33 PM
To: [email protected]; Keir Fraser
Cc: Ben Guthro; xen-devel; Dave Winchell
Subject: Re: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Dan, Keir:

Preliminary tests results indicate an error of .1% for Linux 64 bit guests configured
for hpet with xen-unstable as is. As we have discussed many times, the ntp requirement is .05%.
Tests on the patch we just submitted for hpet have indicated errors of .0012%
on this platform under similar test conditions and .03% on other platforms.

Windows vista64 has an error of 11% using hpet with the xen-unstable bits.
In an overnight test with our hpet patch, the Windows vista error was .008%.

The tests are with two or three guests on a physical node, all under load, and with
the ratio of vcpus to phys cpus > 1.

I will continue to run tests over the next few days.

thanks,
Dave

Quoted 3 lines · Dan Magenheimer

Hi Dave and Ben --

When running tests on xen-unstable (without your patch),

please ensure

Quoted 2 lines

that hpet=1 is set in the hvm config and also I think that when hpet is the clocksource on RHEL4-32, the clock IS resilient to

missed ticks

Quoted 1 line

so timer_mode should be 2 (vs when pit is the clocksource

on RHEL4-32,

Quoted 4 lines

all clock ticks must be delivered and so timer_mode should be 0).

Per

http://lists.xensource.com/archives/html/xen-devel/2008-06/msg
00098.html it's

Quoted 8 lines

my intent to clean this up, but I won't get to it until next week.

Thanks,
Dan

    -----Original Message-----
    *From:* [email protected]
    [mailto:[email protected]]*On 

Behalf Of *Dave

Quoted 9 lines

Winchell
*Sent:* Friday, June 06, 2008 4:46 AM
*To:* Keir Fraser; Ben Guthro; xen-devel
*Cc:* [email protected]; Dave Winchell *Subject:* RE: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Keir,

I think the changes are required. We'll run some tests

today today so

Quoted 7 lines

that we have some data to talk about.

-Dave

-----Original Message-----
From: [email protected] on behalf

of Keir Fraser

Quoted 8 lines

Sent: Fri 6/6/2008 4:58 AM
To: Ben Guthro; xen-devel
Cc: [email protected]
Subject: Re: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Are these patches needed now the timers are built on Xen system time rather
than host TSC? Dan has reported much better

time-keeping with his

Quoted 2 lines

patch
checked in, and it¹s for sure a lot less invasive than

this patchset.

Quoted 10 lines

     -- Keir

    On 5/6/08 15:59, "Ben Guthro" <[email protected]> wrote:

    >
    > 1. Introduction
    >
    > This patch improves the hpet based guest clock in 

terms of drift and

Quoted 2 lines

Quoted 2 lines

monotonicity.
Prior to this work the drift with hpet was greater

than 2%, far

Quoted 10 lines

above the .05%
> limit
> for ntp to synchronize. With this code, the drift ranges from .001% to .0033%
> depending
> on guest and physical platform.
>
> Using hpet allows guest operating systems to provide monotonic time to their
> applications. Time sources other than hpet are not

monotonic because

Quoted 1 line

Quoted 1 line

of their reliance on tsc, which is not synchronized

across physical

Quoted 31 lines

Quoted 3 lines

processors.

Windows 2k864 and many Linux guests are supported with two

policies, one for
> guests
> that handle missed clock interrupts and the other for guests that require the
> correct number of interrupts.
>
> Guests may use hpet for the timing source even if the physical platform has no
> visible
> hpet. Migration is supported between physical machines which differ in
> physical
> hpet visibility.
>
> Most of the changes are in hpet.c. Two general facilities are added to track
> interrupt
> progress. The ideas here and the facilities would be useful in vpt.c, for
> other time
> sources, though no attempt is made here to improve vpt.c. >
> The following sections discuss hpet dependencies, interrupt delivery policies,
> live migration,
> test results, and relation to recent work with monotonic time. >
>

Quoted 3 lines

> 2. Virtual Hpet dependencies
>
> The virtual hpet depends on the ability to read the

physical or

Quoted 2 lines

simulated
> (see discussion below) hpet. For timekeeping, the

virtual hpet

Quoted 32 lines

also depends
> on two new interrupt notification facilities to implement its policies for
> interrupt delivery.
>
> 2.1. Two modes of low-level hpet main counter reads. >
> In this implementation, the virtual hpet reads with read_64_main_counter(),
> exported by
> time.c, either the real physical hpet main counter register directly or a
> "simulated"
> hpet main counter.
>
> The simulated mode uses a monotonic version of get_s_time() (NOW()), where the
> last
> time value is returned whenever the current time value is less than the last
> time
> value. In simulated mode, since it is layered on s_time, the underlying
> hardware
> can be hpet or some other device. The frequency of the main counter in
> simulated
> mode is the same as the standard physical hpet frequency, allowing live
> migration
> between nodes that are configured differently.
>

Quoted 1 line

> If the physical platform does not have an hpet

device, or if xen

Quoted 9 lines

is configured
> not
> to use the device, then the simulated method is used. If there is a physical
> hpet device,
> and xen has initialized it, then either simulated or physical mode can be
> used.
> This is governed by a boot time option, hpet-avoid.

Setting this

Quoted 3 lines

option to 1
> gives the
> simulated mode and 0 the physical mode. The default

is physical

Quoted 5 lines

mode.
>
> A disadvantage of the physical mode is that may take longer to read the device
> than in simulated mode. On some platforms the cost is

about the

Quoted 15 lines

same (less
> than 250 nsec) for
> physical and simulated modes, while on others physical cost is much higher
> than simulated.
> A disadvantage of the simulated mode is that it can return the same value
> for the counter in consecutive calls.
>
> 2.2. Interrupt notification facilities.
>
> Two interrupt notification facilities are introduced, one is > hvm_isa_irq_assert_cb()
> and the other hvm_register_intr_en_notif().
>

Quoted 5 lines

> The vhpet uses hvm_isa_irq_assert_cb to deliver interrupts to the vioapic.
> hvm_isa_irq_assert_cb allows a callback to be passed along to > vioapic_deliver()
> and this callback is called with a mask of the vcpus

which will

Quoted 4 lines

get the
> interrupt. This callback is made before any vcpus receive an interrupt.
>

Quoted 8 lines

> Vhpet uses hvm_register_intr_en_notif() to register a handler for a particular
> vector that will be called when that vector is injected in > [vmx,svm]_intr_assist()
> and also when the guest finishes handling the interrupt. Here finished is
> defined
> as the point when the guest re-enables interrupts or

lowers the

Quoted 4 lines

tpr value.
> EOI is not used as the end of interrupt as this is sometimes returned before
> the interrupt handler has done its work. A flag is

passed to the

Quoted 3 lines

handler
> indicating
> whether this is the injection point (post = 1) or the

interrupt

Quoted 6 lines

finished (post
> = 0) point.
> The need for the finished point callback is discussed in the missed ticks
> policy section.
>

Quoted 7 lines

> To prevent a possible early trigger of the finished callback, intr_en_notif
> logic
> has a two stage arm, the first at injection (hvm_intr_en_notif_arm()) and the
> second when
> interrupts are seen to be disabled

(hvm_intr_en_notif_disarm()).

Quoted 3 lines

Once fully
> armed, re-enabling
> interrupts will cause hvm_intr_en_notif_disarm() to

make the end

Quoted 2 lines

of interrupt
> callback. hvm_intr_en_notif_arm() and

hvm_intr_en_notif_disarm()

Quoted 6 lines

are called by
> [vmx,svm]_intr_assist().
>
> 3. Interrupt delivery policies
>
> The existing hpet interrupt delivery is preserved.

This includes

Quoted 8 lines

Quoted 1 line

vcpu round robin delivery used by Linux and broadcast delivery

used by
> Windows.
>
> There are two policies for interrupt delivery, one for Windows 2k8-64 and the
> other
> for Linux. The Linux policy takes advantage of the

(guest) Linux

Quoted 3 lines

missed tick
> and offset
> calculations and does not attempt to deliver the

right number of

Quoted 10 lines

interrupts.
> The Windows policy delivers the correct number of interrupts, even if
> sometimes much
> closer to each other than the period. The policies are similar to those in
> vpt.c, though
> there are some important differences.
>
> Policies are selected with an HVMOP_set_param

hypercall with index

Quoted 2 lines

Quoted 2 lines

HVM_PARAM_TIMER_MODE.
Two new values are added,

HVM_HPET_guest_computes_missed_ticks and

Quoted 7 lines

Quoted 1 line

HVM_HPET_guest_does_not_compute_missed_ticks. The reason that

two new ones
> are added is that
> in some guests (32bit Linux) a no-missed policy is needed for clock sources
> other than hpet
> and a missed ticks policy for hpet. It was felt that

there would

Quoted 14 lines

be less
> confusion by simply
> introducing the two hpet policies.
>
> 3.1. The missed ticks policy
>
> The Linux clock interrupt handler for hpet calculates missed ticks and offset
> using the hpet
> main counter. The algorithm works well when the time since the last interrupt
> is greater than
> or equal to a period and poorly otherwise.
>

Quoted 1 line

> The missed ticks policy ensures that no two clock

interrupts are

Quoted 6 lines

delivered to
> the guest at
> a time interval less than a period. A time stamp (hpet main counter value) is
> recorded (by a
> callback registered with hvm_register_intr_en_notif)

when Linux

Quoted 3 lines

finishes
> handling the clock
> interrupt. Then, ensuing interrupts are delivered to

the vioapic

Quoted 9 lines

only if the
> current main
> counter value is a period greater than when the last interrupt was handled.
>
> Tests showed a significant improvement in clock drift with end of interrupt
> time stamps
> versus beginning of interrupt[1]. It is believed that

the reason

Quoted 3 lines

for the
> improvement
> is that the clock interrupt handler goes for a

spinlock and can

Quoted 11 lines

be therefore
> delayed in its
> processing. Furthermore, the main counter is read by the guest under the lock.
> The net
> effect is that if we time stamp injection, we can get the difference in time
> between successive interrupt handler lock acquisitions to be less than the
> period.
>

Quoted 3 lines

> 3.2. The no-missed ticks policy
>
> Windows 2k864 keeps very poor time with the missed

ticks policy.

Quoted 6 lines

So the
> no-missed ticks policy
> was developed. In the no-missed ticks policy we deliver the correct number of
> interrupts,
> even if they are spaced less than a period apart

(when catching up).

Quoted 50 lines

Quoted 2 lines

Windows 2k864 uses a broadcast mode in the interrupt routing

such that
> all vcpus get the clock interrupt. The best Windows drift performance was
> achieved when the
> policy code ensured that all the previous interrupts (on the various vcpus)
> had been injected
> before injecting the next interrupt to the vioapic.. >
> The policy code works as follows. It uses the hvm_isa_irq_assert_cb() to
> record
> the vcpus to be interrupted in h->hpet.pending_mask. Then, in the callback
> registered
> with hvm_register_intr_en_notif() at post=1 time it clears the current vcpu in
> the pending_mask.
> When the pending_mask is clear it decrements hpet.intr_pending_nr and if
> intr_pending_nr is still
> non-zero posts another interrupt to the ioapic with hvm_isa_irq_assert_cb().
> Intr_pending_nr is incremented in hpet_route_decision_not_missed_ticks().
>
> The missed ticks policy intr_en_notif callback also uses the pending_mask
> method. So even though
> Linux does not broadcast its interrupts, the code could handle it if it did.
> In this case the end of interrupt time stamp is made when the pending_mask is
> clear.
>
> 4. Live Migration
>
> Live migration with hpet preserves the current offset of the guest clock with
> respect
> to ntp. This is accomplished by migrating all of the state in the h->hpet data
> structure
> in the usual way. The hp->mc_offset is recalculated on the receiving node so
> that the
> guest sees a continuous hpet main counter.
>

Quoted 6 lines

> Code as been added to xc_domain_save.c to send a small message after the
> domain context is sent. The contents of the message is the physical tsc
> timestamp, last_tsc,
> read just before the message is sent. When the

last_tsc message

Quoted 39 lines

is received in
> xc_domain_restore.c,
> another physical tsc timestamp, cur_tsc, is read. The two timestamps are
> loaded into the domain
> structure as last_tsc_sender and first_tsc_receiver with hypercalls. Then
> xc_domain_hvm_setcontext
> is called so that hpet_load has access to these time stamps. Hpet_load uses
> the timestamps
> to account for the time spent saving and loading the domain context. With this
> technique,
> the only neglected time is the time spent sending a small network message.
>
> 5. Test Results
>
> Some recent test results are:
>
> 5.1 Linux 4u664 and Windows 2k864 load test.
> Duration: 70 hours.
> Test date: 6/2/08
> Loads: usex -b48 on Linux; burn-in on Windows
> Guest vcpus: 8 for Linux; 2 for Windows
> Hardware: 8 physical cpu AMD
> Clock drift : Linux: .0012% Windows: .009%
>
> 5.2 Linux 4u664, Linux 4u464 , and Windows 2k864 no-load test
> Duration: 23 hours.
> Test date: 6/3/08
> Loads: none
> Guest vcpus: 8 for each Linux; 2 for Windows
> Hardware: 4 physical cpu AMD
> Clock drift : Linux: .033% Windows: .019%
>
> 6. Relation to recent work in xen-unstable
>

Quoted 7 lines

> There is a similarity between hvm_get_guest_time() in xen-unstable and
> read_64_main_counter()
> in this code. However, read_64_main_counter() is more tuned to the needs of
> hpet.c. It has no
> "set" operation, only the get. It isolates the mode,

physical or

Quoted 6 lines

simulated, in
> read_64_main_counter()
> itself. It uses no vcpu or domain state as it is a physical entity, in either
> mode. And it provides a real
> physical mode for every read for those applications

that desire

Quoted 5 lines

this.
>
> 7. Conclusion
>
> The virtual hpet is improved by this patch in terms

of accuracy and

Quoted 2 lines

Quoted 2 lines

monotonicity.
Tests performed to date verify this and more testing

is under way.

Quoted 12 lines

Quoted 4 lines

8. Future Work

Testing with Windows Vista will be performed soon. The reason

for accuracy
> variations
> on different platforms using the physical hpet device will be investigated.
> Additional overhead measurements on simulated vs physical hpet mode will be
> made.
>

Quoted 3 lines

Quoted 3 lines

Footnotes:

1. I don't recall the accuracy improvement with end

of interrupt

Quoted 23 lines

stamping, but
> it was
> significant, perhaps better than two to one improvement. It would be a very
> simple matter
> to re-measure the improvement as the facility can call back at injection time
> as well.
>
>
> Signed-off-by: Dave Winchell <[email protected]> > <mailto:[email protected]>
> Signed-off-by: Ben Guthro <[email protected]> > <mailto:[email protected]>
>
>
> _______________________________________________
> Xen-devel mailing list
> [email protected]
> http://lists.xensource.com/xen-devel

While I am fully supportive of offering hardware hpet as an option for hvm guests (let's call it hwhpet=1 for shorthand), I am very surprised by your preliminary results; the most obvious conclusion is that Xen system time is losing time at the rate of 1000 PPM though its possible there's a bug somewhere else in the "time stack". Your Windows result is jaw-dropping and inexplicable, though I have to admit ignorance of how Windows manages time.

I think with my recent patch and hpet=1 (essentially the same as your emulated hpet), hvm guest time should track Xen system time. I wonder if domain0 (which if I understand correctly is directly using Xen system time) is also seeing an error of .1%? Also I wonder for the skew you are seeing (in both hvm guests and domain0) is time moving too fast or two slow?

Although hwhpet=1 is a fine alternative in many cases, it may be unavailable on some systems and may cause significant performance issues on others. So I think we will still need to track down the poor accuracy when hwhpet=0.

And if for some reason
Xen system time can't be made accurate enough (< 0.05%), then I think we should consider building Xen system time itself on top of hardware hpet instead of TSC... at least when Xen discovers a capable hpet.

One more thought... do you know the accuracy of the TSC crystals on your test systems? I posted a patch awhile ago that was intended to test that, though I guess it was only testing skew of different TSCs on the same system, not TSCs against an external time source.

Or maybe there's a computation error somewhere in the hvm hpet scaling code? Hmmm...

-----Original Message-----
From: Dave Winchell [mailto:[email protected]]
Sent: Friday, June 06, 2008 1:33 PM
To: [email protected]; Keir Fraser
Cc: Ben Guthro; xen-devel; Dave Winchell
Subject: Re: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Dan, Keir:

Preliminary tests results indicate an error of .1% for Linux 64 bit guests configured
for hpet with xen-unstable as is. As we have discussed many times, the ntp requirement is .05%.
Tests on the patch we just submitted for hpet have indicated errors of .0012%
on this platform under similar test conditions and .03% on other platforms.

Windows vista64 has an error of 11% using hpet with the xen-unstable bits.
In an overnight test with our hpet patch, the Windows vista error was .008%.

The tests are with two or three guests on a physical node, all under load, and with
the ratio of vcpus to phys cpus > 1.

I will continue to run tests over the next few days.

thanks,
Dave

Quoted 3 lines · Dan Magenheimer

Hi Dave and Ben --

When running tests on xen-unstable (without your patch),

please ensure

Quoted 2 lines

that hpet=1 is set in the hvm config and also I think that when hpet is the clocksource on RHEL4-32, the clock IS resilient to

missed ticks

Quoted 1 line

so timer_mode should be 2 (vs when pit is the clocksource

on RHEL4-32,

Quoted 4 lines

all clock ticks must be delivered and so timer_mode should be 0).

Per

http://lists.xensource.com/archives/html/xen-devel/2008-06/msg
00098.html it's

Quoted 8 lines

my intent to clean this up, but I won't get to it until next week.

Thanks,
Dan

    -----Original Message-----
    *From:* [email protected]
    [mailto:[email protected]]*On

Behalf Of *Dave

Quoted 9 lines

Winchell
*Sent:* Friday, June 06, 2008 4:46 AM
*To:* Keir Fraser; Ben Guthro; xen-devel
*Cc:* [email protected]; Dave Winchell *Subject:* RE: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Keir,

I think the changes are required. We'll run some tests

today today so

Quoted 7 lines

that we have some data to talk about.

-Dave

-----Original Message-----
From: [email protected] on behalf

of Keir Fraser

Quoted 8 lines

Sent: Fri 6/6/2008 4:58 AM
To: Ben Guthro; xen-devel
Cc: [email protected]
Subject: Re: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Are these patches needed now the timers are built on Xen system time rather
than host TSC? Dan has reported much better

time-keeping with his

Quoted 2 lines

patch
checked in, and it¹s for sure a lot less invasive than

this patchset.

Quoted 10 lines

     -- Keir

    On 5/6/08 15:59, "Ben Guthro" <[email protected]> wrote:

    >
    > 1. Introduction
    >
    > This patch improves the hpet based guest clock in

terms of drift and

Quoted 2 lines

Quoted 2 lines

monotonicity.
Prior to this work the drift with hpet was greater

than 2%, far

Quoted 10 lines

above the .05%
> limit
> for ntp to synchronize. With this code, the drift ranges from .001% to .0033%
> depending
> on guest and physical platform.
>
> Using hpet allows guest operating systems to provide monotonic time to their
> applications. Time sources other than hpet are not

monotonic because

Quoted 1 line

Quoted 1 line

of their reliance on tsc, which is not synchronized

across physical

Quoted 31 lines

Quoted 3 lines

processors.

Windows 2k864 and many Linux guests are supported with two

policies, one for
> guests
> that handle missed clock interrupts and the other for guests that require the
> correct number of interrupts.
>
> Guests may use hpet for the timing source even if the physical platform has no
> visible
> hpet. Migration is supported between physical machines which differ in
> physical
> hpet visibility.
>
> Most of the changes are in hpet.c. Two general facilities are added to track
> interrupt
> progress. The ideas here and the facilities would be useful in vpt.c, for
> other time
> sources, though no attempt is made here to improve vpt.c. >
> The following sections discuss hpet dependencies, interrupt delivery policies,
> live migration,
> test results, and relation to recent work with monotonic time. >
>

Quoted 3 lines

> 2. Virtual Hpet dependencies
>
> The virtual hpet depends on the ability to read the

physical or

Quoted 2 lines

simulated
> (see discussion below) hpet. For timekeeping, the

virtual hpet

Quoted 32 lines

also depends
> on two new interrupt notification facilities to implement its policies for
> interrupt delivery.
>
> 2.1. Two modes of low-level hpet main counter reads. >
> In this implementation, the virtual hpet reads with read_64_main_counter(),
> exported by
> time.c, either the real physical hpet main counter register directly or a
> "simulated"
> hpet main counter.
>
> The simulated mode uses a monotonic version of get_s_time() (NOW()), where the
> last
> time value is returned whenever the current time value is less than the last
> time
> value. In simulated mode, since it is layered on s_time, the underlying
> hardware
> can be hpet or some other device. The frequency of the main counter in
> simulated
> mode is the same as the standard physical hpet frequency, allowing live
> migration
> between nodes that are configured differently.
>

Quoted 1 line

> If the physical platform does not have an hpet

device, or if xen

Quoted 9 lines

is configured
> not
> to use the device, then the simulated method is used. If there is a physical
> hpet device,
> and xen has initialized it, then either simulated or physical mode can be
> used.
> This is governed by a boot time option, hpet-avoid.

Setting this

Quoted 3 lines

option to 1
> gives the
> simulated mode and 0 the physical mode. The default

is physical

Quoted 5 lines

mode.
>
> A disadvantage of the physical mode is that may take longer to read the device
> than in simulated mode. On some platforms the cost is

about the

Quoted 15 lines

same (less
> than 250 nsec) for
> physical and simulated modes, while on others physical cost is much higher
> than simulated.
> A disadvantage of the simulated mode is that it can return the same value
> for the counter in consecutive calls.
>
> 2.2. Interrupt notification facilities.
>
> Two interrupt notification facilities are introduced, one is > hvm_isa_irq_assert_cb()
> and the other hvm_register_intr_en_notif().
>

Quoted 5 lines

> The vhpet uses hvm_isa_irq_assert_cb to deliver interrupts to the vioapic.
> hvm_isa_irq_assert_cb allows a callback to be passed along to > vioapic_deliver()
> and this callback is called with a mask of the vcpus

which will

Quoted 4 lines

get the
> interrupt. This callback is made before any vcpus receive an interrupt.
>

Quoted 8 lines

> Vhpet uses hvm_register_intr_en_notif() to register a handler for a particular
> vector that will be called when that vector is injected in > [vmx,svm]_intr_assist()
> and also when the guest finishes handling the interrupt. Here finished is
> defined
> as the point when the guest re-enables interrupts or

lowers the

Quoted 4 lines

tpr value.
> EOI is not used as the end of interrupt as this is sometimes returned before
> the interrupt handler has done its work. A flag is

passed to the

Quoted 3 lines

handler
> indicating
> whether this is the injection point (post = 1) or the

interrupt

Quoted 6 lines

finished (post
> = 0) point.
> The need for the finished point callback is discussed in the missed ticks
> policy section.
>

Quoted 7 lines

> To prevent a possible early trigger of the finished callback, intr_en_notif
> logic
> has a two stage arm, the first at injection (hvm_intr_en_notif_arm()) and the
> second when
> interrupts are seen to be disabled

(hvm_intr_en_notif_disarm()).

Quoted 3 lines

Once fully
> armed, re-enabling
> interrupts will cause hvm_intr_en_notif_disarm() to

make the end

Quoted 2 lines

of interrupt
> callback. hvm_intr_en_notif_arm() and

hvm_intr_en_notif_disarm()

Quoted 6 lines

are called by
> [vmx,svm]_intr_assist().
>
> 3. Interrupt delivery policies
>
> The existing hpet interrupt delivery is preserved.

This includes

Quoted 8 lines

Quoted 1 line

vcpu round robin delivery used by Linux and broadcast delivery

used by
> Windows.
>
> There are two policies for interrupt delivery, one for Windows 2k8-64 and the
> other
> for Linux. The Linux policy takes advantage of the

(guest) Linux

Quoted 3 lines

missed tick
> and offset
> calculations and does not attempt to deliver the

right number of

Quoted 10 lines

interrupts.
> The Windows policy delivers the correct number of interrupts, even if
> sometimes much
> closer to each other than the period. The policies are similar to those in
> vpt.c, though
> there are some important differences.
>
> Policies are selected with an HVMOP_set_param

hypercall with index

Quoted 2 lines

Quoted 2 lines

HVM_PARAM_TIMER_MODE.
Two new values are added,

HVM_HPET_guest_computes_missed_ticks and

Quoted 7 lines

Quoted 1 line

HVM_HPET_guest_does_not_compute_missed_ticks. The reason that

two new ones
> are added is that
> in some guests (32bit Linux) a no-missed policy is needed for clock sources
> other than hpet
> and a missed ticks policy for hpet. It was felt that

there would

Quoted 14 lines

be less
> confusion by simply
> introducing the two hpet policies.
>
> 3.1. The missed ticks policy
>
> The Linux clock interrupt handler for hpet calculates missed ticks and offset
> using the hpet
> main counter. The algorithm works well when the time since the last interrupt
> is greater than
> or equal to a period and poorly otherwise.
>

Quoted 1 line

> The missed ticks policy ensures that no two clock

interrupts are

Quoted 6 lines

delivered to
> the guest at
> a time interval less than a period. A time stamp (hpet main counter value) is
> recorded (by a
> callback registered with hvm_register_intr_en_notif)

when Linux

Quoted 3 lines

finishes
> handling the clock
> interrupt. Then, ensuing interrupts are delivered to

the vioapic

Quoted 9 lines

only if the
> current main
> counter value is a period greater than when the last interrupt was handled.
>
> Tests showed a significant improvement in clock drift with end of interrupt
> time stamps
> versus beginning of interrupt[1]. It is believed that

the reason

Quoted 3 lines

for the
> improvement
> is that the clock interrupt handler goes for a

spinlock and can

Quoted 11 lines

be therefore
> delayed in its
> processing. Furthermore, the main counter is read by the guest under the lock.
> The net
> effect is that if we time stamp injection, we can get the difference in time
> between successive interrupt handler lock acquisitions to be less than the
> period.
>

Quoted 3 lines

> 3.2. The no-missed ticks policy
>
> Windows 2k864 keeps very poor time with the missed

ticks policy.

Quoted 6 lines

So the
> no-missed ticks policy
> was developed. In the no-missed ticks policy we deliver the correct number of
> interrupts,
> even if they are spaced less than a period apart

(when catching up).

Quoted 50 lines

Quoted 2 lines

Windows 2k864 uses a broadcast mode in the interrupt routing

such that
> all vcpus get the clock interrupt. The best Windows drift performance was
> achieved when the
> policy code ensured that all the previous interrupts (on the various vcpus)
> had been injected
> before injecting the next interrupt to the vioapic.. >
> The policy code works as follows. It uses the hvm_isa_irq_assert_cb() to
> record
> the vcpus to be interrupted in h->hpet.pending_mask. Then, in the callback
> registered
> with hvm_register_intr_en_notif() at post=1 time it clears the current vcpu in
> the pending_mask.
> When the pending_mask is clear it decrements hpet.intr_pending_nr and if
> intr_pending_nr is still
> non-zero posts another interrupt to the ioapic with hvm_isa_irq_assert_cb().
> Intr_pending_nr is incremented in hpet_route_decision_not_missed_ticks().
>
> The missed ticks policy intr_en_notif callback also uses the pending_mask
> method. So even though
> Linux does not broadcast its interrupts, the code could handle it if it did.
> In this case the end of interrupt time stamp is made when the pending_mask is
> clear.
>
> 4. Live Migration
>
> Live migration with hpet preserves the current offset of the guest clock with
> respect
> to ntp. This is accomplished by migrating all of the state in the h->hpet data
> structure
> in the usual way. The hp->mc_offset is recalculated on the receiving node so
> that the
> guest sees a continuous hpet main counter.
>

Quoted 6 lines

> Code as been added to xc_domain_save.c to send a small message after the
> domain context is sent. The contents of the message is the physical tsc
> timestamp, last_tsc,
> read just before the message is sent. When the

last_tsc message

Quoted 39 lines

is received in
> xc_domain_restore.c,
> another physical tsc timestamp, cur_tsc, is read. The two timestamps are
> loaded into the domain
> structure as last_tsc_sender and first_tsc_receiver with hypercalls. Then
> xc_domain_hvm_setcontext
> is called so that hpet_load has access to these time stamps. Hpet_load uses
> the timestamps
> to account for the time spent saving and loading the domain context. With this
> technique,
> the only neglected time is the time spent sending a small network message.
>
> 5. Test Results
>
> Some recent test results are:
>
> 5.1 Linux 4u664 and Windows 2k864 load test.
> Duration: 70 hours.
> Test date: 6/2/08
> Loads: usex -b48 on Linux; burn-in on Windows
> Guest vcpus: 8 for Linux; 2 for Windows
> Hardware: 8 physical cpu AMD
> Clock drift : Linux: .0012% Windows: .009%
>
> 5.2 Linux 4u664, Linux 4u464 , and Windows 2k864 no-load test
> Duration: 23 hours.
> Test date: 6/3/08
> Loads: none
> Guest vcpus: 8 for each Linux; 2 for Windows
> Hardware: 4 physical cpu AMD
> Clock drift : Linux: .033% Windows: .019%
>
> 6. Relation to recent work in xen-unstable
>

Quoted 7 lines

> There is a similarity between hvm_get_guest_time() in xen-unstable and
> read_64_main_counter()
> in this code. However, read_64_main_counter() is more tuned to the needs of
> hpet.c. It has no
> "set" operation, only the get. It isolates the mode,

physical or

Quoted 6 lines

simulated, in
> read_64_main_counter()
> itself. It uses no vcpu or domain state as it is a physical entity, in either
> mode. And it provides a real
> physical mode for every read for those applications

that desire

Quoted 5 lines

this.
>
> 7. Conclusion
>
> The virtual hpet is improved by this patch in terms

of accuracy and

Quoted 2 lines

Quoted 2 lines

monotonicity.
Tests performed to date verify this and more testing

is under way.

Quoted 12 lines

Quoted 4 lines

8. Future Work

Testing with Windows Vista will be performed soon. The reason

for accuracy
> variations
> on different platforms using the physical hpet device will be investigated.
> Additional overhead measurements on simulated vs physical hpet mode will be
> made.
>

Quoted 3 lines

Quoted 3 lines

Footnotes:

1. I don't recall the accuracy improvement with end

of interrupt

Quoted 23 lines

stamping, but
> it was
> significant, perhaps better than two to one improvement. It would be a very
> simple matter
> to re-measure the improvement as the facility can call back at injection time
> as well.
>
>
> Signed-off-by: Dave Winchell <[email protected]> > <mailto:[email protected]>
> Signed-off-by: Ben Guthro <[email protected]> > <mailto:[email protected]>
>
>
> _______________________________________________
> Xen-devel mailing list
> [email protected]
> http://lists.xensource.com/xen-devel

for the counter in consecutive calls.

While I am fully supportive of offering hardware hpet as an option for hvm guests (let's call it hwhpet=1 for shorthand), I am very surprised by your preliminary results; the most obvious conclusion is that Xen system time is losing time at the rate of 1000 PPM though its possible there's a bug somewhere else in the "time stack". Your Windows result is jaw-dropping and inexplicable, though I have to admit ignorance of how Windows manages time.

I think with my recent patch and hpet=1 (essentially the same as your emulated hpet), hvm guest time should track Xen system time. I wonder if domain0 (which if I understand correctly is directly using Xen system time) is also seeing an error of .1%? Also I wonder for the skew you are seeing (in both hvm guests and domain0) is time moving too fast or two slow?

Although hwhpet=1 is a fine alternative in many cases, it may be unavailable on some systems and may cause significant performance issues on others. So I think we will still need to track down the poor accuracy when hwhpet=0.

And if for some reason
Xen system time can't be made accurate enough (< 0.05%), then I think we should consider building Xen system time itself on top of hardware hpet instead of TSC... at least when Xen discovers a capable hpet.

One more thought... do you know the accuracy of the TSC crystals on your test systems? I posted a patch awhile ago that was intended to test that, though I guess it was only testing skew of different TSCs on the same system, not TSCs against an external time source.

Or maybe there's a computation error somewhere in the hvm hpet scaling code? Hmmm...

-----Original Message-----
From: Dave Winchell [mailto:[email protected]]
Sent: Friday, June 06, 2008 1:33 PM
To: [email protected]; Keir Fraser
Cc: Ben Guthro; xen-devel; Dave Winchell
Subject: Re: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Dan, Keir:

Preliminary tests results indicate an error of .1% for Linux 64 bit guests configured
for hpet with xen-unstable as is. As we have discussed many times, the ntp requirement is .05%.
Tests on the patch we just submitted for hpet have indicated errors of .0012%
on this platform under similar test conditions and .03% on other platforms.

Windows vista64 has an error of 11% using hpet with the xen-unstable bits.
In an overnight test with our hpet patch, the Windows vista error was .008%.

The tests are with two or three guests on a physical node, all under load, and with
the ratio of vcpus to phys cpus > 1.

I will continue to run tests over the next few days.

thanks,
Dave

Quoted 3 lines · Dan Magenheimer

Hi Dave and Ben --

When running tests on xen-unstable (without your patch),

please ensure

Quoted 2 lines

that hpet=1 is set in the hvm config and also I think that when hpet is the clocksource on RHEL4-32, the clock IS resilient to

missed ticks

Quoted 1 line

so timer_mode should be 2 (vs when pit is the clocksource

on RHEL4-32,

Quoted 4 lines

all clock ticks must be delivered and so timer_mode should be 0).

Per

http://lists.xensource.com/archives/html/xen-devel/2008-06/msg
00098.html it's

Quoted 8 lines

my intent to clean this up, but I won't get to it until next week.

Thanks,
Dan

    -----Original Message-----
    *From:* [email protected]
    [mailto:[email protected]]*On

Behalf Of *Dave

Quoted 9 lines

Winchell
*Sent:* Friday, June 06, 2008 4:46 AM
*To:* Keir Fraser; Ben Guthro; xen-devel
*Cc:* [email protected]; Dave Winchell *Subject:* RE: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Keir,

I think the changes are required. We'll run some tests

today today so

Quoted 7 lines

that we have some data to talk about.

-Dave

-----Original Message-----
From: [email protected] on behalf

of Keir Fraser

Quoted 8 lines

Sent: Fri 6/6/2008 4:58 AM
To: Ben Guthro; xen-devel
Cc: [email protected]
Subject: Re: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Are these patches needed now the timers are built on Xen system time rather
than host TSC? Dan has reported much better

time-keeping with his

Quoted 2 lines

patch
checked in, and it¹s for sure a lot less invasive than

this patchset.

Quoted 10 lines

     -- Keir

    On 5/6/08 15:59, "Ben Guthro" <[email protected]> wrote:

    >
    > 1. Introduction
    >
    > This patch improves the hpet based guest clock in

terms of drift and

Quoted 2 lines

Quoted 2 lines

monotonicity.
Prior to this work the drift with hpet was greater

than 2%, far

Quoted 10 lines

above the .05%
> limit
> for ntp to synchronize. With this code, the drift ranges from .001% to .0033%
> depending
> on guest and physical platform.
>
> Using hpet allows guest operating systems to provide monotonic time to their
> applications. Time sources other than hpet are not

monotonic because

Quoted 1 line

Quoted 1 line

of their reliance on tsc, which is not synchronized

across physical

Quoted 31 lines

Quoted 3 lines

processors.

Windows 2k864 and many Linux guests are supported with two

policies, one for
> guests
> that handle missed clock interrupts and the other for guests that require the
> correct number of interrupts.
>
> Guests may use hpet for the timing source even if the physical platform has no
> visible
> hpet. Migration is supported between physical machines which differ in
> physical
> hpet visibility.
>
> Most of the changes are in hpet.c. Two general facilities are added to track
> interrupt
> progress. The ideas here and the facilities would be useful in vpt.c, for
> other time
> sources, though no attempt is made here to improve vpt.c. >
> The following sections discuss hpet dependencies, interrupt delivery policies,
> live migration,
> test results, and relation to recent work with monotonic time. >
>

Quoted 3 lines

> 2. Virtual Hpet dependencies
>
> The virtual hpet depends on the ability to read the

physical or

Quoted 2 lines

simulated
> (see discussion below) hpet. For timekeeping, the

virtual hpet

Quoted 32 lines

also depends
> on two new interrupt notification facilities to implement its policies for
> interrupt delivery.
>
> 2.1. Two modes of low-level hpet main counter reads. >
> In this implementation, the virtual hpet reads with read_64_main_counter(),
> exported by
> time.c, either the real physical hpet main counter register directly or a
> "simulated"
> hpet main counter.
>
> The simulated mode uses a monotonic version of get_s_time() (NOW()), where the
> last
> time value is returned whenever the current time value is less than the last
> time
> value. In simulated mode, since it is layered on s_time, the underlying
> hardware
> can be hpet or some other device. The frequency of the main counter in
> simulated
> mode is the same as the standard physical hpet frequency, allowing live
> migration
> between nodes that are configured differently.
>

Quoted 1 line

> If the physical platform does not have an hpet

device, or if xen

Quoted 9 lines

is configured
> not
> to use the device, then the simulated method is used. If there is a physical
> hpet device,
> and xen has initialized it, then either simulated or physical mode can be
> used.
> This is governed by a boot time option, hpet-avoid.

Setting this

Quoted 3 lines

option to 1
> gives the
> simulated mode and 0 the physical mode. The default

is physical

Quoted 5 lines

mode.
>
> A disadvantage of the physical mode is that may take longer to read the device
> than in simulated mode. On some platforms the cost is

about the

Quoted 15 lines

same (less
> than 250 nsec) for
> physical and simulated modes, while on others physical cost is much higher
> than simulated.
> A disadvantage of the simulated mode is that it can return the same value
> for the counter in consecutive calls.
>
> 2.2. Interrupt notification facilities.
>
> Two interrupt notification facilities are introduced, one is > hvm_isa_irq_assert_cb()
> and the other hvm_register_intr_en_notif().
>

Quoted 5 lines

> The vhpet uses hvm_isa_irq_assert_cb to deliver interrupts to the vioapic.
> hvm_isa_irq_assert_cb allows a callback to be passed along to > vioapic_deliver()
> and this callback is called with a mask of the vcpus

which will

Quoted 4 lines

get the
> interrupt. This callback is made before any vcpus receive an interrupt.
>

Quoted 8 lines

> Vhpet uses hvm_register_intr_en_notif() to register a handler for a particular
> vector that will be called when that vector is injected in > [vmx,svm]_intr_assist()
> and also when the guest finishes handling the interrupt. Here finished is
> defined
> as the point when the guest re-enables interrupts or

lowers the

Quoted 4 lines

tpr value.
> EOI is not used as the end of interrupt as this is sometimes returned before
> the interrupt handler has done its work. A flag is

passed to the

Quoted 3 lines

handler
> indicating
> whether this is the injection point (post = 1) or the

interrupt

Quoted 6 lines

finished (post
> = 0) point.
> The need for the finished point callback is discussed in the missed ticks
> policy section.
>

Quoted 7 lines

> To prevent a possible early trigger of the finished callback, intr_en_notif
> logic
> has a two stage arm, the first at injection (hvm_intr_en_notif_arm()) and the
> second when
> interrupts are seen to be disabled

(hvm_intr_en_notif_disarm()).

Quoted 3 lines

Once fully
> armed, re-enabling
> interrupts will cause hvm_intr_en_notif_disarm() to

make the end

Quoted 2 lines

of interrupt
> callback. hvm_intr_en_notif_arm() and

hvm_intr_en_notif_disarm()

Quoted 6 lines

are called by
> [vmx,svm]_intr_assist().
>
> 3. Interrupt delivery policies
>
> The existing hpet interrupt delivery is preserved.

This includes

Quoted 8 lines

Quoted 1 line

vcpu round robin delivery used by Linux and broadcast delivery

used by
> Windows.
>
> There are two policies for interrupt delivery, one for Windows 2k8-64 and the
> other
> for Linux. The Linux policy takes advantage of the

(guest) Linux

Quoted 3 lines

missed tick
> and offset
> calculations and does not attempt to deliver the

right number of

Quoted 10 lines

interrupts.
> The Windows policy delivers the correct number of interrupts, even if
> sometimes much
> closer to each other than the period. The policies are similar to those in
> vpt.c, though
> there are some important differences.
>
> Policies are selected with an HVMOP_set_param

hypercall with index

Quoted 2 lines

Quoted 2 lines

HVM_PARAM_TIMER_MODE.
Two new values are added,

HVM_HPET_guest_computes_missed_ticks and

Quoted 7 lines

Quoted 1 line

HVM_HPET_guest_does_not_compute_missed_ticks. The reason that

two new ones
> are added is that
> in some guests (32bit Linux) a no-missed policy is needed for clock sources
> other than hpet
> and a missed ticks policy for hpet. It was felt that

there would

Quoted 14 lines

be less
> confusion by simply
> introducing the two hpet policies.
>
> 3.1. The missed ticks policy
>
> The Linux clock interrupt handler for hpet calculates missed ticks and offset
> using the hpet
> main counter. The algorithm works well when the time since the last interrupt
> is greater than
> or equal to a period and poorly otherwise.
>

Quoted 1 line

> The missed ticks policy ensures that no two clock

interrupts are

Quoted 6 lines

delivered to
> the guest at
> a time interval less than a period. A time stamp (hpet main counter value) is
> recorded (by a
> callback registered with hvm_register_intr_en_notif)

when Linux

Quoted 3 lines

finishes
> handling the clock
> interrupt. Then, ensuing interrupts are delivered to

the vioapic

Quoted 9 lines

only if the
> current main
> counter value is a period greater than when the last interrupt was handled.
>
> Tests showed a significant improvement in clock drift with end of interrupt
> time stamps
> versus beginning of interrupt[1]. It is believed that

the reason

Quoted 3 lines

for the
> improvement
> is that the clock interrupt handler goes for a

spinlock and can

Quoted 11 lines

be therefore
> delayed in its
> processing. Furthermore, the main counter is read by the guest under the lock.
> The net
> effect is that if we time stamp injection, we can get the difference in time
> between successive interrupt handler lock acquisitions to be less than the
> period.
>

Quoted 3 lines

> 3.2. The no-missed ticks policy
>
> Windows 2k864 keeps very poor time with the missed

ticks policy.

Quoted 6 lines

So the
> no-missed ticks policy
> was developed. In the no-missed ticks policy we deliver the correct number of
> interrupts,
> even if they are spaced less than a period apart

(when catching up).

Quoted 50 lines

Quoted 2 lines

Windows 2k864 uses a broadcast mode in the interrupt routing

such that
> all vcpus get the clock interrupt. The best Windows drift performance was
> achieved when the
> policy code ensured that all the previous interrupts (on the various vcpus)
> had been injected
> before injecting the next interrupt to the vioapic.. >
> The policy code works as follows. It uses the hvm_isa_irq_assert_cb() to
> record
> the vcpus to be interrupted in h->hpet.pending_mask. Then, in the callback
> registered
> with hvm_register_intr_en_notif() at post=1 time it clears the current vcpu in
> the pending_mask.
> When the pending_mask is clear it decrements hpet.intr_pending_nr and if
> intr_pending_nr is still
> non-zero posts another interrupt to the ioapic with hvm_isa_irq_assert_cb().
> Intr_pending_nr is incremented in hpet_route_decision_not_missed_ticks().
>
> The missed ticks policy intr_en_notif callback also uses the pending_mask
> method. So even though
> Linux does not broadcast its interrupts, the code could handle it if it did.
> In this case the end of interrupt time stamp is made when the pending_mask is
> clear.
>
> 4. Live Migration
>
> Live migration with hpet preserves the current offset of the guest clock with
> respect
> to ntp. This is accomplished by migrating all of the state in the h->hpet data
> structure
> in the usual way. The hp->mc_offset is recalculated on the receiving node so
> that the
> guest sees a continuous hpet main counter.
>

Quoted 6 lines

> Code as been added to xc_domain_save.c to send a small message after the
> domain context is sent. The contents of the message is the physical tsc
> timestamp, last_tsc,
> read just before the message is sent. When the

last_tsc message

Quoted 39 lines

is received in
> xc_domain_restore.c,
> another physical tsc timestamp, cur_tsc, is read. The two timestamps are
> loaded into the domain
> structure as last_tsc_sender and first_tsc_receiver with hypercalls. Then
> xc_domain_hvm_setcontext
> is called so that hpet_load has access to these time stamps. Hpet_load uses
> the timestamps
> to account for the time spent saving and loading the domain context. With this
> technique,
> the only neglected time is the time spent sending a small network message.
>
> 5. Test Results
>
> Some recent test results are:
>
> 5.1 Linux 4u664 and Windows 2k864 load test.
> Duration: 70 hours.
> Test date: 6/2/08
> Loads: usex -b48 on Linux; burn-in on Windows
> Guest vcpus: 8 for Linux; 2 for Windows
> Hardware: 8 physical cpu AMD
> Clock drift : Linux: .0012% Windows: .009%
>
> 5.2 Linux 4u664, Linux 4u464 , and Windows 2k864 no-load test
> Duration: 23 hours.
> Test date: 6/3/08
> Loads: none
> Guest vcpus: 8 for each Linux; 2 for Windows
> Hardware: 4 physical cpu AMD
> Clock drift : Linux: .033% Windows: .019%
>
> 6. Relation to recent work in xen-unstable
>

Quoted 7 lines

> There is a similarity between hvm_get_guest_time() in xen-unstable and
> read_64_main_counter()
> in this code. However, read_64_main_counter() is more tuned to the needs of
> hpet.c. It has no
> "set" operation, only the get. It isolates the mode,

physical or

Quoted 6 lines

simulated, in
> read_64_main_counter()
> itself. It uses no vcpu or domain state as it is a physical entity, in either
> mode. And it provides a real
> physical mode for every read for those applications

that desire

Quoted 5 lines

this.
>
> 7. Conclusion
>
> The virtual hpet is improved by this patch in terms

of accuracy and

Quoted 2 lines

Quoted 2 lines

monotonicity.
Tests performed to date verify this and more testing

is under way.

Quoted 12 lines

Quoted 4 lines

8. Future Work

Testing with Windows Vista will be performed soon. The reason

for accuracy
> variations
> on different platforms using the physical hpet device will be investigated.
> Additional overhead measurements on simulated vs physical hpet mode will be
> made.
>

Quoted 3 lines

Quoted 3 lines

Footnotes:

1. I don't recall the accuracy improvement with end

of interrupt

Quoted 23 lines

stamping, but
> it was
> significant, perhaps better than two to one improvement. It would be a very
> simple matter
> to re-measure the improvement as the facility can call back at injection time
> as well.
>
>
> Signed-off-by: Dave Winchell <[email protected]> > <mailto:[email protected]>
> Signed-off-by: Ben Guthro <[email protected]> > <mailto:[email protected]>
>
>
> _______________________________________________
> Xen-devel mailing list
> [email protected]
> http://lists.xensource.com/xen-devel

Could you please be very precise, when you say "Linux", as to what you are (and are not) testing? Specifically:
1) kernel version number and/or distro info

2) 32 vs 64

3) kernel parameters specified

4) config file parameters

5) relevant CPU info that may be passed through by Xen to hvm guests (e.g. whether "tsc is synchronized")

6) relevant xen boot parameters (if any)

As we've seen, different combinations of the above can yield very different test results. We'd like to confirm your tests, but if we can avoid unnecessary additional iterations (due to mismatches on the above), that would be helpful.

Our testing goal is to ensure that there is at least one known good combination of parameters for each of RHEL3, RHEL4, and RHEL5 (both 32 and 64) and that works on both tsc-synchronized and tsc-unsynchronized Intel and AMD boxes. And hopefully that works with and without a real physical hpet available.

We don't have a good test environment for Windows time, but if you can provide the same test configuration detail, we may be able to do some testing.

While I am fully supportive of offering hardware hpet as an option for hvm guests (let's call it hwhpet=1 for shorthand), I am very surprised by your preliminary results; the most obvious conclusion is that Xen system time is losing time at the rate of 1000 PPM though its possible there's a bug somewhere else in the "time stack". Your Windows result is jaw-dropping and inexplicable, though I have to admit ignorance of how Windows manages time.

I think with my recent patch and hpet=1 (essentially the same as your emulated hpet), hvm guest time should track Xen system time. I wonder if domain0 (which if I understand correctly is directly using Xen system time) is also seeing an error of .1%? Also I wonder for the skew you are seeing (in both hvm guests and domain0) is time moving too fast or two slow?

Although hwhpet=1 is a fine alternative in many cases, it may be unavailable on some systems and may cause significant performance issues on others. So I think we will still need to track down the poor accuracy when hwhpet=0.

And if for some reason
Xen system time can't be made accurate enough (< 0.05%), then I think we should consider building Xen system time itself on top of hardware hpet instead of TSC... at least when Xen discovers a capable hpet.

One more thought... do you know the accuracy of the TSC crystals on your test systems? I posted a patch awhile ago that was intended to test that, though I guess it was only testing skew of different TSCs on the same system, not TSCs against an external time source.

Or maybe there's a computation error somewhere in the hvm hpet scaling code? Hmmm...

-----Original Message-----
From: Dave Winchell [mailto:[email protected]]
Sent: Friday, June 06, 2008 1:33 PM
To: [email protected]; Keir Fraser
Cc: Ben Guthro; xen-devel; Dave Winchell
Subject: Re: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Dan, Keir:

Preliminary tests results indicate an error of .1% for Linux 64 bit guests configured
for hpet with xen-unstable as is. As we have discussed many times, the ntp requirement is .05%.
Tests on the patch we just submitted for hpet have indicated errors of .0012%
on this platform under similar test conditions and .03% on other platforms.

Windows vista64 has an error of 11% using hpet with the xen-unstable bits.
In an overnight test with our hpet patch, the Windows vista error was .008%.

The tests are with two or three guests on a physical node, all under load, and with
the ratio of vcpus to phys cpus > 1.

I will continue to run tests over the next few days.

thanks,
Dave

Quoted 3 lines · Dan Magenheimer

Hi Dave and Ben --

When running tests on xen-unstable (without your patch),

please ensure

Quoted 2 lines

that hpet=1 is set in the hvm config and also I think that when hpet is the clocksource on RHEL4-32, the clock IS resilient to

missed ticks

Quoted 1 line

so timer_mode should be 2 (vs when pit is the clocksource

on RHEL4-32,

Quoted 4 lines

all clock ticks must be delivered and so timer_mode should be 0).

Per

http://lists.xensource.com/archives/html/xen-devel/2008-06/msg
00098.html it's

Quoted 8 lines

my intent to clean this up, but I won't get to it until next week.

Thanks,
Dan

    -----Original Message-----
    *From:* [email protected]
    [mailto:[email protected]]*On

Behalf Of *Dave

Quoted 9 lines

Winchell
*Sent:* Friday, June 06, 2008 4:46 AM
*To:* Keir Fraser; Ben Guthro; xen-devel
*Cc:* [email protected]; Dave Winchell *Subject:* RE: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Keir,

I think the changes are required. We'll run some tests

today today so

Quoted 7 lines

that we have some data to talk about.

-Dave

-----Original Message-----
From: [email protected] on behalf

of Keir Fraser

Quoted 8 lines

Sent: Fri 6/6/2008 4:58 AM
To: Ben Guthro; xen-devel
Cc: [email protected]
Subject: Re: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Are these patches needed now the timers are built on Xen system time rather
than host TSC? Dan has reported much better

time-keeping with his

Quoted 2 lines

patch
checked in, and it¹s for sure a lot less invasive than

this patchset.

Quoted 10 lines

     -- Keir

    On 5/6/08 15:59, "Ben Guthro" <[email protected]> wrote:

    >
    > 1. Introduction
    >
    > This patch improves the hpet based guest clock in

terms of drift and

Quoted 2 lines

Quoted 2 lines

monotonicity.
Prior to this work the drift with hpet was greater

than 2%, far

Quoted 10 lines

above the .05%
> limit
> for ntp to synchronize. With this code, the drift ranges from .001% to .0033%
> depending
> on guest and physical platform.
>
> Using hpet allows guest operating systems to provide monotonic time to their
> applications. Time sources other than hpet are not

monotonic because

Quoted 1 line

Quoted 1 line

of their reliance on tsc, which is not synchronized

across physical

Quoted 31 lines

Quoted 3 lines

processors.

Windows 2k864 and many Linux guests are supported with two

policies, one for
> guests
> that handle missed clock interrupts and the other for guests that require the
> correct number of interrupts.
>
> Guests may use hpet for the timing source even if the physical platform has no
> visible
> hpet. Migration is supported between physical machines which differ in
> physical
> hpet visibility.
>
> Most of the changes are in hpet.c. Two general facilities are added to track
> interrupt
> progress. The ideas here and the facilities would be useful in vpt.c, for
> other time
> sources, though no attempt is made here to improve vpt.c. >
> The following sections discuss hpet dependencies, interrupt delivery policies,
> live migration,
> test results, and relation to recent work with monotonic time. >
>

Quoted 3 lines

> 2. Virtual Hpet dependencies
>
> The virtual hpet depends on the ability to read the

physical or

Quoted 2 lines

simulated
> (see discussion below) hpet. For timekeeping, the

virtual hpet

Quoted 32 lines

also depends
> on two new interrupt notification facilities to implement its policies for
> interrupt delivery.
>
> 2.1. Two modes of low-level hpet main counter reads. >
> In this implementation, the virtual hpet reads with read_64_main_counter(),
> exported by
> time.c, either the real physical hpet main counter register directly or a
> "simulated"
> hpet main counter.
>
> The simulated mode uses a monotonic version of get_s_time() (NOW()), where the
> last
> time value is returned whenever the current time value is less than the last
> time
> value. In simulated mode, since it is layered on s_time, the underlying
> hardware
> can be hpet or some other device. The frequency of the main counter in
> simulated
> mode is the same as the standard physical hpet frequency, allowing live
> migration
> between nodes that are configured differently.
>

Quoted 1 line

> If the physical platform does not have an hpet

device, or if xen

Quoted 9 lines

is configured
> not
> to use the device, then the simulated method is used. If there is a physical
> hpet device,
> and xen has initialized it, then either simulated or physical mode can be
> used.
> This is governed by a boot time option, hpet-avoid.

Setting this

Quoted 3 lines

option to 1
> gives the
> simulated mode and 0 the physical mode. The default

is physical

Quoted 5 lines

mode.
>
> A disadvantage of the physical mode is that may take longer to read the device
> than in simulated mode. On some platforms the cost is

about the

Quoted 15 lines

same (less
> than 250 nsec) for
> physical and simulated modes, while on others physical cost is much higher
> than simulated.
> A disadvantage of the simulated mode is that it can return the same value
> for the counter in consecutive calls.
>
> 2.2. Interrupt notification facilities.
>
> Two interrupt notification facilities are introduced, one is > hvm_isa_irq_assert_cb()
> and the other hvm_register_intr_en_notif().
>

Quoted 5 lines

> The vhpet uses hvm_isa_irq_assert_cb to deliver interrupts to the vioapic.
> hvm_isa_irq_assert_cb allows a callback to be passed along to > vioapic_deliver()
> and this callback is called with a mask of the vcpus

which will

Quoted 4 lines

get the
> interrupt. This callback is made before any vcpus receive an interrupt.
>

Quoted 8 lines

> Vhpet uses hvm_register_intr_en_notif() to register a handler for a particular
> vector that will be called when that vector is injected in > [vmx,svm]_intr_assist()
> and also when the guest finishes handling the interrupt. Here finished is
> defined
> as the point when the guest re-enables interrupts or

lowers the

Quoted 4 lines

tpr value.
> EOI is not used as the end of interrupt as this is sometimes returned before
> the interrupt handler has done its work. A flag is

passed to the

Quoted 3 lines

handler
> indicating
> whether this is the injection point (post = 1) or the

interrupt

Quoted 6 lines

finished (post
> = 0) point.
> The need for the finished point callback is discussed in the missed ticks
> policy section.
>

Quoted 7 lines

> To prevent a possible early trigger of the finished callback, intr_en_notif
> logic
> has a two stage arm, the first at injection (hvm_intr_en_notif_arm()) and the
> second when
> interrupts are seen to be disabled

(hvm_intr_en_notif_disarm()).

Quoted 3 lines

Once fully
> armed, re-enabling
> interrupts will cause hvm_intr_en_notif_disarm() to

make the end

Quoted 2 lines

of interrupt
> callback. hvm_intr_en_notif_arm() and

hvm_intr_en_notif_disarm()

Quoted 6 lines

are called by
> [vmx,svm]_intr_assist().
>
> 3. Interrupt delivery policies
>
> The existing hpet interrupt delivery is preserved.

This includes

Quoted 8 lines

Quoted 1 line

vcpu round robin delivery used by Linux and broadcast delivery

used by
> Windows.
>
> There are two policies for interrupt delivery, one for Windows 2k8-64 and the
> other
> for Linux. The Linux policy takes advantage of the

(guest) Linux

Quoted 3 lines

missed tick
> and offset
> calculations and does not attempt to deliver the

right number of

Quoted 10 lines

interrupts.
> The Windows policy delivers the correct number of interrupts, even if
> sometimes much
> closer to each other than the period. The policies are similar to those in
> vpt.c, though
> there are some important differences.
>
> Policies are selected with an HVMOP_set_param

hypercall with index

Quoted 2 lines

Quoted 2 lines

HVM_PARAM_TIMER_MODE.
Two new values are added,

HVM_HPET_guest_computes_missed_ticks and

Quoted 7 lines

Quoted 1 line

HVM_HPET_guest_does_not_compute_missed_ticks. The reason that

two new ones
> are added is that
> in some guests (32bit Linux) a no-missed policy is needed for clock sources
> other than hpet
> and a missed ticks policy for hpet. It was felt that

there would

Quoted 14 lines

be less
> confusion by simply
> introducing the two hpet policies.
>
> 3.1. The missed ticks policy
>
> The Linux clock interrupt handler for hpet calculates missed ticks and offset
> using the hpet
> main counter. The algorithm works well when the time since the last interrupt
> is greater than
> or equal to a period and poorly otherwise.
>

Quoted 1 line

> The missed ticks policy ensures that no two clock

interrupts are

Quoted 6 lines

delivered to
> the guest at
> a time interval less than a period. A time stamp (hpet main counter value) is
> recorded (by a
> callback registered with hvm_register_intr_en_notif)

when Linux

Quoted 3 lines

finishes
> handling the clock
> interrupt. Then, ensuing interrupts are delivered to

the vioapic

Quoted 9 lines

only if the
> current main
> counter value is a period greater than when the last interrupt was handled.
>
> Tests showed a significant improvement in clock drift with end of interrupt
> time stamps
> versus beginning of interrupt[1]. It is believed that

the reason

Quoted 3 lines

for the
> improvement
> is that the clock interrupt handler goes for a

spinlock and can

Quoted 11 lines

be therefore
> delayed in its
> processing. Furthermore, the main counter is read by the guest under the lock.
> The net
> effect is that if we time stamp injection, we can get the difference in time
> between successive interrupt handler lock acquisitions to be less than the
> period.
>

Quoted 3 lines

> 3.2. The no-missed ticks policy
>
> Windows 2k864 keeps very poor time with the missed

ticks policy.

Quoted 6 lines

So the
> no-missed ticks policy
> was developed. In the no-missed ticks policy we deliver the correct number of
> interrupts,
> even if they are spaced less than a period apart

(when catching up).

Quoted 50 lines

Quoted 2 lines

Windows 2k864 uses a broadcast mode in the interrupt routing

such that
> all vcpus get the clock interrupt. The best Windows drift performance was
> achieved when the
> policy code ensured that all the previous interrupts (on the various vcpus)
> had been injected
> before injecting the next interrupt to the vioapic.. >
> The policy code works as follows. It uses the hvm_isa_irq_assert_cb() to
> record
> the vcpus to be interrupted in h->hpet.pending_mask. Then, in the callback
> registered
> with hvm_register_intr_en_notif() at post=1 time it clears the current vcpu in
> the pending_mask.
> When the pending_mask is clear it decrements hpet.intr_pending_nr and if
> intr_pending_nr is still
> non-zero posts another interrupt to the ioapic with hvm_isa_irq_assert_cb().
> Intr_pending_nr is incremented in hpet_route_decision_not_missed_ticks().
>
> The missed ticks policy intr_en_notif callback also uses the pending_mask
> method. So even though
> Linux does not broadcast its interrupts, the code could handle it if it did.
> In this case the end of interrupt time stamp is made when the pending_mask is
> clear.
>
> 4. Live Migration
>
> Live migration with hpet preserves the current offset of the guest clock with
> respect
> to ntp. This is accomplished by migrating all of the state in the h->hpet data
> structure
> in the usual way. The hp->mc_offset is recalculated on the receiving node so
> that the
> guest sees a continuous hpet main counter.
>

Quoted 6 lines

> Code as been added to xc_domain_save.c to send a small message after the
> domain context is sent. The contents of the message is the physical tsc
> timestamp, last_tsc,
> read just before the message is sent. When the

last_tsc message

Quoted 39 lines

is received in
> xc_domain_restore.c,
> another physical tsc timestamp, cur_tsc, is read. The two timestamps are
> loaded into the domain
> structure as last_tsc_sender and first_tsc_receiver with hypercalls. Then
> xc_domain_hvm_setcontext
> is called so that hpet_load has access to these time stamps. Hpet_load uses
> the timestamps
> to account for the time spent saving and loading the domain context. With this
> technique,
> the only neglected time is the time spent sending a small network message.
>
> 5. Test Results
>
> Some recent test results are:
>
> 5.1 Linux 4u664 and Windows 2k864 load test.
> Duration: 70 hours.
> Test date: 6/2/08
> Loads: usex -b48 on Linux; burn-in on Windows
> Guest vcpus: 8 for Linux; 2 for Windows
> Hardware: 8 physical cpu AMD
> Clock drift : Linux: .0012% Windows: .009%
>
> 5.2 Linux 4u664, Linux 4u464 , and Windows 2k864 no-load test
> Duration: 23 hours.
> Test date: 6/3/08
> Loads: none
> Guest vcpus: 8 for each Linux; 2 for Windows
> Hardware: 4 physical cpu AMD
> Clock drift : Linux: .033% Windows: .019%
>
> 6. Relation to recent work in xen-unstable
>

Quoted 7 lines

> There is a similarity between hvm_get_guest_time() in xen-unstable and
> read_64_main_counter()
> in this code. However, read_64_main_counter() is more tuned to the needs of
> hpet.c. It has no
> "set" operation, only the get. It isolates the mode,

physical or

Quoted 6 lines

simulated, in
> read_64_main_counter()
> itself. It uses no vcpu or domain state as it is a physical entity, in either
> mode. And it provides a real
> physical mode for every read for those applications

that desire

Quoted 5 lines

this.
>
> 7. Conclusion
>
> The virtual hpet is improved by this patch in terms

of accuracy and

Quoted 2 lines

Quoted 2 lines

monotonicity.
Tests performed to date verify this and more testing

is under way.

Quoted 12 lines

Quoted 4 lines

8. Future Work

Testing with Windows Vista will be performed soon. The reason

for accuracy
> variations
> on different platforms using the physical hpet device will be investigated.
> Additional overhead measurements on simulated vs physical hpet mode will be
> made.
>

Quoted 3 lines

Quoted 3 lines

Footnotes:

1. I don't recall the accuracy improvement with end

of interrupt

Quoted 23 lines

stamping, but
> it was
> significant, perhaps better than two to one improvement. It would be a very
> simple matter
> to re-measure the improvement as the facility can call back at injection time
> as well.
>
>
> Signed-off-by: Dave Winchell <[email protected]> > <mailto:[email protected]>
> Signed-off-by: Ben Guthro <[email protected]> > <mailto:[email protected]>
>
>
> _______________________________________________
> Xen-devel mailing list
> [email protected]
> http://lists.xensource.com/xen-devel

Quoted 1 line

Quoted 1 line

4) config file parameters

Hpet enabled. Timer_mode set to HVM_HPET_guest_computes_missed_ticks for all Linux guests and to HVM_HPET_guest_does_not_compute_missed_ticks for Windows 2k8-64 and Vista 64.
8 vcpus for Linux and 2 for Windows.

These new HVM_HPET options seems a weird design choice. It appears that you can only set these or one of the old options, so there¹s not actually any independence between the mode used by vpt.c and the mode used by hpet.c. At guest install time you ought to be able to tell whether the guest will use hpet or not based on its version (RHELx, SLESy, Winz etc etc) and decide whether missed-ticks accounting is required or not.

I¹d be more agreeable to a partch that stripped out the physical hpet accesses (since you say they are not the reason for the improvement in accuracy), built hpet on top of vpt, and added the necessary extra mechanisms to deal with interrupt broadcasts into vpt.c. And which was split into more separate pieces of mechanism.

Quoted 1 line

Quoted 1 line

4) config file parameters

Hpet enabled. Timer_mode set to HVM_HPET_guest_computes_missed_ticks for all Linux guests and to HVM_HPET_guest_does_not_compute_missed_ticks for Windows 2k8-64 and Vista 64.
8 vcpus for Linux and 2 for Windows.

Quoted 6 lines

Quoted 4 lines

I¹d be more agreeable to a partch that stripped out the physical hpet accesses (since you say they are not the reason for the improvement in accuracy), built hpet on top of vpt, and added the necessary extra mechanisms to deal with interrupt broadcasts into vpt.c. And which was

split

Quoted 1 line

into more separate pieces of mechanism.

ok, I'll work on this. How much time do I have to make the release you are working on?

Quoted 2 lines

ok, I'll work on this. How much time do I have to make the release you are working on?

I¹m thinking feature freeze at the end of the month.

On 9/6/08 13:03, "Keir Fraser" <[email protected]> wrote:

ok, I'll work on this. How much time do I have to make the release you are
working on?

I’m thinking feature freeze at the end of the month.

Oh, while I’m commenting on the current version of the patch, I will point out that changes to the save/restore format need careful thought. At the very least we’d like to be backward compatible (old images restore on new Xen) even if we don’t achieve compatibility the other way round. I didn’t really look too closely at this aspect of the patches so perhaps these patches are fine in this regard.

Either way I think the core re-architecting of the hpet device model to handle missed ticks and so on is independent of interface changes anyway (apart from reasonable extensions to the timer_mode option). Changes to save/restore format, addition of extra debugging code, and peripheral things like that it’d be nice to have in separate patches. It makes the core stuff easier to review and more likely to get accepted without quibble (because there’s less of it, and it is all dedicated to a single purpose which I think we all agree is where we want to be).

Thanks,
Keir

At guest install time you ought to be able to tell whether the guest will use hpet or not based on its version (RHELx, SLESy, Winz etc etc) and decide whether missed-ticks accounting is required or not.

4) config file parameters

Possibly there are bugs in the hpet device model which are fixed by Dave's patch. If this is actually the case, it would be nice to break those out as separate patches, as I think an 11% drift must largely be due to device-model bugs rather than relatively insignificant differences between hvm_get_guest_time() and physical HPET main counter.

Although hwhpet=1 is a fine alternative in many cases, it may be unavailable on some systems and may cause significant performance issues on others. So I think we will still need to track down the poor accuracy when hwhpet=0. And if for some reason Xen system time can't be made accurate enough (< 0.05%), then I think we should consider building Xen system time itself on top of hardware hpet instead of TSC... at least when Xen discovers a capable hpet.

Or maybe there's a computation error somewhere in the hvm hpet scaling code? Hmmm...

Quoted 3 lines

At guest install time you ought to be able to tell whether the guest will use hpet or not based on its version (RHELx, SLESy, Winz etc etc) and decide whether missed-ticks accounting is required or not.

Unfortunately this is not true on Linux, at least without gathering (and hardcoding) more information about the system. Whether hpet is used or not is dependent not only on the OS/version and hvm config parameters, but also on kernel command line parameters and even the underlying CPU. For example, on RHEL5u1, if the tsc is synchronized and the CPU is Intel, and no kernel parameters are chosen, tsc will be chosen as the default clocksource even if hpet is present. Ugly.

I'm wondering what is "magic" about 0.03% in all the non-hw-hpet measurements.

Is that just the accuracy of the underlying tsc
on your test system, e.g. the skew of tsc relative to an external (ntp) source? Or is Xen (tsc-based) system time skewing that much on an overcommitted system (and skewing much less than 0.03% on an unloaded system)?

Running the following on dom0 both on an unloaded and overcommitted system (with ntpd off in dom0 and all guests) might be interesting:

# ntpdate $NTPSERVER; sleep 3600; ntpdate -q $NTPSERVER

-----Original Message-----
From: Dave Winchell [mailto:[email protected]]
Sent: Saturday, June 07, 2008 3:21 PM
To: Keir Fraser; [email protected]
Cc: Ben Guthro; xen-devel; Dave Winchell
Subject: RE: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Quoted 7 lines

Possibly there are bugs in the hpet device model which are fixed by Dave's patch. If this is actually the case, it would be nice to break those out as separate patches, as I think an 11% drift must largely be due to device-model bugs rather than relatively insignificant differences between hvm_get_guest_time() and physical HPET main counter.

Hi Keir,

I tried an experiment on Friday where I short circuited the missed ticks policy code in the hpet.c patch, but used the physical hpet each access. The result for Linux was a drift of .1%, same as the xen-unstable bits.

Conversely I get very good drift numbers, i.e., under .03%, when using the missed ticks policy code and running in simulated mode (layered on stime) when stime uses hpet.

So clearly, the improvement from .1% to .03% is due to the policy code. I haven't run the short circuit test with the windows policy but I can do that on Monday.

Note: For Windows and Linux I get < .03% drift using the policy code and running simulated mode whether stime is using hpet or some other device.

regards,
Dave

-----Original Message-----
From: Keir Fraser [mailto:[email protected]]
Sent: Fri 6/6/2008 6:34 PM
To: [email protected]; Dave Winchell
Cc: Ben Guthro; xen-devel
Subject: Re: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Quoted 10 lines · Dan Magenheimer

Although hwhpet=1 is a fine alternative in many cases, it may be unavailable on some systems and may cause significant performance issues on others. So I think we will still need to track down the poor accuracy when hwhpet=0. And if for some reason Xen system time can't be made accurate enough (< 0.05%), then I think we should consider building Xen system time itself on top of hardware hpet instead of TSC... at least when Xen discovers a capable hpet.

Yes, this would be a sensible extra timer_mode: have hvm_get_guest_time() call to the platform time read function, and bypass TSC altogether. This would be cleaner than having only the vHPET code punch through to the physical HPET: instead we have the boot-time chosen platform timesource used by all virtual timers.

Quoted 4 lines

Or maybe there's a computation error somewhere in the hvm hpet scaling code? Hmmm...

Possibly there are bugs in the hpet device model which are fixed by Dave's patch. If this is actually the case, it would be nice to break those out as separate patches, as I think an 11% drift must largely be due to device-model bugs rather than relatively insignificant differences between hvm_get_guest_time() and physical HPET main counter.

-- Keir

Quoted 1 line

Quoted 1 line

At guest install time you ought to be able to tell whether

the guest

Quoted 1 line

Quoted 1 line

will use hpet or not based on its version (RHELx, SLESy,

Winz etc etc)

Quoted 7 lines

Quoted 1 line

and decide whether missed-ticks accounting is required or not.

Unfortunately this is not true on Linux, at least without gathering (and hardcoding) more information about the system. Whether hpet is used or not is dependent not only on the OS/version and hvm config parameters, but also on kernel command line parameters and even the underlying CPU. For example, on RHEL5u1, if the tsc is

synchronized

Quoted 1 line

and the CPU is Intel, and no kernel parameters are chosen,

tsc will be

Quoted 1 line

chosen as the default clocksource even if hpet is present. Ugly.

It's not immediately obvious that adding further independent configuration
knobs to twiddle would make our lives that much easier. However it certainly
increases the test matrix.

In your example above, by synchronised TSC do you mean constant-rate TSC?
That can at least be hidden in CPUID now.

The Linux policy is more subtle, but is required to go from .1% to .03%.

While I am fully supportive of offering hardware hpet as an option for hvm guests (let's call it hwhpet=1 for shorthand), I am very surprised by your preliminary results; the most obvious conclusion is that Xen system time is losing time at the rate of 1000 PPM though its possible there's a bug somewhere else in the "time stack". Your Windows result is jaw-dropping and inexplicable, though I have to admit ignorance of how Windows manages time.

I think with my recent patch and hpet=1 (essentially the same as your emulated hpet), hvm guest time should track Xen system time. I wonder if domain0 (which if I understand correctly is directly using Xen system time) is also seeing an error of .1%? Also I wonder for the skew you are seeing (in both hvm guests and domain0) is time moving too fast or two slow?

Although hwhpet=1 is a fine alternative in many cases, it may be unavailable on some systems and may cause significant performance issues on others. So I think we will still need to track down the poor accuracy when hwhpet=0.

And if for some reason
Xen system time can't be made accurate enough (< 0.05%), then I think we should consider building Xen system time itself on top of hardware hpet instead of TSC... at least when Xen discovers a capable hpet.

One more thought... do you know the accuracy of the TSC crystals on your test systems? I posted a patch awhile ago that was intended to test that, though I guess it was only testing skew of different TSCs on the same system, not TSCs against an external time source.

Or maybe there's a computation error somewhere in the hvm hpet scaling code? Hmmm...

-----Original Message-----
From: Dave Winchell [mailto:[email protected]]
Sent: Friday, June 06, 2008 1:33 PM
To: [email protected]; Keir Fraser
Cc: Ben Guthro; xen-devel; Dave Winchell
Subject: Re: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Dan, Keir:

Preliminary tests results indicate an error of .1% for Linux 64 bit guests configured
for hpet with xen-unstable as is. As we have discussed many times, the ntp requirement is .05%.
Tests on the patch we just submitted for hpet have indicated errors of .0012%
on this platform under similar test conditions and .03% on other platforms.

Windows vista64 has an error of 11% using hpet with the xen-unstable bits.
In an overnight test with our hpet patch, the Windows vista error was .008%.

The tests are with two or three guests on a physical node, all under load, and with
the ratio of vcpus to phys cpus > 1.

I will continue to run tests over the next few days.

thanks,
Dave

Quoted 3 lines · Dan Magenheimer

Hi Dave and Ben --

When running tests on xen-unstable (without your patch),

please ensure

Quoted 2 lines

that hpet=1 is set in the hvm config and also I think that when hpet is the clocksource on RHEL4-32, the clock IS resilient to

missed ticks

Quoted 1 line

so timer_mode should be 2 (vs when pit is the clocksource

on RHEL4-32,

Quoted 4 lines

all clock ticks must be delivered and so timer_mode should be 0).

Per

http://lists.xensource.com/archives/html/xen-devel/2008-06/msg
00098.html it's

Quoted 8 lines

my intent to clean this up, but I won't get to it until next week.

Thanks,
Dan

    -----Original Message-----
    *From:* [email protected]
    [mailto:[email protected]]*On

Behalf Of *Dave

Quoted 9 lines

Winchell
*Sent:* Friday, June 06, 2008 4:46 AM
*To:* Keir Fraser; Ben Guthro; xen-devel
*Cc:* [email protected]; Dave Winchell *Subject:* RE: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Keir,

I think the changes are required. We'll run some tests

today today so

Quoted 7 lines

that we have some data to talk about.

-Dave

-----Original Message-----
From: [email protected] on behalf

of Keir Fraser

Quoted 8 lines

Sent: Fri 6/6/2008 4:58 AM
To: Ben Guthro; xen-devel
Cc: [email protected]
Subject: Re: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Are these patches needed now the timers are built on Xen system time rather
than host TSC? Dan has reported much better

time-keeping with his

Quoted 2 lines

patch
checked in, and it¹s for sure a lot less invasive than

this patchset.

Quoted 10 lines

     -- Keir

    On 5/6/08 15:59, "Ben Guthro" <[email protected]> wrote:

    >
    > 1. Introduction
    >
    > This patch improves the hpet based guest clock in

terms of drift and

Quoted 2 lines

Quoted 2 lines

monotonicity.
Prior to this work the drift with hpet was greater

than 2%, far

Quoted 10 lines

above the .05%
> limit
> for ntp to synchronize. With this code, the drift ranges from .001% to .0033%
> depending
> on guest and physical platform.
>
> Using hpet allows guest operating systems to provide monotonic time to their
> applications. Time sources other than hpet are not

monotonic because

Quoted 1 line

Quoted 1 line

of their reliance on tsc, which is not synchronized

across physical

Quoted 31 lines

Quoted 3 lines

processors.

Windows 2k864 and many Linux guests are supported with two

policies, one for
> guests
> that handle missed clock interrupts and the other for guests that require the
> correct number of interrupts.
>
> Guests may use hpet for the timing source even if the physical platform has no
> visible
> hpet. Migration is supported between physical machines which differ in
> physical
> hpet visibility.
>
> Most of the changes are in hpet.c. Two general facilities are added to track
> interrupt
> progress. The ideas here and the facilities would be useful in vpt.c, for
> other time
> sources, though no attempt is made here to improve vpt.c. >
> The following sections discuss hpet dependencies, interrupt delivery policies,
> live migration,
> test results, and relation to recent work with monotonic time. >
>

Quoted 3 lines

> 2. Virtual Hpet dependencies
>
> The virtual hpet depends on the ability to read the

physical or

Quoted 2 lines

simulated
> (see discussion below) hpet. For timekeeping, the

virtual hpet

Quoted 32 lines

also depends
> on two new interrupt notification facilities to implement its policies for
> interrupt delivery.
>
> 2.1. Two modes of low-level hpet main counter reads. >
> In this implementation, the virtual hpet reads with read_64_main_counter(),
> exported by
> time.c, either the real physical hpet main counter register directly or a
> "simulated"
> hpet main counter.
>
> The simulated mode uses a monotonic version of get_s_time() (NOW()), where the
> last
> time value is returned whenever the current time value is less than the last
> time
> value. In simulated mode, since it is layered on s_time, the underlying
> hardware
> can be hpet or some other device. The frequency of the main counter in
> simulated
> mode is the same as the standard physical hpet frequency, allowing live
> migration
> between nodes that are configured differently.
>

Quoted 1 line

> If the physical platform does not have an hpet

device, or if xen

Quoted 9 lines

is configured
> not
> to use the device, then the simulated method is used. If there is a physical
> hpet device,
> and xen has initialized it, then either simulated or physical mode can be
> used.
> This is governed by a boot time option, hpet-avoid.

Setting this

Quoted 3 lines

option to 1
> gives the
> simulated mode and 0 the physical mode. The default

is physical

Quoted 5 lines

mode.
>
> A disadvantage of the physical mode is that may take longer to read the device
> than in simulated mode. On some platforms the cost is

about the

Quoted 15 lines

same (less
> than 250 nsec) for
> physical and simulated modes, while on others physical cost is much higher
> than simulated.
> A disadvantage of the simulated mode is that it can return the same value
> for the counter in consecutive calls.
>
> 2.2. Interrupt notification facilities.
>
> Two interrupt notification facilities are introduced, one is > hvm_isa_irq_assert_cb()
> and the other hvm_register_intr_en_notif().
>

Quoted 5 lines

> The vhpet uses hvm_isa_irq_assert_cb to deliver interrupts to the vioapic.
> hvm_isa_irq_assert_cb allows a callback to be passed along to > vioapic_deliver()
> and this callback is called with a mask of the vcpus

which will

Quoted 4 lines

get the
> interrupt. This callback is made before any vcpus receive an interrupt.
>

Quoted 8 lines

> Vhpet uses hvm_register_intr_en_notif() to register a handler for a particular
> vector that will be called when that vector is injected in > [vmx,svm]_intr_assist()
> and also when the guest finishes handling the interrupt. Here finished is
> defined
> as the point when the guest re-enables interrupts or

lowers the

Quoted 4 lines

tpr value.
> EOI is not used as the end of interrupt as this is sometimes returned before
> the interrupt handler has done its work. A flag is

passed to the

Quoted 3 lines

handler
> indicating
> whether this is the injection point (post = 1) or the

interrupt

Quoted 6 lines

finished (post
> = 0) point.
> The need for the finished point callback is discussed in the missed ticks
> policy section.
>

Quoted 7 lines

> To prevent a possible early trigger of the finished callback, intr_en_notif
> logic
> has a two stage arm, the first at injection (hvm_intr_en_notif_arm()) and the
> second when
> interrupts are seen to be disabled

(hvm_intr_en_notif_disarm()).

Quoted 3 lines

Once fully
> armed, re-enabling
> interrupts will cause hvm_intr_en_notif_disarm() to

make the end

Quoted 2 lines

of interrupt
> callback. hvm_intr_en_notif_arm() and

hvm_intr_en_notif_disarm()

Quoted 6 lines

are called by
> [vmx,svm]_intr_assist().
>
> 3. Interrupt delivery policies
>
> The existing hpet interrupt delivery is preserved.

This includes

Quoted 8 lines

Quoted 1 line

vcpu round robin delivery used by Linux and broadcast delivery

used by
> Windows.
>
> There are two policies for interrupt delivery, one for Windows 2k8-64 and the
> other
> for Linux. The Linux policy takes advantage of the

(guest) Linux

Quoted 3 lines

missed tick
> and offset
> calculations and does not attempt to deliver the

right number of

Quoted 10 lines

interrupts.
> The Windows policy delivers the correct number of interrupts, even if
> sometimes much
> closer to each other than the period. The policies are similar to those in
> vpt.c, though
> there are some important differences.
>
> Policies are selected with an HVMOP_set_param

hypercall with index

Quoted 2 lines

Quoted 2 lines

HVM_PARAM_TIMER_MODE.
Two new values are added,

HVM_HPET_guest_computes_missed_ticks and

Quoted 7 lines

Quoted 1 line

HVM_HPET_guest_does_not_compute_missed_ticks. The reason that

two new ones
> are added is that
> in some guests (32bit Linux) a no-missed policy is needed for clock sources
> other than hpet
> and a missed ticks policy for hpet. It was felt that

there would

Quoted 14 lines

be less
> confusion by simply
> introducing the two hpet policies.
>
> 3.1. The missed ticks policy
>
> The Linux clock interrupt handler for hpet calculates missed ticks and offset
> using the hpet
> main counter. The algorithm works well when the time since the last interrupt
> is greater than
> or equal to a period and poorly otherwise.
>

Quoted 1 line

> The missed ticks policy ensures that no two clock

interrupts are

Quoted 6 lines

delivered to
> the guest at
> a time interval less than a period. A time stamp (hpet main counter value) is
> recorded (by a
> callback registered with hvm_register_intr_en_notif)

when Linux

Quoted 3 lines

finishes
> handling the clock
> interrupt. Then, ensuing interrupts are delivered to

the vioapic

Quoted 9 lines

only if the
> current main
> counter value is a period greater than when the last interrupt was handled.
>
> Tests showed a significant improvement in clock drift with end of interrupt
> time stamps
> versus beginning of interrupt[1]. It is believed that

the reason

Quoted 3 lines

for the
> improvement
> is that the clock interrupt handler goes for a

spinlock and can

Quoted 11 lines

be therefore
> delayed in its
> processing. Furthermore, the main counter is read by the guest under the lock.
> The net
> effect is that if we time stamp injection, we can get the difference in time
> between successive interrupt handler lock acquisitions to be less than the
> period.
>

Quoted 3 lines

> 3.2. The no-missed ticks policy
>
> Windows 2k864 keeps very poor time with the missed

ticks policy.

Quoted 6 lines

So the
> no-missed ticks policy
> was developed. In the no-missed ticks policy we deliver the correct number of
> interrupts,
> even if they are spaced less than a period apart

(when catching up).

Quoted 50 lines

Quoted 2 lines

Windows 2k864 uses a broadcast mode in the interrupt routing

such that
> all vcpus get the clock interrupt. The best Windows drift performance was
> achieved when the
> policy code ensured that all the previous interrupts (on the various vcpus)
> had been injected
> before injecting the next interrupt to the vioapic.. >
> The policy code works as follows. It uses the hvm_isa_irq_assert_cb() to
> record
> the vcpus to be interrupted in h->hpet.pending_mask. Then, in the callback
> registered
> with hvm_register_intr_en_notif() at post=1 time it clears the current vcpu in
> the pending_mask.
> When the pending_mask is clear it decrements hpet.intr_pending_nr and if
> intr_pending_nr is still
> non-zero posts another interrupt to the ioapic with hvm_isa_irq_assert_cb().
> Intr_pending_nr is incremented in hpet_route_decision_not_missed_ticks().
>
> The missed ticks policy intr_en_notif callback also uses the pending_mask
> method. So even though
> Linux does not broadcast its interrupts, the code could handle it if it did.
> In this case the end of interrupt time stamp is made when the pending_mask is
> clear.
>
> 4. Live Migration
>
> Live migration with hpet preserves the current offset of the guest clock with
> respect
> to ntp. This is accomplished by migrating all of the state in the h->hpet data
> structure
> in the usual way. The hp->mc_offset is recalculated on the receiving node so
> that the
> guest sees a continuous hpet main counter.
>

Quoted 6 lines

> Code as been added to xc_domain_save.c to send a small message after the
> domain context is sent. The contents of the message is the physical tsc
> timestamp, last_tsc,
> read just before the message is sent. When the

last_tsc message

Quoted 39 lines

is received in
> xc_domain_restore.c,
> another physical tsc timestamp, cur_tsc, is read. The two timestamps are
> loaded into the domain
> structure as last_tsc_sender and first_tsc_receiver with hypercalls. Then
> xc_domain_hvm_setcontext
> is called so that hpet_load has access to these time stamps. Hpet_load uses
> the timestamps
> to account for the time spent saving and loading the domain context. With this
> technique,
> the only neglected time is the time spent sending a small network message.
>
> 5. Test Results
>
> Some recent test results are:
>
> 5.1 Linux 4u664 and Windows 2k864 load test.
> Duration: 70 hours.
> Test date: 6/2/08
> Loads: usex -b48 on Linux; burn-in on Windows
> Guest vcpus: 8 for Linux; 2 for Windows
> Hardware: 8 physical cpu AMD
> Clock drift : Linux: .0012% Windows: .009%
>
> 5.2 Linux 4u664, Linux 4u464 , and Windows 2k864 no-load test
> Duration: 23 hours.
> Test date: 6/3/08
> Loads: none
> Guest vcpus: 8 for each Linux; 2 for Windows
> Hardware: 4 physical cpu AMD
> Clock drift : Linux: .033% Windows: .019%
>
> 6. Relation to recent work in xen-unstable
>

Quoted 7 lines

> There is a similarity between hvm_get_guest_time() in xen-unstable and
> read_64_main_counter()
> in this code. However, read_64_main_counter() is more tuned to the needs of
> hpet.c. It has no
> "set" operation, only the get. It isolates the mode,

physical or

Quoted 6 lines

simulated, in
> read_64_main_counter()
> itself. It uses no vcpu or domain state as it is a physical entity, in either
> mode. And it provides a real
> physical mode for every read for those applications

that desire

Quoted 5 lines

this.
>
> 7. Conclusion
>
> The virtual hpet is improved by this patch in terms

of accuracy and

Quoted 2 lines

Quoted 2 lines

monotonicity.
Tests performed to date verify this and more testing

is under way.

Quoted 12 lines

Quoted 4 lines

8. Future Work

Testing with Windows Vista will be performed soon. The reason

for accuracy
> variations
> on different platforms using the physical hpet device will be investigated.
> Additional overhead measurements on simulated vs physical hpet mode will be
> made.
>

Quoted 3 lines

> Footnotes:
>
> 1. I don't recall the accuracy improvement with end

of interrupt

Quoted 23 lines

stamping, but
> it was
> significant, perhaps better than two to one improvement. It would be a very
> simple matter
> to re-measure the improvement as the facility can call back at injection time
> as well.
>
>
> Signed-off-by: Dave Winchell <[email protected]> > <mailto:[email protected]>
> Signed-off-by: Ben Guthro <[email protected]> > <mailto:[email protected]>
>
>
> _______________________________________________
> Xen-devel mailing list
> [email protected]
> http://lists.xensource.com/xen-devel

Quoted 2 lines

The Linux policy is more subtle, but is required to go from .1% to .03%.

Thanks for the good documentation which I hadn't thoroughly read until now.
I now understand that the essence of your
hpet missed ticks policy is to ensure that ticks are never delivered too close together. But I'm trying to understand WHY your patch works, in other words, what problem it is countering.

I care about this for more reasons than just
because it is interesting: (1) I'd like to feel confident that it is fixing a bug rather than just a symptom of a bug; and (2) I wonder how universally it is applicable.

I see from code examination in mark_offset_hpet() in RHEL4u5/arch/i386/kernel/timers/timer_hpet.c, that the correction for lost ticks is just plain wrong in a virtual environment. (Suppose for example that a virtual tick was delivered every 1.999*hpet_tick... I think the clock would be off by 50%!) Is this the bug that is being "countered" by your policy?

However, the lost tick handling in RHEL5u1/kernel/timer.c (which I think is used also for hpet) is much better so I am eager to find out if your policy works there too.
If the hpet missed tick policy works for both, though, I should be happy, though I wonder about upstream kernels (e.g. the trend toward tickless).

That said, I'd rather
see this get into Xen 3.3 and worry about upstream kernels later :-)

The Linux policy is more subtle, but is required to go from .1% to .03%.

While I am fully supportive of offering hardware hpet as an option for hvm guests (let's call it hwhpet=1 for shorthand), I am very surprised by your preliminary results; the most obvious conclusion is that Xen system time is losing time at the rate of 1000 PPM though its possible there's a bug somewhere else in the "time stack". Your Windows result is jaw-dropping and inexplicable, though I have to admit ignorance of how Windows manages time.

I think with my recent patch and hpet=1 (essentially the same as your emulated hpet), hvm guest time should track Xen system time. I wonder if domain0 (which if I understand correctly is directly using Xen system time) is also seeing an error of .1%? Also I wonder for the skew you are seeing (in both hvm guests and domain0) is time moving too fast or two slow?

Although hwhpet=1 is a fine alternative in many cases, it may be unavailable on some systems and may cause significant performance issues on others. So I think we will still need to track down the poor accuracy when hwhpet=0.

And if for some reason
Xen system time can't be made accurate enough (< 0.05%), then I think we should consider building Xen system time itself on top of hardware hpet instead of TSC... at least when Xen discovers a capable hpet.

One more thought... do you know the accuracy of the TSC crystals on your test systems? I posted a patch awhile ago that was intended to test that, though I guess it was only testing skew of different TSCs on the same system, not TSCs against an external time source.

Or maybe there's a computation error somewhere in the hvm hpet scaling code? Hmmm...

-----Original Message-----
From: Dave Winchell [mailto:[email protected]]
Sent: Friday, June 06, 2008 1:33 PM
To: [email protected]; Keir Fraser
Cc: Ben Guthro; xen-devel; Dave Winchell
Subject: Re: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Dan, Keir:

Preliminary tests results indicate an error of .1% for Linux 64 bit guests configured
for hpet with xen-unstable as is. As we have discussed many times, the ntp requirement is .05%.
Tests on the patch we just submitted for hpet have indicated errors of .0012%
on this platform under similar test conditions and .03% on other platforms.

Windows vista64 has an error of 11% using hpet with the xen-unstable bits.
In an overnight test with our hpet patch, the Windows vista error was .008%.

The tests are with two or three guests on a physical node, all under load, and with
the ratio of vcpus to phys cpus > 1.

I will continue to run tests over the next few days.

thanks,
Dave

Quoted 3 lines · Dan Magenheimer

Hi Dave and Ben --

When running tests on xen-unstable (without your patch),

please ensure

Quoted 2 lines

that hpet=1 is set in the hvm config and also I think that when hpet is the clocksource on RHEL4-32, the clock IS resilient to

missed ticks

Quoted 1 line

so timer_mode should be 2 (vs when pit is the clocksource

on RHEL4-32,

Quoted 4 lines

all clock ticks must be delivered and so timer_mode should be 0).

Per

http://lists.xensource.com/archives/html/xen-devel/2008-06/msg
00098.html it's

Quoted 8 lines

my intent to clean this up, but I won't get to it until next week.

Thanks,
Dan

    -----Original Message-----
    *From:* [email protected]
    [mailto:[email protected]]*On

Behalf Of *Dave

Quoted 9 lines

Winchell
*Sent:* Friday, June 06, 2008 4:46 AM
*To:* Keir Fraser; Ben Guthro; xen-devel
*Cc:* [email protected]; Dave Winchell *Subject:* RE: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Keir,

I think the changes are required. We'll run some tests

today today so

Quoted 7 lines

that we have some data to talk about.

-Dave

-----Original Message-----
From: [email protected] on behalf

of Keir Fraser

Quoted 8 lines

Sent: Fri 6/6/2008 4:58 AM
To: Ben Guthro; xen-devel
Cc: [email protected]
Subject: Re: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Are these patches needed now the timers are built on Xen system time rather
than host TSC? Dan has reported much better

time-keeping with his

Quoted 2 lines

patch
checked in, and it¹s for sure a lot less invasive than

this patchset.

Quoted 10 lines

     -- Keir

    On 5/6/08 15:59, "Ben Guthro" <[email protected]> wrote:

    >
    > 1. Introduction
    >
    > This patch improves the hpet based guest clock in

terms of drift and

Quoted 2 lines

Quoted 2 lines

monotonicity.
Prior to this work the drift with hpet was greater

than 2%, far

Quoted 10 lines

above the .05%
> limit
> for ntp to synchronize. With this code, the drift ranges from .001% to .0033%
> depending
> on guest and physical platform.
>
> Using hpet allows guest operating systems to provide monotonic time to their
> applications. Time sources other than hpet are not

monotonic because

Quoted 1 line

Quoted 1 line

of their reliance on tsc, which is not synchronized

across physical

Quoted 31 lines

Quoted 3 lines

processors.

Windows 2k864 and many Linux guests are supported with two

policies, one for
> guests
> that handle missed clock interrupts and the other for guests that require the
> correct number of interrupts.
>
> Guests may use hpet for the timing source even if the physical platform has no
> visible
> hpet. Migration is supported between physical machines which differ in
> physical
> hpet visibility.
>
> Most of the changes are in hpet.c. Two general facilities are added to track
> interrupt
> progress. The ideas here and the facilities would be useful in vpt.c, for
> other time
> sources, though no attempt is made here to improve vpt.c. >
> The following sections discuss hpet dependencies, interrupt delivery policies,
> live migration,
> test results, and relation to recent work with monotonic time. >
>

Quoted 3 lines

> 2. Virtual Hpet dependencies
>
> The virtual hpet depends on the ability to read the

physical or

Quoted 2 lines

simulated
> (see discussion below) hpet. For timekeeping, the

virtual hpet

Quoted 32 lines

also depends
> on two new interrupt notification facilities to implement its policies for
> interrupt delivery.
>
> 2.1. Two modes of low-level hpet main counter reads. >
> In this implementation, the virtual hpet reads with read_64_main_counter(),
> exported by
> time.c, either the real physical hpet main counter register directly or a
> "simulated"
> hpet main counter.
>
> The simulated mode uses a monotonic version of get_s_time() (NOW()), where the
> last
> time value is returned whenever the current time value is less than the last
> time
> value. In simulated mode, since it is layered on s_time, the underlying
> hardware
> can be hpet or some other device. The frequency of the main counter in
> simulated
> mode is the same as the standard physical hpet frequency, allowing live
> migration
> between nodes that are configured differently.
>

Quoted 1 line

> If the physical platform does not have an hpet

device, or if xen

Quoted 9 lines

is configured
> not
> to use the device, then the simulated method is used. If there is a physical
> hpet device,
> and xen has initialized it, then either simulated or physical mode can be
> used.
> This is governed by a boot time option, hpet-avoid.

Setting this

Quoted 3 lines

option to 1
> gives the
> simulated mode and 0 the physical mode. The default

is physical

Quoted 5 lines

mode.
>
> A disadvantage of the physical mode is that may take longer to read the device
> than in simulated mode. On some platforms the cost is

about the

Quoted 15 lines

same (less
> than 250 nsec) for
> physical and simulated modes, while on others physical cost is much higher
> than simulated.
> A disadvantage of the simulated mode is that it can return the same value
> for the counter in consecutive calls.
>
> 2.2. Interrupt notification facilities.
>
> Two interrupt notification facilities are introduced, one is > hvm_isa_irq_assert_cb()
> and the other hvm_register_intr_en_notif().
>

Quoted 5 lines

> The vhpet uses hvm_isa_irq_assert_cb to deliver interrupts to the vioapic.
> hvm_isa_irq_assert_cb allows a callback to be passed along to > vioapic_deliver()
> and this callback is called with a mask of the vcpus

which will

Quoted 4 lines

get the
> interrupt. This callback is made before any vcpus receive an interrupt.
>

Quoted 8 lines

> Vhpet uses hvm_register_intr_en_notif() to register a handler for a particular
> vector that will be called when that vector is injected in > [vmx,svm]_intr_assist()
> and also when the guest finishes handling the interrupt. Here finished is
> defined
> as the point when the guest re-enables interrupts or

lowers the

Quoted 4 lines

tpr value.
> EOI is not used as the end of interrupt as this is sometimes returned before
> the interrupt handler has done its work. A flag is

passed to the

Quoted 3 lines

handler
> indicating
> whether this is the injection point (post = 1) or the

interrupt

Quoted 6 lines

finished (post
> = 0) point.
> The need for the finished point callback is discussed in the missed ticks
> policy section.
>

Quoted 7 lines

> To prevent a possible early trigger of the finished callback, intr_en_notif
> logic
> has a two stage arm, the first at injection (hvm_intr_en_notif_arm()) and the
> second when
> interrupts are seen to be disabled

(hvm_intr_en_notif_disarm()).

Quoted 3 lines

Once fully
> armed, re-enabling
> interrupts will cause hvm_intr_en_notif_disarm() to

make the end

Quoted 2 lines

of interrupt
> callback. hvm_intr_en_notif_arm() and

hvm_intr_en_notif_disarm()

Quoted 6 lines

are called by
> [vmx,svm]_intr_assist().
>
> 3. Interrupt delivery policies
>
> The existing hpet interrupt delivery is preserved.

This includes

Quoted 8 lines

Quoted 1 line

vcpu round robin delivery used by Linux and broadcast delivery

used by
> Windows.
>
> There are two policies for interrupt delivery, one for Windows 2k8-64 and the
> other
> for Linux. The Linux policy takes advantage of the

(guest) Linux

Quoted 3 lines

missed tick
> and offset
> calculations and does not attempt to deliver the

right number of

Quoted 10 lines

interrupts.
> The Windows policy delivers the correct number of interrupts, even if
> sometimes much
> closer to each other than the period. The policies are similar to those in
> vpt.c, though
> there are some important differences.
>
> Policies are selected with an HVMOP_set_param

hypercall with index

Quoted 2 lines

Quoted 2 lines

HVM_PARAM_TIMER_MODE.
Two new values are added,

HVM_HPET_guest_computes_missed_ticks and

Quoted 7 lines

Quoted 1 line

HVM_HPET_guest_does_not_compute_missed_ticks. The reason that

two new ones
> are added is that
> in some guests (32bit Linux) a no-missed policy is needed for clock sources
> other than hpet
> and a missed ticks policy for hpet. It was felt that

there would

Quoted 14 lines

be less
> confusion by simply
> introducing the two hpet policies.
>
> 3.1. The missed ticks policy
>
> The Linux clock interrupt handler for hpet calculates missed ticks and offset
> using the hpet
> main counter. The algorithm works well when the time since the last interrupt
> is greater than
> or equal to a period and poorly otherwise.
>

Quoted 1 line

> The missed ticks policy ensures that no two clock

interrupts are

Quoted 6 lines

delivered to
> the guest at
> a time interval less than a period. A time stamp (hpet main counter value) is
> recorded (by a
> callback registered with hvm_register_intr_en_notif)

when Linux

Quoted 3 lines

finishes
> handling the clock
> interrupt. Then, ensuing interrupts are delivered to

the vioapic

Quoted 9 lines

only if the
> current main
> counter value is a period greater than when the last interrupt was handled.
>
> Tests showed a significant improvement in clock drift with end of interrupt
> time stamps
> versus beginning of interrupt[1]. It is believed that

the reason

Quoted 3 lines

for the
> improvement
> is that the clock interrupt handler goes for a

spinlock and can

Quoted 11 lines

be therefore
> delayed in its
> processing. Furthermore, the main counter is read by the guest under the lock.
> The net
> effect is that if we time stamp injection, we can get the difference in time
> between successive interrupt handler lock acquisitions to be less than the
> period.
>

Quoted 3 lines

> 3.2. The no-missed ticks policy
>
> Windows 2k864 keeps very poor time with the missed

ticks policy.

Quoted 6 lines

So the
> no-missed ticks policy
> was developed. In the no-missed ticks policy we deliver the correct number of
> interrupts,
> even if they are spaced less than a period apart

(when catching up).

Quoted 50 lines

Quoted 2 lines

Windows 2k864 uses a broadcast mode in the interrupt routing

such that
> all vcpus get the clock interrupt. The best Windows drift performance was
> achieved when the
> policy code ensured that all the previous interrupts (on the various vcpus)
> had been injected
> before injecting the next interrupt to the vioapic.. >
> The policy code works as follows. It uses the hvm_isa_irq_assert_cb() to
> record
> the vcpus to be interrupted in h->hpet.pending_mask. Then, in the callback
> registered
> with hvm_register_intr_en_notif() at post=1 time it clears the current vcpu in
> the pending_mask.
> When the pending_mask is clear it decrements hpet.intr_pending_nr and if
> intr_pending_nr is still
> non-zero posts another interrupt to the ioapic with hvm_isa_irq_assert_cb().
> Intr_pending_nr is incremented in hpet_route_decision_not_missed_ticks().
>
> The missed ticks policy intr_en_notif callback also uses the pending_mask
> method. So even though
> Linux does not broadcast its interrupts, the code could handle it if it did.
> In this case the end of interrupt time stamp is made when the pending_mask is
> clear.
>
> 4. Live Migration
>
> Live migration with hpet preserves the current offset of the guest clock with
> respect
> to ntp. This is accomplished by migrating all of the state in the h->hpet data
> structure
> in the usual way. The hp->mc_offset is recalculated on the receiving node so
> that the
> guest sees a continuous hpet main counter.
>

Quoted 6 lines

> Code as been added to xc_domain_save.c to send a small message after the
> domain context is sent. The contents of the message is the physical tsc
> timestamp, last_tsc,
> read just before the message is sent. When the

last_tsc message

Quoted 39 lines

is received in
> xc_domain_restore.c,
> another physical tsc timestamp, cur_tsc, is read. The two timestamps are
> loaded into the domain
> structure as last_tsc_sender and first_tsc_receiver with hypercalls. Then
> xc_domain_hvm_setcontext
> is called so that hpet_load has access to these time stamps. Hpet_load uses
> the timestamps
> to account for the time spent saving and loading the domain context. With this
> technique,
> the only neglected time is the time spent sending a small network message.
>
> 5. Test Results
>
> Some recent test results are:
>
> 5.1 Linux 4u664 and Windows 2k864 load test.
> Duration: 70 hours.
> Test date: 6/2/08
> Loads: usex -b48 on Linux; burn-in on Windows
> Guest vcpus: 8 for Linux; 2 for Windows
> Hardware: 8 physical cpu AMD
> Clock drift : Linux: .0012% Windows: .009%
>
> 5.2 Linux 4u664, Linux 4u464 , and Windows 2k864 no-load test
> Duration: 23 hours.
> Test date: 6/3/08
> Loads: none
> Guest vcpus: 8 for each Linux; 2 for Windows
> Hardware: 4 physical cpu AMD
> Clock drift : Linux: .033% Windows: .019%
>
> 6. Relation to recent work in xen-unstable
>

Quoted 7 lines

> There is a similarity between hvm_get_guest_time() in xen-unstable and
> read_64_main_counter()
> in this code. However, read_64_main_counter() is more tuned to the needs of
> hpet.c. It has no
> "set" operation, only the get. It isolates the mode,

physical or

Quoted 6 lines

simulated, in
> read_64_main_counter()
> itself. It uses no vcpu or domain state as it is a physical entity, in either
> mode. And it provides a real
> physical mode for every read for those applications

that desire

Quoted 5 lines

this.
>
> 7. Conclusion
>
> The virtual hpet is improved by this patch in terms

of accuracy and

Quoted 2 lines

Quoted 2 lines

monotonicity.
Tests performed to date verify this and more testing

is under way.

Quoted 12 lines

Quoted 4 lines

8. Future Work

Testing with Windows Vista will be performed soon. The reason

for accuracy
> variations
> on different platforms using the physical hpet device will be investigated.
> Additional overhead measurements on simulated vs physical hpet mode will be
> made.
>

Quoted 3 lines

Quoted 3 lines

Footnotes:

1. I don't recall the accuracy improvement with end

of interrupt

Quoted 23 lines

stamping, but
> it was
> significant, perhaps better than two to one improvement. It would be a very
> simple matter
> to re-measure the improvement as the facility can call back at injection time
> as well.
>
>
> Signed-off-by: Dave Winchell <[email protected]> > <mailto:[email protected]>
> Signed-off-by: Ben Guthro <[email protected]> > <mailto:[email protected]>
>
>
> _______________________________________________
> Xen-devel mailing list
> [email protected]
> http://lists.xensource.com/xen-devel

I'll tell you what I recall about this. Tomorrow I'll check the guest code to verify. I think that Linux declares a full tick, even if the interrupt is early. That's the problem.

On the other hand, if the interrupt is late it in effect declares a tick plus fraction. If it just declared the fraction in the first place, we could deliver the interrupts whenever we wanted.

Quoted 2 lines

The Linux policy is more subtle, but is required to go from .1% to .03%.

Thanks for the good documentation which I hadn't thoroughly read until now.
I now understand that the essence of your
hpet missed ticks policy is to ensure that ticks are never delivered too close together. But I'm trying to understand WHY your patch works, in other words, what problem it is countering.

I care about this for more reasons than just
because it is interesting: (1) I'd like to feel confident that it is fixing a bug rather than just a symptom of a bug; and (2) I wonder how universally it is applicable.

I see from code examination in mark_offset_hpet() in RHEL4u5/arch/i386/kernel/timers/timer_hpet.c, that the correction for lost ticks is just plain wrong in a virtual environment. (Suppose for example that a virtual tick was delivered every 1.999*hpet_tick... I think the clock would be off by 50%!) Is this the bug that is being "countered" by your policy?

However, the lost tick handling in RHEL5u1/kernel/timer.c (which I think is used also for hpet) is much better so I am eager to find out if your policy works there too.
If the hpet missed tick policy works for both, though, I should be happy, though I wonder about upstream kernels (e.g. the trend toward tickless).

That said, I'd rather
see this get into Xen 3.3 and worry about upstream kernels later :-)

The Linux policy is more subtle, but is required to go from .1% to .03%.

While I am fully supportive of offering hardware hpet as an option for hvm guests (let's call it hwhpet=1 for shorthand), I am very surprised by your preliminary results; the most obvious conclusion is that Xen system time is losing time at the rate of 1000 PPM though its possible there's a bug somewhere else in the "time stack". Your Windows result is jaw-dropping and inexplicable, though I have to admit ignorance of how Windows manages time.

I think with my recent patch and hpet=1 (essentially the same as your emulated hpet), hvm guest time should track Xen system time. I wonder if domain0 (which if I understand correctly is directly using Xen system time) is also seeing an error of .1%? Also I wonder for the skew you are seeing (in both hvm guests and domain0) is time moving too fast or two slow?

Although hwhpet=1 is a fine alternative in many cases, it may be unavailable on some systems and may cause significant performance issues on others. So I think we will still need to track down the poor accuracy when hwhpet=0.

And if for some reason
Xen system time can't be made accurate enough (< 0.05%), then I think we should consider building Xen system time itself on top of hardware hpet instead of TSC... at least when Xen discovers a capable hpet.

One more thought... do you know the accuracy of the TSC crystals on your test systems? I posted a patch awhile ago that was intended to test that, though I guess it was only testing skew of different TSCs on the same system, not TSCs against an external time source.

Or maybe there's a computation error somewhere in the hvm hpet scaling code? Hmmm...

-----Original Message-----
From: Dave Winchell [mailto:[email protected]]
Sent: Friday, June 06, 2008 1:33 PM
To: [email protected]; Keir Fraser
Cc: Ben Guthro; xen-devel; Dave Winchell
Subject: Re: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Dan, Keir:

Preliminary tests results indicate an error of .1% for Linux 64 bit guests configured
for hpet with xen-unstable as is. As we have discussed many times, the ntp requirement is .05%.
Tests on the patch we just submitted for hpet have indicated errors of .0012%
on this platform under similar test conditions and .03% on other platforms.

Windows vista64 has an error of 11% using hpet with the xen-unstable bits.
In an overnight test with our hpet patch, the Windows vista error was .008%.

The tests are with two or three guests on a physical node, all under load, and with
the ratio of vcpus to phys cpus > 1.

I will continue to run tests over the next few days.

thanks,
Dave

Quoted 3 lines · Dan Magenheimer

Hi Dave and Ben --

When running tests on xen-unstable (without your patch),

please ensure

Quoted 2 lines

that hpet=1 is set in the hvm config and also I think that when hpet is the clocksource on RHEL4-32, the clock IS resilient to

missed ticks

Quoted 1 line

so timer_mode should be 2 (vs when pit is the clocksource

on RHEL4-32,

Quoted 4 lines

all clock ticks must be delivered and so timer_mode should be 0).

Per

http://lists.xensource.com/archives/html/xen-devel/2008-06/msg
00098.html it's

Quoted 8 lines

my intent to clean this up, but I won't get to it until next week.

Thanks,
Dan

    -----Original Message-----
    *From:* [email protected]
    [mailto:[email protected]]*On

Behalf Of *Dave

Quoted 9 lines

Winchell
*Sent:* Friday, June 06, 2008 4:46 AM
*To:* Keir Fraser; Ben Guthro; xen-devel
*Cc:* [email protected]; Dave Winchell *Subject:* RE: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Keir,

I think the changes are required. We'll run some tests

today today so

Quoted 7 lines

that we have some data to talk about.

-Dave

-----Original Message-----
From: [email protected] on behalf

of Keir Fraser

Quoted 8 lines

Sent: Fri 6/6/2008 4:58 AM
To: Ben Guthro; xen-devel
Cc: [email protected]
Subject: Re: [Xen-devel] [PATCH 0/2] Improve hpet accuracy

Are these patches needed now the timers are built on Xen system time rather
than host TSC? Dan has reported much better

time-keeping with his

Quoted 2 lines

patch
checked in, and it¹s for sure a lot less invasive than

this patchset.

Quoted 10 lines

     -- Keir

    On 5/6/08 15:59, "Ben Guthro" <[email protected]> wrote:

    >
    > 1. Introduction
    >
    > This patch improves the hpet based guest clock in

terms of drift and

Quoted 2 lines

Quoted 2 lines

monotonicity.
Prior to this work the drift with hpet was greater

than 2%, far

Quoted 10 lines

above the .05%
> limit
> for ntp to synchronize. With this code, the drift ranges from .001% to .0033%
> depending
> on guest and physical platform.
>
> Using hpet allows guest operating systems to provide monotonic time to their
> applications. Time sources other than hpet are not

monotonic because

Quoted 1 line

Quoted 1 line

of their reliance on tsc, which is not synchronized

across physical

Quoted 31 lines

Quoted 3 lines

processors.

Windows 2k864 and many Linux guests are supported with two

policies, one for
> guests
> that handle missed clock interrupts and the other for guests that require the
> correct number of interrupts.
>
> Guests may use hpet for the timing source even if the physical platform has no
> visible
> hpet. Migration is supported between physical machines which differ in
> physical
> hpet visibility.
>
> Most of the changes are in hpet.c. Two general facilities are added to track
> interrupt
> progress. The ideas here and the facilities would be useful in vpt.c, for
> other time
> sources, though no attempt is made here to improve vpt.c. >
> The following sections discuss hpet dependencies, interrupt delivery policies,
> live migration,
> test results, and relation to recent work with monotonic time. >
>

Quoted 3 lines

> 2. Virtual Hpet dependencies
>
> The virtual hpet depends on the ability to read the

physical or

Quoted 2 lines

simulated
> (see discussion below) hpet. For timekeeping, the

virtual hpet

Quoted 32 lines

also depends
> on two new interrupt notification facilities to implement its policies for
> interrupt delivery.
>
> 2.1. Two modes of low-level hpet main counter reads. >
> In this implementation, the virtual hpet reads with read_64_main_counter(),
> exported by
> time.c, either the real physical hpet main counter register directly or a
> "simulated"
> hpet main counter.
>
> The simulated mode uses a monotonic version of get_s_time() (NOW()), where the
> last
> time value is returned whenever the current time value is less than the last
> time
> value. In simulated mode, since it is layered on s_time, the underlying
> hardware
> can be hpet or some other device. The frequency of the main counter in
> simulated
> mode is the same as the standard physical hpet frequency, allowing live
> migration
> between nodes that are configured differently.
>

Quoted 1 line

> If the physical platform does not have an hpet

device, or if xen

Quoted 9 lines

is configured
> not
> to use the device, then the simulated method is used. If there is a physical
> hpet device,
> and xen has initialized it, then either simulated or physical mode can be
> used.
> This is governed by a boot time option, hpet-avoid.

Setting this

Quoted 3 lines

option to 1
> gives the
> simulated mode and 0 the physical mode. The default

is physical

Quoted 5 lines

mode.
>
> A disadvantage of the physical mode is that may take longer to read the device
> than in simulated mode. On some platforms the cost is

about the

Quoted 15 lines

same (less
> than 250 nsec) for
> physical and simulated modes, while on others physical cost is much higher
> than simulated.
> A disadvantage of the simulated mode is that it can return the same value
> for the counter in consecutive calls.
>
> 2.2. Interrupt notification facilities.
>
> Two interrupt notification facilities are introduced, one is > hvm_isa_irq_assert_cb()
> and the other hvm_register_intr_en_notif().
>

Quoted 5 lines

> The vhpet uses hvm_isa_irq_assert_cb to deliver interrupts to the vioapic.
> hvm_isa_irq_assert_cb allows a callback to be passed along to > vioapic_deliver()
> and this callback is called with a mask of the vcpus

which will

Quoted 4 lines

get the
> interrupt. This callback is made before any vcpus receive an interrupt.
>

Quoted 8 lines

> Vhpet uses hvm_register_intr_en_notif() to register a handler for a particular
> vector that will be called when that vector is injected in > [vmx,svm]_intr_assist()
> and also when the guest finishes handling the interrupt. Here finished is
> defined
> as the point when the guest re-enables interrupts or

lowers the

Quoted 4 lines

tpr value.
> EOI is not used as the end of interrupt as this is sometimes returned before
> the interrupt handler has done its work. A flag is

passed to the

Quoted 3 lines

handler
> indicating
> whether this is the injection point (post = 1) or the

interrupt

Quoted 6 lines

finished (post
> = 0) point.
> The need for the finished point callback is discussed in the missed ticks
> policy section.
>

Quoted 7 lines

> To prevent a possible early trigger of the finished callback, intr_en_notif
> logic
> has a two stage arm, the first at injection (hvm_intr_en_notif_arm()) and the
> second when
> interrupts are seen to be disabled

(hvm_intr_en_notif_disarm()).

Quoted 3 lines

Once fully
> armed, re-enabling
> interrupts will cause hvm_intr_en_notif_disarm() to

make the end

Quoted 2 lines

of interrupt
> callback. hvm_intr_en_notif_arm() and

hvm_intr_en_notif_disarm()

Quoted 6 lines

are called by
> [vmx,svm]_intr_assist().
>
> 3. Interrupt delivery policies
>
> The existing hpet interrupt delivery is preserved.

This includes

Quoted 8 lines

Quoted 1 line

vcpu round robin delivery used by Linux and broadcast delivery

used by
> Windows.
>
> There are two policies for interrupt delivery, one for Windows 2k8-64 and the
> other
> for Linux. The Linux policy takes advantage of the

(guest) Linux

Quoted 3 lines

missed tick
> and offset
> calculations and does not attempt to deliver the

right number of

Quoted 10 lines

interrupts.
> The Windows policy delivers the correct number of interrupts, even if
> sometimes much
> closer to each other than the period. The policies are similar to those in
> vpt.c, though
> there are some important differences.
>
> Policies are selected with an HVMOP_set_param

hypercall with index

Quoted 2 lines

Quoted 2 lines

HVM_PARAM_TIMER_MODE.
Two new values are added,

HVM_HPET_guest_computes_missed_ticks and

Quoted 7 lines

Quoted 1 line

HVM_HPET_guest_does_not_compute_missed_ticks. The reason that

two new ones
> are added is that
> in some guests (32bit Linux) a no-missed policy is needed for clock sources
> other than hpet
> and a missed ticks policy for hpet. It was felt that

there would

Quoted 14 lines

be less
> confusion by simply
> introducing the two hpet policies.
>
> 3.1. The missed ticks policy
>
> The Linux clock interrupt handler for hpet calculates missed ticks and offset
> using the hpet
> main counter. The algorithm works well when the time since the last interrupt
> is greater than
> or equal to a period and poorly otherwise.
>

Quoted 1 line

> The missed ticks policy ensures that no two clock

interrupts are

Quoted 6 lines

delivered to
> the guest at
> a time interval less than a period. A time stamp (hpet main counter value) is
> recorded (by a
> callback registered with hvm_register_intr_en_notif)

when Linux

Quoted 3 lines

finishes
> handling the clock
> interrupt. Then, ensuing interrupts are delivered to

the vioapic

Quoted 9 lines

only if the
> current main
> counter value is a period greater than when the last interrupt was handled.
>
> Tests showed a significant improvement in clock drift with end of interrupt
> time stamps
> versus beginning of interrupt[1]. It is believed that

the reason

Quoted 3 lines

for the
> improvement
> is that the clock interrupt handler goes for a

spinlock and can

Quoted 11 lines

be therefore
> delayed in its
> processing. Furthermore, the main counter is read by the guest under the lock.
> The net
> effect is that if we time stamp injection, we can get the difference in time
> between successive interrupt handler lock acquisitions to be less than the
> period.
>

Quoted 3 lines

> 3.2. The no-missed ticks policy
>
> Windows 2k864 keeps very poor time with the missed

ticks policy.

Quoted 6 lines

So the
> no-missed ticks policy
> was developed. In the no-missed ticks policy we deliver the correct number of
> interrupts,
> even if they are spaced less than a period apart

(when catching up).

Quoted 50 lines

Quoted 2 lines

Windows 2k864 uses a broadcast mode in the interrupt routing

such that
> all vcpus get the clock interrupt. The best Windows drift performance was
> achieved when the
> policy code ensured that all the previous interrupts (on the various vcpus)
> had been injected
> before injecting the next interrupt to the vioapic.. >
> The policy code works as follows. It uses the hvm_isa_irq_assert_cb() to
> record
> the vcpus to be interrupted in h->hpet.pending_mask. Then, in the callback
> registered
> with hvm_register_intr_en_notif() at post=1 time it clears the current vcpu in
> the pending_mask.
> When the pending_mask is clear it decrements hpet.intr_pending_nr and if
> intr_pending_nr is still
> non-zero posts another interrupt to the ioapic with hvm_isa_irq_assert_cb().
> Intr_pending_nr is incremented in hpet_route_decision_not_missed_ticks().
>
> The missed ticks policy intr_en_notif callback also uses the pending_mask
> method. So even though
> Linux does not broadcast its interrupts, the code could handle it if it did.
> In this case the end of interrupt time stamp is made when the pending_mask is
> clear.
>
> 4. Live Migration
>
> Live migration with hpet preserves the current offset of the guest clock with
> respect
> to ntp. This is accomplished by migrating all of the state in the h->hpet data
> structure
> in the usual way. The hp->mc_offset is recalculated on the receiving node so
> that the
> guest sees a continuous hpet main counter.
>

Quoted 6 lines

> Code as been added to xc_domain_save.c to send a small message after the
> domain context is sent. The contents of the message is the physical tsc
> timestamp, last_tsc,
> read just before the message is sent. When the

last_tsc message

Quoted 39 lines

is received in
> xc_domain_restore.c,
> another physical tsc timestamp, cur_tsc, is read. The two timestamps are
> loaded into the domain
> structure as last_tsc_sender and first_tsc_receiver with hypercalls. Then
> xc_domain_hvm_setcontext
> is called so that hpet_load has access to these time stamps. Hpet_load uses
> the timestamps
> to account for the time spent saving and loading the domain context. With this
> technique,
> the only neglected time is the time spent sending a small network message.
>
> 5. Test Results
>
> Some recent test results are:
>
> 5.1 Linux 4u664 and Windows 2k864 load test.
> Duration: 70 hours.
> Test date: 6/2/08
> Loads: usex -b48 on Linux; burn-in on Windows
> Guest vcpus: 8 for Linux; 2 for Windows
> Hardware: 8 physical cpu AMD
> Clock drift : Linux: .0012% Windows: .009%
>
> 5.2 Linux 4u664, Linux 4u464 , and Windows 2k864 no-load test
> Duration: 23 hours.
> Test date: 6/3/08
> Loads: none
> Guest vcpus: 8 for each Linux; 2 for Windows
> Hardware: 4 physical cpu AMD
> Clock drift : Linux: .033% Windows: .019%
>
> 6. Relation to recent work in xen-unstable
>

Quoted 7 lines

> There is a similarity between hvm_get_guest_time() in xen-unstable and
> read_64_main_counter()
> in this code. However, read_64_main_counter() is more tuned to the needs of
> hpet.c. It has no
> "set" operation, only the get. It isolates the mode,

physical or

Quoted 6 lines

simulated, in
> read_64_main_counter()
> itself. It uses no vcpu or domain state as it is a physical entity, in either
> mode. And it provides a real
> physical mode for every read for those applications

that desire

Quoted 5 lines

this.
>
> 7. Conclusion
>
> The virtual hpet is improved by this patch in terms

of accuracy and

Quoted 2 lines

Quoted 2 lines

monotonicity.
Tests performed to date verify this and more testing

is under way.

Quoted 12 lines

Quoted 4 lines

8. Future Work

Testing with Windows Vista will be performed soon. The reason

for accuracy
> variations
> on different platforms using the physical hpet device will be investigated.
> Additional overhead measurements on simulated vs physical hpet mode will be
> made.
>

Quoted 3 lines

> Footnotes:
>
> 1. I don't recall the accuracy improvement with end

of interrupt

Quoted 23 lines

stamping, but
> it was
> significant, perhaps better than two to one improvement. It would be a very
> simple matter
> to re-measure the improvement as the facility can call back at injection time
> as well.
>
>
> Signed-off-by: Dave Winchell <[email protected]> > <mailto:[email protected]>
> Signed-off-by: Ben Guthro <[email protected]> > <mailto:[email protected]>
>
>
> _______________________________________________
> Xen-devel mailing list
> [email protected]
> http://lists.xensource.com/xen-devel