time: fix time accounting for x86 HVM guests
cover time: fix time accounting for x86 HVM guests
Hello,
When not emulating the TSC the guest time value calculated by using the vCPU time info page in HVM mode would drift between time synchronization intervals. First patch fixes the drift, second patch makes the calculation of cpu_khz round up the value for better accuracy.
Thanks, Roger.
Roger Pau Monne (2):
x86/time: use native TSC scaling factors when TSC is not scaled xen/cpu: round up cpu_khz calculations
xen/arch/arm/time.c | 4 ++-- xen/arch/riscv/time.c | 2 +- xen/arch/x86/time.c | 20 ++++++++++++++------ 3 files changed, 17 insertions(+), 9 deletions(-)
Signature · 1 line
2.53.0
PATCH 1/2 x86/time: use native TSC scaling factors when TSC is not scaled
When running HVM guest in native TSC mode avoid using the recalculated vTSC scaling factors based on the cpu_khz value. Using the kHz based frequency leads to the TSC scaling values possibly not being the same as the ones used by the per CPU cpu_time->tsc_scale field, which introduces skew between the guest and Xen's calculations of the system time.
On a 2gHz system, where the frequency is possibly detected as 1999999999Hz (note this is a worse-case scenario), the cpu_khz variable will be set to 1999999kHz, and hence 999Hz cycles will be not accounted for per second. Over a second (the time synchronization period), this leads to a skew of:
cycles * 1 / (Hz freq) = 999 / 1999999999 = 499,5ns
So far this has gone unnoticed because the time synchronization rendezvous forces the update of the tsc_timestamp and system_time fields in the vCPU time info area, and hence the skew only accumulates up to the rendezvous period. Attempting to remove the rendezvous causes the skew to grow unbounded.
Fix by using the native TSC scaling values (as used by Xen) when the guest TSC is not scaled.
---
I'm worried about the usage of cpu_khz beyond simple printing it for informational purposes. Overall I think it would be safer to store the frequency in Hz, as to avoid losing the least significant digits.
In any case, that's a different change.
--- xen/arch/x86/time.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-)
xen/arch/x86/time.c +13 -5
@@ -1710,17 +1710,25 @@ static void collect_time_info(const struct vcpu *v,else {if (is_hvm_domain (d) && hvm_tsc_scaling_supported ) - { tsc_stamp =hvm_scale_tsc (d, t->stamp.local_tsc); - u->tsc_to_system_mul = d->arch.vtsc_to_ns.mul_frac; - u->tsc_shift = d->arch.vtsc_to_ns.shift; - }else - { tsc_stamp = t->stamp.local_tsc; + +/* +* HVM guests using the native TSC ratio should use the same per-CPU +* scaling factors as Xen. This ensures time keeping is always in sync +* between Xen and the guest. +*/ +if ( tsc_stamp == t->stamp.local_tsc ) + { u->tsc_to_system_mul = t->tsc_scale.mul_frac; u->tsc_shift = t->tsc_scale.shift; } +else + { + u->tsc_to_system_mul = d->arch.vtsc_to_ns.mul_frac; + u->tsc_shift = d->arch.vtsc_to_ns.shift; + } } u->tsc_timestamp = tsc_stamp;
Signature · 1 line
2.53.0
Re: [PATCH 1/2] x86/time: use native TSC scaling factors when TSC is not scaled
Quoted 22 lines · Roger Pau Monne
When running HVM guest in native TSC mode avoid using the recalculated vTSC scaling factors based on the
cpu_khzvalue. Using the kHz based frequency leads to the TSC scaling values possibly not being the same as the ones used by the per CPUcpu_time->tsc_scalefield, which introduces skew between the guest and Xen's calculations of the system time.On a 2gHz system, where the frequency is possibly detected as 1999999999Hz (note this is a worse-case scenario), the
cpu_khzvariable will be set to 1999999kHz, and hence 999Hz cycles will be not accounted for per second. Over a second (the time synchronization period), this leads to a skew of:cycles * 1 / (Hz freq) = 999 / 1999999999 = 499,5ns
So far this has gone unnoticed because the time synchronization rendezvous forces the update of the
tsc_timestampandsystem_timefields in the vCPU time info area, and hence the skew only accumulates up to the rendezvous period. Attempting to remove the rendezvous causes the skew to grow unbounded.Fix by using the native TSC scaling values (as used by Xen) when the guest TSC is not scaled.
Quoted 8 lines
Fixes eab8a90be723 ("x86/time: scale host TSC in pvclock properly")Signed-off-by Roger Pau Monné <[email protected]>---
I'm worried about the usage ofcpu_khzbeyond simple printing it for informational purposes. Overall I think it would be safer to store the frequency in Hz, as to avoid losing the least significant digits.In any case, that's a different change.
I'm not quite sure - improving accuracy is of course a good thing, but will we ever be able to do any such calculations error free, when already the detected frequency isn't exactly precise?
xen/arch/x86/time.c static void collect_time_info(const struct vcpu *v, · line 1713
Quoted 8 lines
xen/arch/x86/time.c +0
-1@@ -1710,17 +1710,25 @@ static void collect_time_info(const struct vcpu *v,else {if (is_hvm_domain (d) && hvm_tsc_scaling_supported ) - { tsc_stamp =hvm_scale_tsc (d, t->stamp.local_tsc);
This is a potentially imprecise calculation. How likely is it that its result will indeed ...
xen/arch/x86/time.c static void collect_time_info(const struct vcpu *v, · line 1722
Quoted 13 lines
xen/arch/x86/time.c +7
-4- u->tsc_to_system_mul = d->arch.vtsc_to_ns.mul_frac; - u->tsc_shift = d->arch.vtsc_to_ns.shift; - }else - { tsc_stamp = t->stamp.local_tsc; + +/* +* HVM guests using the native TSC ratio should use the same per-CPU +* scaling factors as Xen. This ensures time keeping is always in sync +* between Xen and the guest. +*/ +if ( tsc_stamp == t->stamp.local_tsc )
... exactly match t->stamp.local_tsc? Don't we possibly need a (small) error margin? (In which case of course the next question would be: How to establish such a margin?)
Jan
Quoted 12 lines
xen/arch/x86/time.c +6
-0+ { u->tsc_to_system_mul = t->tsc_scale.mul_frac; u->tsc_shift = t->tsc_scale.shift; } + else + { + u->tsc_to_system_mul = d->arch.vtsc_to_ns.mul_frac; + u->tsc_shift = d->arch.vtsc_to_ns.shift; + } } u->tsc_timestamp = tsc_stamp;
Re: [PATCH 1/2] x86/time: use native TSC scaling factors when TSC is not scaled
Quoted 62 lines · Jan Beulich
Quoted 22 lines · Roger Pau Monne
When running HVM guest in native TSC mode avoid using the recalculated vTSC scaling factors based on the
cpu_khzvalue. Using the kHz based frequency leads to the TSC scaling values possibly not being the same as the ones used by the per CPUcpu_time->tsc_scalefield, which introduces skew between the guest and Xen's calculations of the system time.On a 2gHz system, where the frequency is possibly detected as 1999999999Hz (note this is a worse-case scenario), the
cpu_khzvariable will be set to 1999999kHz, and hence 999Hz cycles will be not accounted for per second. Over a second (the time synchronization period), this leads to a skew of:cycles * 1 / (Hz freq) = 999 / 1999999999 = 499,5ns
So far this has gone unnoticed because the time synchronization rendezvous forces the update of the
tsc_timestampandsystem_timefields in the vCPU time info area, and hence the skew only accumulates up to the rendezvous period. Attempting to remove the rendezvous causes the skew to grow unbounded.Fix by using the native TSC scaling values (as used by Xen) when the guest TSC is not scaled.
Quoted 8 lines
Fixes eab8a90be723 ("x86/time: scale host TSC in pvclock properly")Signed-off-by Roger Pau Monné <[email protected]>---
I'm worried about the usage ofcpu_khzbeyond simple printing it for informational purposes. Overall I think it would be safer to store the frequency in Hz, as to avoid losing the least significant digits.In any case, that's a different change.
I'm not quite sure - improving accuracy is of course a good thing, but will we ever be able to do any such calculations error free, when already the detected frequency isn't exactly precise?
xen/arch/x86/time.c static void collect_time_info(const struct vcpu *v, · line 1713
Quoted 8 lines
xen/arch/x86/time.c +0
-1@@ -1710,17 +1710,25 @@ static void collect_time_info(const struct vcpu *v,else {if (is_hvm_domain (d) && hvm_tsc_scaling_supported ) - { tsc_stamp =hvm_scale_tsc (d, t->stamp.local_tsc);This is a potentially imprecise calculation. How likely is it that its result will indeed ...
xen/arch/x86/time.c static void collect_time_info(const struct vcpu *v, · line 1722
Quoted 13 lines
xen/arch/x86/time.c +7
-4- u->tsc_to_system_mul = d->arch.vtsc_to_ns.mul_frac; - u->tsc_shift = d->arch.vtsc_to_ns.shift; - }else - { tsc_stamp = t->stamp.local_tsc; + +/* +* HVM guests using the native TSC ratio should use the same per-CPU +* scaling factors as Xen. This ensures time keeping is always in sync +* between Xen and the guest. +*/ +if ( tsc_stamp == t->stamp.local_tsc )
Quoted 3 lines
... exactly match
t->stamp.local_tsc? Don't we possibly need a (small) error margin? (In which case of course the next question would be: How to establish such a margin?)
hvm_scale_tsc() has:
if ( ratio == hvm_default_tsc_scaling_ratio )
return tsc;So when using no scaling the input value is the output value, and hence tsc_stamp will match exactly t->stamp.local_tsc.
Thanks, Roger.
Jan Beulich
Reviewed-byRe: [PATCH 1/2] x86/time: use native TSC scaling factors when TSC is not scaled
Quoted 67 lines · Roger Pau Monné
Quoted 62 lines · Jan Beulich
Quoted 22 lines · Roger Pau Monne
When running HVM guest in native TSC mode avoid using the recalculated vTSC scaling factors based on the
cpu_khzvalue. Using the kHz based frequency leads to the TSC scaling values possibly not being the same as the ones used by the per CPUcpu_time->tsc_scalefield, which introduces skew between the guest and Xen's calculations of the system time.On a 2gHz system, where the frequency is possibly detected as 1999999999Hz (note this is a worse-case scenario), the
cpu_khzvariable will be set to 1999999kHz, and hence 999Hz cycles will be not accounted for per second. Over a second (the time synchronization period), this leads to a skew of:cycles * 1 / (Hz freq) = 999 / 1999999999 = 499,5ns
So far this has gone unnoticed because the time synchronization rendezvous forces the update of the
tsc_timestampandsystem_timefields in the vCPU time info area, and hence the skew only accumulates up to the rendezvous period. Attempting to remove the rendezvous causes the skew to grow unbounded.Fix by using the native TSC scaling values (as used by Xen) when the guest TSC is not scaled.
Quoted 8 lines
Fixes eab8a90be723 ("x86/time: scale host TSC in pvclock properly")Signed-off-by Roger Pau Monné <[email protected]>---
I'm worried about the usage ofcpu_khzbeyond simple printing it for informational purposes. Overall I think it would be safer to store the frequency in Hz, as to avoid losing the least significant digits.In any case, that's a different change.
I'm not quite sure - improving accuracy is of course a good thing, but will we ever be able to do any such calculations error free, when already the detected frequency isn't exactly precise?
xen/arch/x86/time.c static void collect_time_info(const struct vcpu *v, · line 1713
Quoted 8 lines
xen/arch/x86/time.c +0
-1@@ -1710,17 +1710,25 @@ static void collect_time_info(const struct vcpu *v,else {if (is_hvm_domain (d) && hvm_tsc_scaling_supported ) - { tsc_stamp =hvm_scale_tsc (d, t->stamp.local_tsc);This is a potentially imprecise calculation. How likely is it that its result will indeed ...
xen/arch/x86/time.c static void collect_time_info(const struct vcpu *v, · line 1722
Quoted 13 lines
xen/arch/x86/time.c +7
-4- u->tsc_to_system_mul = d->arch.vtsc_to_ns.mul_frac; - u->tsc_shift = d->arch.vtsc_to_ns.shift; - }else - { tsc_stamp = t->stamp.local_tsc; + +/* +* HVM guests using the native TSC ratio should use the same per-CPU +* scaling factors as Xen. This ensures time keeping is always in sync +* between Xen and the guest. +*/ +if ( tsc_stamp == t->stamp.local_tsc )Quoted 3 lines
... exactly match
t->stamp.local_tsc? Don't we possibly need a (small) error margin? (In which case of course the next question would be: How to establish such a margin?)
Quoted 7 lines
hvm_scale_tsc()has:if ( ratio == hvm_default_tsc_scaling_ratio ) return tsc;So when using no scaling the input value is the output value, and hence
tsc_stampwill match exactlyt->stamp.local_tsc.
Ouch. I did look at the function, but managed to have all my attention drawn to the asm() there. I'm sorry for the noise. As it's strictly an improvement:
The other, earlier remark remains applicable, though.
Jan
Re: [PATCH 1/2] x86/time: use native TSC scaling factors when TSC is not scaled
In reply to Jan BeulichQuoted 30 lines · Jan Beulich
Quoted 29 lines · Roger Pau Monne
When running HVM guest in native TSC mode avoid using the recalculated vTSC scaling factors based on the
cpu_khzvalue. Using the kHz based frequency leads to the TSC scaling values possibly not being the same as the ones used by the per CPUcpu_time->tsc_scalefield, which introduces skew between the guest and Xen's calculations of the system time.On a 2gHz system, where the frequency is possibly detected as 1999999999Hz (note this is a worse-case scenario), the
cpu_khzvariable will be set to 1999999kHz, and hence 999Hz cycles will be not accounted for per second. Over a second (the time synchronization period), this leads to a skew of:cycles * 1 / (Hz freq) = 999 / 1999999999 = 499,5ns
So far this has gone unnoticed because the time synchronization rendezvous forces the update of the
tsc_timestampandsystem_timefields in the vCPU time info area, and hence the skew only accumulates up to the rendezvous period. Attempting to remove the rendezvous causes the skew to grow unbounded.Fix by using the native TSC scaling values (as used by Xen) when the guest TSC is not scaled.
Fixes eab8a90be723 ("x86/time: scale host TSC in pvclock properly")Signed-off-by Roger Pau Monné <[email protected]>---
I'm worried about the usage ofcpu_khzbeyond simple printing it for informational purposes. Overall I think it would be safer to store the frequency in Hz, as to avoid losing the least significant digits.
Quoted 5 lines
Quoted 1 line
In any case, that's a different change.
I'm not quite sure - improving accuracy is of course a good thing, but will we ever be able to do any such calculations error free, when already the detected frequency isn't exactly precise?
I think getting them fully accurate is not strictly required. The specific issue here was that the guest was supposedly running with the native TSC frequency, but the vCPU time info scaling factors where (slightly) different from the ones using natively by Xen, hence resulting in a time skew.
When the guest runs with a different TSC frequency Xen already accounts for it properly, and hence there's no skew.
However, as noted in the next patch, I don't really see the benefit of storing the frequency in kHz instead of using plain Hz.
Thanks, Roger.
Re: [PATCH 1/2] x86/time: use native TSC scaling factors when TSC is not scaled
Quoted 37 lines · Roger Pau Monné
Quoted 30 lines · Jan Beulich
Quoted 29 lines · Roger Pau Monne
When running HVM guest in native TSC mode avoid using the recalculated vTSC scaling factors based on the
cpu_khzvalue. Using the kHz based frequency leads to the TSC scaling values possibly not being the same as the ones used by the per CPUcpu_time->tsc_scalefield, which introduces skew between the guest and Xen's calculations of the system time.On a 2gHz system, where the frequency is possibly detected as 1999999999Hz (note this is a worse-case scenario), the
cpu_khzvariable will be set to 1999999kHz, and hence 999Hz cycles will be not accounted for per second. Over a second (the time synchronization period), this leads to a skew of:cycles * 1 / (Hz freq) = 999 / 1999999999 = 499,5ns
So far this has gone unnoticed because the time synchronization rendezvous forces the update of the
tsc_timestampandsystem_timefields in the vCPU time info area, and hence the skew only accumulates up to the rendezvous period. Attempting to remove the rendezvous causes the skew to grow unbounded.Fix by using the native TSC scaling values (as used by Xen) when the guest TSC is not scaled.
Fixes eab8a90be723 ("x86/time: scale host TSC in pvclock properly")Signed-off-by Roger Pau Monné <[email protected]>---
I'm worried about the usage ofcpu_khzbeyond simple printing it for informational purposes. Overall I think it would be safer to store the frequency in Hz, as to avoid losing the least significant digits.Quoted 5 lines
Quoted 1 line
In any case, that's a different change.
I'm not quite sure - improving accuracy is of course a good thing, but will we ever be able to do any such calculations error free, when already the detected frequency isn't exactly precise?
Quoted 8 lines
I think getting them fully accurate is not strictly required. The specific issue here was that the guest was supposedly running with the native TSC frequency, but the vCPU time info scaling factors where (slightly) different from the ones using natively by Xen, hence resulting in a time skew.
When the guest runs with a different TSC frequency Xen already accounts for it properly, and hence there's no skew.
As "properly" isn't "accurate", I expect there'll still be some skew.
Quoted 2 lines
However, as noted in the next patch, I don't really see the benefit of storing the frequency in kHz instead of using plain Hz.
That would reduce the error, yes.
Jan
PATCH 2/2 xen/cpu: round up cpu_khz calculations
In reply to Roger Pau MonneAll arches truncate the cpu_khz without taking into account the less significant digits. Instead use DIV_ROUND_UP() when scaling from Hz to kHz to get as more accurate kHz value.
---
While the possibly more accurate value is nice, I'm not sure it's actually fixing any functional bug, and hence the lack of "Fixes:" tag.
--- xen/arch/arm/time.c | 4 ++-- xen/arch/riscv/time.c | 2 +- xen/arch/x86/time.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-)
xen/arch/arm/time.c +2 -2
@@ -118,7 +118,7 @@ static void __init preinit_dt_xen_time(void) res =dt_property_read_u32 (timer,"clock-frequency" , &rate);if ( res ) { - cpu_khz = rate /1000 ; + cpu_khz =DIV_ROUND_UP (rate,1000 );validate_timer_frequency (); timer_dt_clock_frequency = rate; }
@@ -136,7 +136,7 @@ void __init preinit_xen_time(void)if ( !cpu_khz ) { - cpu_khz = (READ_SYSREG (CNTFRQ_EL0 ) &CNTFRQ_MASK ) /1000 ; + cpu_khz =DIV_ROUND_UP (READ_SYSREG (CNTFRQ_EL0 ) &CNTFRQ_MASK ,1000 );validate_timer_frequency (); }
xen/arch/riscv/time.c +1 -1
@@ -40,7 +40,7 @@ static void __init preinit_dt_xen_time(void)if ( !dt_property_read_u32 (timer,"timebase-frequency" , &rate) )panic ("Unable to find clock frequency\n" ); - cpu_khz = rate /1000 ; + cpu_khz =DIV_ROUND_UP (rate,1000 ); }int reprogram_timer (s_time_t timeout)
xen/arch/x86/time.c +1 -1
@@ -2642,7 +2642,7 @@ void __init early_time_init(void)set_time_scale (&t->tsc_scale, tmp); t->stamp.local_tsc = boot_tsc_stamp; - cpu_khz = tmp /1000 ; + cpu_khz =DIV_ROUND_UP (tmp,1000 );printk ("Detected %lu.%03lu MHz processor.\n" , cpu_khz /1000 , cpu_khz %1000 );
Signature · 1 line
2.53.0
Re: [PATCH 2/2] xen/cpu: round up cpu_khz calculations
Quoted 5 lines · Roger Pau Monne
All arches truncate the
cpu_khzwithout taking into account the less significant digits. Instead useDIV_ROUND_UP()when scaling from Hz to kHz to get as more accurate kHz value.Signed-off-by Roger Pau Monné <[email protected]>
Couldn't DIV_ROUND be used here instead for a round-to-closest?
Quoted 49 lines
---
While the possibly more accurate value is nice, I'm not sure it's actually fixing any functional bug, and hence the lack of "Fixes:" tag.--- xen/arch/arm/time.c | 4 ++-- xen/arch/riscv/time.c | 2 +- xen/arch/x86/time.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-)xen/arch/arm/time.c +2
-2@@ -118,7 +118,7 @@ static void __init preinit_dt_xen_time(void) res =dt_property_read_u32 (timer,"clock-frequency" , &rate);if ( res ) { - cpu_khz = rate /1000 ; + cpu_khz =DIV_ROUND_UP (rate,1000 );validate_timer_frequency (); timer_dt_clock_frequency = rate; }@@ -136,7 +136,7 @@ void __init preinit_xen_time(void)if ( !cpu_khz ) { - cpu_khz = (READ_SYSREG (CNTFRQ_EL0 ) &CNTFRQ_MASK ) /1000 ; + cpu_khz =DIV_ROUND_UP (READ_SYSREG (CNTFRQ_EL0 ) &CNTFRQ_MASK ,1000 );validate_timer_frequency (); }xen/arch/riscv/time.c +1
-1@@ -40,7 +40,7 @@ static void __init preinit_dt_xen_time(void)if ( !dt_property_read_u32 (timer,"timebase-frequency" , &rate) )panic ("Unable to find clock frequency\n" ); - cpu_khz = rate /1000 ; + cpu_khz =DIV_ROUND_UP (rate,1000 ); }int reprogram_timer (s_time_t timeout)xen/arch/x86/time.c +0
-0
xen/arch/x86/time.c void __init early_time_init(void) · line 2648
Quoted 8 lines
xen/arch/x86/time.c +1
-1@@ -2642,7 +2642,7 @@ void __init early_time_init(void)set_time_scale (&t->tsc_scale, tmp); t->stamp.local_tsc = boot_tsc_stamp; - cpu_khz = tmp /1000 ; + cpu_khz =DIV_ROUND_UP (tmp,1000 );printk ("Detected %lu.%03lu MHz processor.\n" , cpu_khz /1000 , cpu_khz %1000 );
--
Ngoc Tu Dinh | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
Re: [PATCH 2/2] xen/cpu: round up cpu_khz calculations
Quoted 8 lines · Tu Dinh
Quoted 5 lines · Roger Pau Monne
All arches truncate the
cpu_khzwithout taking into account the less significant digits. Instead useDIV_ROUND_UP()when scaling from Hz to kHz to get as more accurate kHz value.Signed-off-by Roger Pau Monné <[email protected]>Couldn't
DIV_ROUNDbe used here instead for a round-to-closest?
My bad, I got confused with the macro names.
Roger.
Re: [PATCH 2/2] xen/cpu: round up cpu_khz calculations
In reply to Roger Pau MonneQuoted 8 lines · Roger Pau Monne
All arches truncate the
cpu_khzwithout taking into account the less significant digits. Instead useDIV_ROUND_UP()when scaling from Hz to kHz to get as more accurate kHz value.Signed-off-by Roger Pau Monné <[email protected]>---
While the possibly more accurate value is nice, I'm not sure it's actually fixing any functional bug, and hence the lack of "Fixes:" tag.
With, as suggested by Tu, DIV_ROUND() used instead (which then statistically indeed is an improvement in accuracy)
As it's only statistically, nevertheless there is a risk of (perceived) regressions, just to mention it.
Jan
Quoted 7 lines · Roger Pau Monne
Hello,
When not emulating the TSC the guest time value calculated by using the vCPU time info page in HVM mode would drift between time synchronization intervals. First patch fixes the drift, second patch makes the calculation of
cpu_khzround up the value for better accuracy.
Quoted 5 lines
Thanks, Roger.
Roger Pau Monne (2):
x86/time: use native TSC scaling factors when TSC is not scaled xen/cpu: round upcpu_khzcalculations
Thanks Roger, this fixed an outstanding bug I was seeing!
Quoted 6 lines · Roger Pau Monne
Hello,
When not emulating the TSC the guest time value calculated by using the vCPU time info page in HVM mode would drift between time synchronization intervals. First patch fixes the drift, second patch makes the calculation of
cpu_khzround up the value for better accuracy.
I confirm with with those patches, and without the other fix ("x86/time:
do not kill calibration timer on suspend") the post-S3 issue is also gone!
Signature · 3 lines
Best Regards, Marek Marczykowski-Górecki Invisible Things Lab