new config option vtsc_tolerance_khz to avoid TSC emulation

Add an option to control when vTSC emulation will be activated for a domU with tsc_mode=default. Without such option each TSC access from domU will be emulated, which causes a significant perfomance drop for workloads that make use of rdtsc.

One option to avoid the TSC option is to run domUs with tsc_mode=native. This has the drawback that migrating a domU from a "2.3GHz" class host to a "2.4GHz" class host may change the rate at wich the TSC counter increases, the domU may not be prepared for that.

With the new option the host admin can decide how a domU should behave when it is migrated across systems of the same class. Since there is always some jitter when Xen calibrates the cpu_khz value, all hosts of the same class will most likely have slightly different values. As a result vTSC emulation is unavoidable. Data collected during the incident which triggered this change showed a jitter of up to 200 KHz across systems of the same class.

Existing padding fields are reused to store vtsc_khz_tolerance as u16.

index 2c1a6e1422..0b36265e4f 100644

docs/man/xl.cfg.pod.5.in +0 -0

@@ -1891,6 +1891,16 @@ determined in a similar way to that of B<default> TSC mode.
 
 Please see B<xen-tscmode(7)> for more information on this option.

Quoted 20 lines · Olaf Hering

Add an option to control when vTSC emulation will be activated for a domU with tsc_mode=default. Without such option each TSC access from domU will be emulated, which causes a significant perfomance drop for workloads that make use of rdtsc.

One option to avoid the TSC option is to run domUs with tsc_mode=native. This has the drawback that migrating a domU from a "2.3GHz" class host to a "2.4GHz" class host may change the rate at wich the TSC counter increases, the domU may not be prepared for that.

With the new option the host admin can decide how a domU should behave when it is migrated across systems of the same class. Since there is always some jitter when Xen calibrates the cpu_khz value, all hosts of the same class will most likely have slightly different values. As a result vTSC emulation is unavoidable. Data collected during the incident which triggered this change showed a jitter of up to 200 KHz across systems of the same class.

Existing padding fields are reused to store vtsc_khz_tolerance as u16.

[...]

Quoted 6 lines

index 2c1a6e1422..0b36265e4f 100644

docs/man/xl.cfg.pod.5.in +0 -0

@@ -1891,6 +1891,16 @@ determined in a similar way to that of B<default> TSC mode.
 
 Please see B<xen-tscmode(7)> for more information on this option.

docs/man/xl.cfg.pod.5.in determined in a similar way to that of B<default> TSC mode. · line 1902

Quoted 10 lines

docs/man/xl.cfg.pod.5.in +9 -0

 
+=item B<vtsc_tolerance_khz="KHZ">
+
+B<(x86 only, relevant only for tsc_mode=default)>
+When a domU is started, the CPU frequency of the host is used by the domU for
+TSC related time measurement. Once the domU is either migrated or
+saved/restored on another host that CPU frequency has to be emulated to avoid
+timedrift. To avoid the performance penalty of the TSC emulation, allow a
+certain amount of jitter of the measured CPU frequency on the hosts the domU
+is supposed to run on.

"Default value is 0, i.e. no tolerance".

Can we get an agreement on whether this idea the right approach in general before I do detail review?

Quoted 41 lines · Wei Liu

Quoted 20 lines · Olaf Hering

Add an option to control when vTSC emulation will be activated for a domU with tsc_mode=default. Without such option each TSC access from domU will be emulated, which causes a significant perfomance drop for workloads that make use of rdtsc.

One option to avoid the TSC option is to run domUs with tsc_mode=native. This has the drawback that migrating a domU from a "2.3GHz" class host to a "2.4GHz" class host may change the rate at wich the TSC counter increases, the domU may not be prepared for that.

With the new option the host admin can decide how a domU should behave when it is migrated across systems of the same class. Since there is always some jitter when Xen calibrates the cpu_khz value, all hosts of the same class will most likely have slightly different values. As a result vTSC emulation is unavoidable. Data collected during the incident which triggered this change showed a jitter of up to 200 KHz across systems of the same class.

Existing padding fields are reused to store vtsc_khz_tolerance as u16.

[...]

Quoted 6 lines

index 2c1a6e1422..0b36265e4f 100644

docs/man/xl.cfg.pod.5.in +0 -0

@@ -1891,6 +1891,16 @@ determined in a similar way to that of B<default> TSC mode.
 
 Please see B<xen-tscmode(7)> for more information on this option.

docs/man/xl.cfg.pod.5.in determined in a similar way to that of B<default> TSC mode. · line 1902

Quoted 10 lines

docs/man/xl.cfg.pod.5.in +9 -0

 
+=item B<vtsc_tolerance_khz="KHZ">
+
+B<(x86 only, relevant only for tsc_mode=default)>
+When a domU is started, the CPU frequency of the host is used by the domU for
+TSC related time measurement. Once the domU is either migrated or
+saved/restored on another host that CPU frequency has to be emulated to avoid
+timedrift. To avoid the performance penalty of the TSC emulation, allow a
+certain amount of jitter of the measured CPU frequency on the hosts the domU
+is supposed to run on.

"Default value is 0, i.e. no tolerance".

Quoted 2 lines

Can we get an agreement on whether this idea the right approach in general before I do detail review?

I'm not super-familiar with this area, but looking from the outside I think Olaf's approach (having a "tolerance" for equivalence) makes sense. Can't comment on what kind of a hypercall it should be.

Quoted 45 lines · George Dunlap

Quoted 43 lines · Wei Liu

Quoted 20 lines · Olaf Hering

Add an option to control when vTSC emulation will be activated for a domU with tsc_mode=default. Without such option each TSC access from domU will be emulated, which causes a significant perfomance drop for workloads that make use of rdtsc.

One option to avoid the TSC option is to run domUs with tsc_mode=native. This has the drawback that migrating a domU from a "2.3GHz" class host to a "2.4GHz" class host may change the rate at wich the TSC counter increases, the domU may not be prepared for that.

With the new option the host admin can decide how a domU should behave when it is migrated across systems of the same class. Since there is always some jitter when Xen calibrates the cpu_khz value, all hosts of the same class will most likely have slightly different values. As a result vTSC emulation is unavoidable. Data collected during the incident which triggered this change showed a jitter of up to 200 KHz across systems of the same class.

Existing padding fields are reused to store vtsc_khz_tolerance as u16.

[...]

Quoted 6 lines

index 2c1a6e1422..0b36265e4f 100644

docs/man/xl.cfg.pod.5.in +0 -0

@@ -1891,6 +1891,16 @@ determined in a similar way to that of B<default> TSC mode.
 
 Please see B<xen-tscmode(7)> for more information on this option.

docs/man/xl.cfg.pod.5.in determined in a similar way to that of B<default> TSC mode. · line 1902

Quoted 10 lines

docs/man/xl.cfg.pod.5.in +9 -0

 
+=item B<vtsc_tolerance_khz="KHZ">
+
+B<(x86 only, relevant only for tsc_mode=default)>
+When a domU is started, the CPU frequency of the host is used by the domU for
+TSC related time measurement. Once the domU is either migrated or
+saved/restored on another host that CPU frequency has to be emulated to avoid
+timedrift. To avoid the performance penalty of the TSC emulation, allow a
+certain amount of jitter of the measured CPU frequency on the hosts the domU
+is supposed to run on.

"Default value is 0, i.e. no tolerance".

Can we get an agreement on whether this idea the right approach in general before I do detail review?

Quoted 4 lines

I'm not super-familiar with this area, but looking from the outside I think Olaf's approach (having a "tolerance" for equivalence) makes sense. Can't comment on what kind of a hypercall it should be.

I have a rough idea how the code should look like -- the patch in its current form looks mostly OK, but not sure if this approach in general is future proof.

Quoted 51 lines · Wei Liu

Quoted 49 lines · George Dunlap

Quoted 41 lines · Wei Liu

Quoted 20 lines · Olaf Hering

Add an option to control when vTSC emulation will be activated for a domU with tsc_mode=default. Without such option each TSC access from domU will be emulated, which causes a significant perfomance drop for workloads that make use of rdtsc.

One option to avoid the TSC option is to run domUs with tsc_mode=native. This has the drawback that migrating a domU from a "2.3GHz" class host to a "2.4GHz" class host may change the rate at wich the TSC counter increases, the domU may not be prepared for that.

With the new option the host admin can decide how a domU should behave when it is migrated across systems of the same class. Since there is always some jitter when Xen calibrates the cpu_khz value, all hosts of the same class will most likely have slightly different values. As a result vTSC emulation is unavoidable. Data collected during the incident which triggered this change showed a jitter of up to 200 KHz across systems of the same class.

Existing padding fields are reused to store vtsc_khz_tolerance as u16.

[...]

Quoted 6 lines

index 2c1a6e1422..0b36265e4f 100644

docs/man/xl.cfg.pod.5.in +0 -0

@@ -1891,6 +1891,16 @@ determined in a similar way to that of B<default> TSC mode.
 
 Please see B<xen-tscmode(7)> for more information on this option.

docs/man/xl.cfg.pod.5.in determined in a similar way to that of B<default> TSC mode. · line 1902

Quoted 10 lines

docs/man/xl.cfg.pod.5.in +9 -0

 
+=item B<vtsc_tolerance_khz="KHZ">
+
+B<(x86 only, relevant only for tsc_mode=default)>
+When a domU is started, the CPU frequency of the host is used by the domU for
+TSC related time measurement. Once the domU is either migrated or
+saved/restored on another host that CPU frequency has to be emulated to avoid
+timedrift. To avoid the performance penalty of the TSC emulation, allow a
+certain amount of jitter of the measured CPU frequency on the hosts the domU
+is supposed to run on.

"Default value is 0, i.e. no tolerance".

Quoted 2 lines

Can we get an agreement on whether this idea the right approach in general before I do detail review?

I'm not super-familiar with this area, but looking from the outside I think Olaf's approach (having a "tolerance" for equivalence) makes sense. Can't comment on what kind of a hypercall it should be.

Quoted 3 lines

I have a rough idea how the code should look like -- the patch in its current form looks mostly OK, but not sure if this approach in general is future proof.

What do you mean? And do you have an idea for a better way to solve the problem? Or do you think it's not a very big problem?

Quoted 1 line

Quoted 1 line · On 27.03.18 at 11:26, <[email protected]> wrote:

xen/arch/x86/time.c +0 -0

@@ -2149,8 +2154,26 @@ void tsc_set_info(struct domain *d,
          * When a guest is created, gtsc_khz is passed in as zero, making
          * d->arch.tsc_khz == cpu_khz. Thus no need to check incarnation.

xen/arch/x86/time.c void tsc_set_info(struct domain *d, · line 2165

Quoted 3 lines

Quoted 1 line

Quoted 1 line · On 27.03.18 at 11:26, <[email protected]> wrote:

xen/arch/x86/time.c +2 -0

+            khz_diff = cpu_khz > gtsc_khz ?
+                       cpu_khz - gtsc_khz : gtsc_khz - cpu_khz;

abs() (or some variant of it, like __builtin_absl(), seeing that we don't appear to have any abstraction right now)?

d%d

Quoted 1 line

Quoted 1 line · On 29.03.18 at 10:17, <[email protected]> wrote:

Quoted 5 lines · Jan Beulich

xen/arch/x86/time.c void tsc_set_info(struct domain *d, · line 2165

Quoted 3 lines

Quoted 1 line

Quoted 1 line · On 27.03.18 at 11:26, <[email protected]> wrote:

xen/arch/x86/time.c +2 -0

+            khz_diff = cpu_khz > gtsc_khz ?
+                       cpu_khz - gtsc_khz : gtsc_khz - cpu_khz;

abs() (or some variant of it, like __builtin_absl(), seeing that we don't appear to have any abstraction right now)?

I see no other usage of *abs*. Really optimize that one-shot function?

Quoted 1 line

d%d

A for an unsigned type? But I see it is done elsewhere, so it must be correct.

When you use abs() or alike in places like this, it is more immediately obvious to the reader what you're doing.

Quoted 1 line

Quoted 1 line · On 29.03.18 at 11:23, <[email protected]> wrote:

Quoted 2 lines · Jan Beulich

When you use abs() or alike in places like this, it is more immediately obvious to the reader what you're doing.

Does every supported compiler actually understand this? int khz_diff = __builtin_abs(cpu_khz - gtsc_khz); Or do we need an inline abs() in case it is not gcc?

Actually I was wrong - we have an abstraction already, just that it's upper case: ABS(). But it requires its input to have signed type.

Quoted 1 line

Quoted 1 line · On 29.03.18 at 12:05, <[email protected]> wrote:

Quoted 2 lines · Jan Beulich

Actually I was wrong - we have an abstraction already, just that it's upper case: ABS(). But it requires its input to have signed type.

Would this be acceptable?
khz_diff = ABS((long)cpu_khz - (long)gtsc_khz);

Add an option to control when vTSC emulation will be activated for a domU with tsc_mode=default. Without such option each TSC access from domU will be emulated, which causes a significant perfomance drop for workloads that make use of rdtsc.

case TSC_MODE_DEFAULT:

xen/arch/x86/time.c +10 -0

+        d->arch.vtsc_tolerance_khz = vtsc_tolerance_khz;
+        /* Fallthrough. */
     case TSC_MODE_ALWAYS_EMULATE:
         d->arch.vtsc_offset = get_s_time() - elapsed_nsec;
         d->arch.tsc_khz = gtsc_khz ?: cpu_khz;

@@ -2149,8 +2154,26 @@ void tsc_set_info(struct domain *d,
          * When a guest is created, gtsc_khz is passed in as zero, making
          * d->arch.tsc_khz == cpu_khz. Thus no need to check incarnation.
          */
+        disable_vtsc = d->arch.tsc_khz == cpu_khz;
+
+        if ( tsc_mode == TSC_MODE_DEFAULT && gtsc_khz &&
+             d->arch.vtsc_tolerance_khz )
+        {
+            uint32_t khz_diff;
+
+            khz_diff = cpu_khz > gtsc_khz ?

IMO if hardware TSC scaling is supported vtsc_tolerance_khz should be ignored, and the TSC should be scaled by the hardware always in order to provide accurate values.

Quoted 1 line

Quoted 1 line · On 29.03.18 at 10:32, <[email protected]> wrote:

Quoted 3 lines · Roger Pau Monné

IMO if hardware TSC scaling is supported vtsc_tolerance_khz should be ignored, and the TSC should be scaled by the hardware always in order to provide accurate values.

Good point, I will double check that part and do nothing if hardware scaling happens to be available on the current host.

AFAICT in the chunk above you will disable vtsc without checking if the hardware supports TSC scaling, which leads to inaccurate TSC values on hardware that could provide accurate results without the software emulation overhead.

Quoted 4 lines · Roger Pau Monné

AFAICT in the chunk above you will disable vtsc without checking if the hardware supports TSC scaling, which leads to inaccurate TSC values on hardware that could provide accurate results without the software emulation overhead.

Is that really the case? Maybe I get the logic wrong, but what I see is:
what ever my change does, or if a HVM domain runs on a host with scaling feature, disable vtsc. hvm_get_tsc_scaling_ratio has no side effects. Isnt the purpose to not emulate vtsc if the hardware supports scaling?