handle tsc_tolerance during migration between identical hosts

Quoted 1 line

Quoted 1 line · On 11.04.17 at 11:39, <[email protected]> wrote:

Testing has shown that domUs with 'tsc_mode=default' can be migrated safely between identical hardware, even if the measured clock frequency differs by a few kHz. A change like shown below would allow to migrate between "2.nnGHz" hosts without enforcing emulation. If the domU is migrated to a host with "2.mmGHz" the frequency jump might have bad effects, and tsc emulation might be good even if it comes with a perfromance penalty.

The change below adds a new boot option to set a tolerance value. I think its up to the host admin to decide, therefore a global option should be enough.

Any opinions,

also on the name of the cmdline option?

I'm afraid successful testing is not a sufficient criteria here. At the very least the (so far missing) documentation needs to very clearly point out the possible risks associated with using this option. But with there not being any functional change when the option is not being made use of, I don't think there's a reason not to add this (for the adventurous).

What I'm not sure about is whether having this as a global (instead of per-guest) setting is really all that useful: As different guests may require different tolerance, having a single unique setting may not allow people to get very far.

Quoted 1 line

also on the name of the cmdline option?

I'm trying to advocate for dashes instead of underscores, and I think it should say vtsc instead of tsc, to make clear it has an effect on guests only.

Quoted 1 line

Quoted 1 line · On 18.04.17 at 11:50, <[email protected]> wrote:

Quoted 6 lines · Jan Beulich

I'm afraid successful testing is not a sufficient criteria here. At the very least the (so far missing) documentation needs to very clearly point out the possible risks associated with using this option. But with there not being any functional change when the option is not being made use of, I don't think there's a reason not to add this (for the adventurous).

This can be handled by refering to the documentation of 'tsc_mode=native', which is what this change actually does.

Quoted 4 lines

What I'm not sure about is whether having this as a global (instead of per-guest) setting is really all that useful: As different guests may require different tolerance, having a single unique setting may not allow people to get very far.

Another change on top of this, which adds a per-domU knob, could be added. This also requires a decision if the per-domU or the global knob has higher priority.

Quoted 10 lines

Quoted 1 line

Quoted 1 line · On 18.04.17 at 11:50, <[email protected]> wrote:

Quoted 6 lines · Jan Beulich

I'm afraid successful testing is not a sufficient criteria here. At the very least the (so far missing) documentation needs to very clearly point out the possible risks associated with using this option. But with there not being any functional change when the option is not being made use of, I don't think there's a reason not to add this (for the adventurous).

Quoted 2 lines

This can be handled by refering to the documentation of 'tsc_mode=native', which is what this change actually does.

Does it? I've just looked at th patch again, and I didn't find any such referral. Are you perhaps talking about some change you only have done locally?

Quoted 1 line

Quoted 1 line · On 19.04.17 at 09:46, <[email protected]> wrote:

Quoted 13 lines · Jan Beulich

Quoted 12 lines

Quoted 1 line

Quoted 1 line · On 18.04.17 at 11:50, <[email protected]> wrote:

Quoted 6 lines · Jan Beulich

I'm afraid successful testing is not a sufficient criteria here. At the very least the (so far missing) documentation needs to very clearly point out the possible risks associated with using this option. But with there not being any functional change when the option is not being made use of, I don't think there's a reason not to add this (for the adventurous).

This can be handled by refering to the documentation of 'tsc_mode=native', which is what this change actually does.

Quoted 3 lines

Does it? I've just looked at th patch again, and I didn't find any such referral. Are you perhaps talking about some change you only have done locally?

To me it looks like ->vtsc decides if there is emulation or not. So in this sense tsc_mode remains "native" if the khz value is within the tolerance range.

Testing has shown that domUs with 'tsc_mode=default' can be migrated safely between identical hardware, even if the measured clock frequency differs by a few kHz. A change like shown below would allow to migrate between "2.nnGHz" hosts without enforcing emulation. If the domU is migrated to a host with "2.mmGHz" the frequency jump might have bad effects, and tsc emulation might be good even if it comes with a perfromance penalty.

The change below adds a new boot option to set a tolerance value. I think its up to the host admin to decide, therefore a global option should be enough.

Any opinions, also on the name of the cmdline option?

Olaf

xen/arch/x86/time.c +3 -0

@@ -43,6 +43,9 @@
 static char __initdata opt_clocksource[10];
 string_param("clocksource", opt_clocksource);
 
+static unsigned int __read_mostly opt_tsc_tolerance;
+integer_param("tsc_tolerance", opt_tsc_tolerance);
+
 unsigned long __read_mostly cpu_khz;  /* CPU clock frequency in kHz. */
 DEFINE_SPINLOCK(rtc_lock);
 unsigned long pit0_ticks;

xen/arch/x86/time.c +1 -0

+    printk(XENLOG_WARNING "%s: %u %x %lx %x %x\n", __func__, d->domain_id, tsc_mode, (unsigned long)elapsed_nsec, gtsc_khz, incarnation);
     if ( is_idle_domain(d) || is_hardware_domain(d) )
     {
         d->arch.vtsc = 0;

@@ -1924,6 +1929,15 @@ void tsc_set_info(struct domain *d,
         d->arch.vtsc_offset = get_s_time() - elapsed_nsec;