new config option vtsc_tolerance_khz to avoid TSC emulation
PATCH v3 1/1 new config option vtsc_tolerance_khz to avoid TSC emulation
Add an option to control when vTSC emulation will be activated for a domU with tsc_mode=default. Without such option each TSC access from domU will be emulated, which causes a significant perfomance drop for workloads that make use of rdtsc.
Add a new domctl XEN_DOMCTL_set_vtsc_tolerance_khz to adjust the tolerance value of a running domU that is supposed to be migrated.
One option to avoid the TSC option is to run domUs with tsc_mode=native. This has the drawback that migrating a domU from a "2.3GHz" class host to a "2.4GHz" class host may change the rate at wich the TSC counter increases, the domU may not be prepared for that.
With this option the host admin can decide how a domU should behave when it is migrated across systems of the same class. Since there is always some jitter when Xen calibrates the cpu_khz value, all hosts of the same class will most likely have slightly different values. As a result vTSC emulation is unavoidable. Data collected during the incident which triggered this change showed a jitter of up to 200 KHz across systems of the same class.
A new utility is added which allows to adjust the vtsc_tolerance_khz value for running domUs. This is useful to avoid emulation for domUs that are already running and which can not be restarted.
The ordering of records sent during migration is important. The value of vtsc_tolerance_khz must be known by the receiving host before configuring TSC, because this is the place where the decision of vTSC emulation is made. Therefore the existing write_tsc_info function is modified to enforce that ordering.
v3:
- rename vtsc_khz_tolerance to vtsc_tolerance_khz
- separate domctls to adjust values
- more docs
- update libxl.h
- update python tests
- flask check bound to tsc permissions
- not runtime tested due to dlsym() build errors in staging
--- .gitignore | 1 + docs/man/xen-tscmode.pod.7 | 16 +++++ docs/man/xl.cfg.pod.5.in | 10 +++ docs/specs/libxc-migration-stream.pandoc | 27 +++++++- tools/libxc/include/xenctrl.h | 8 +++ tools/libxc/xc_domain.c | 25 +++++++ tools/libxc/xc_sr_common_x86.c | 56 +++++++++++++++- tools/libxc/xc_sr_common_x86.h | 11 +++- tools/libxc/xc_sr_restore_x86_hvm.c | 3 + tools/libxc/xc_sr_restore_x86_pv.c | 3 + tools/libxc/xc_sr_save_x86_hvm.c | 2 +- tools/libxc/xc_sr_save_x86_pv.c | 2 +- tools/libxc/xc_sr_stream_format.h | 9 +++ tools/libxl/libxl.h | 6 ++ tools/libxl/libxl_types.idl | 1 + tools/libxl/libxl_x86.c | 2 + tools/misc/Makefile | 4 ++ tools/misc/xen-vtsc.c | 108 +++++++++++++++++++++++++++++++ tools/python/xen/migration/libxc.py | 24 +++++++ tools/python/xen/migration/tests.py | 1 + tools/xl/xl_parse.c | 3 + xen/arch/x86/domctl.c | 8 +++ xen/arch/x86/time.c | 23 ++++++- xen/include/asm-x86/domain.h | 1 + xen/include/public/domctl.h | 10 +++ xen/xsm/flask/hooks.c | 2 + 26 files changed, 357 insertions(+), 9 deletions(-) create mode 100644 tools/misc/xen-vtsc.c
.gitignore +1 -0
@@ -221,6 +221,7 @@ tools/misc/xen-detect tools/misc/xen-diag tools/misc/xen-tmem-list-parse tools/misc/xen-livepatch +tools/misc/xen-vtsc tools/misc/xenperf tools/misc/xenpm tools/misc/xen-hvmctx
docs/man/xen-tscmode.pod.7 +16 -0
@@ -99,6 +99,9 @@ whether or not the VM has been saved/restored/migrated =back +If the tsc_mode is set to "default" the decision to emulate TSC can be +tweaked further with the "vtsc_tolerance_khz" option. + To understand this in more detail, the rest of this document must be read.
@@ -211,6 +214,19 @@ is emulated. Note that, though emulated, the "apparent" TSC frequency will be the TSC frequency of the initial physical machine, even after migration. +Since the calibration of the TSC frequency may not be 100% accurate, the +exact value of the frequency can change even across reboots. This means +also several otherwise identical systems can have a slightly different +TSC frequency. As a result TSC access will be emulated if a domU is +migrated from one host to another, identical host. To avoid the +performance impact of TSC emulation a certain tolerance of the measured +host TSC frequency can be specified with "vtsc_tolerance_khz". If the +measured "cpu_khz" value is within the tolerance range, TSC access +remains native. Otherwise it will be emulated. This allows to migrate +domUs between identical hardware. If the domU will be migrated to a +different kind of hardware, say from a "2.3GHz" to a "2.5GHz" system, +TSC will be emualted to maintain the TSC frequency expected by the domU. + For environments where both TSC-safeness AND highest performance even across migration is a requirement, application code can be specially modified to use an algorithm explicitly designed into Xen for this purpose.
docs/man/xl.cfg.pod.5.in +10 -0
@@ -1891,6 +1891,16 @@ determined in a similar way to that of B<default> TSC mode. Please see B<xen-tscmode(7)> for more information on this option. +=item B<vtsc_tolerance_khz="KHZ"> + +B<(x86 only, relevant only for tsc_mode=default)> +When a domU is started, the CPU frequency of the host is used by the domU for +TSC related time measurement. Once the domU is either migrated or +saved/restored on another host that CPU frequency has to be emulated to avoid +timedrift. To avoid the performance penalty of the TSC emulation, allow a +certain amount of jitter of the measured CPU frequency on the hosts the domU +is supposed to run on. + =item B<localtime=BOOLEAN> Set the real time clock to local time or to UTC. False (0) by default,
docs/specs/libxc-migration-stream.pandoc +25 -2
@@ -3,7 +3,7 @@ Andrew Cooper <<[email protected]>> Wen Congyang <<[email protected]>> Yang Hongyang <<[email protected]>> -% Revision 2 +% Revision 3 Introduction ============
@@ -231,7 +231,9 @@ type 0x00000000: END 0x0000000F: CHECKPOINT_DIRTY_PFN_LIST (Secondary -> Primary) - 0x00000010 - 0x7FFFFFFF: Reserved for future _mandatory_ + 0x00000010: VTSC_TOLERANCE_KHZ + + 0x00000011 - 0x7FFFFFFF: Reserved for future _mandatory_ records. 0x80000000 - 0xFFFFFFFF: Reserved for future _optional_
@@ -624,6 +626,23 @@ The count of pfns is: record->length/sizeof(uint64_t). \clearpage +VTSC_TOLERANCE_KHZ +------------------------- + +Domain TSC information, as accessed by the +XEN\_DOMCTL\_{get,set}\_vtsc\_khz\_tolerance hypercall sub-ops. + + 0 1 2 3 4 5 6 7 octet + +------------------------+------------------------+ + | tolerance | reserved | + +------------------------+------------------------+ + +Field Description +----------- --------------------------------------------------- +tolerance Amount of Jitter the domU can handle after migration. + +\clearpage + Layout ======
@@ -675,10 +694,14 @@ A typical save record for an x86 HVM guest image would look like: 4. TSC\_INFO 5. HVM\_PARAMS 6. HVM\_CONTEXT +7. VTSC\_TOLERANCE\_KHZ HVM\_PARAMS must precede HVM\_CONTEXT, as certain parameters can affect the validity of architectural state in the context. +VTSC\_TOLERANCE\_KHZ must precede TSC\_INFO because the decision to +emulate TSC depends on the tolerance value. + Legacy Images (x86 only) ========================
tools/libxc/include/xenctrl.h +8 -0
@@ -1355,6 +1355,14 @@ int xc_domain_set_time_offset(xc_interface *xch,uint32_t domid, int32_t time_offset_seconds); +int xc_domain_set_vtsc_tolerance_khz (xc_interface *xch, +uint32_t domid, +uint32_t vtsc_tolerance_khz); + +int xc_domain_get_vtsc_tolerance_khz (xc_interface *xch, +uint32_t domid, +uint32_t *vtsc_tolerance_khz); + int xc_domain_set_tsc_info(xc_interface *xch, uint32_t domid, uint32_t tsc_mode,
tools/libxc/xc_domain.c +25 -0
@@ -847,6 +847,31 @@ int xc_domain_disable_migrate(xc_interface *xch, uint32_t domid)return do_domctl (xch, &domctl); } +int xc_domain_set_vtsc_tolerance_khz (xc_interface *xch, +uint32_t domid, +uint32_t vtsc_tolerance_khz) +{ +DECLARE_DOMCTL ; + domctl.cmd = XEN_DOMCTL_set_vtsc_tolerance_khz; + domctl.domain = domid; + domctl.u.vtsc_tolerance_khz.tolerance = vtsc_tolerance_khz; +return do_domctl (xch, &domctl); +} + +int xc_domain_get_vtsc_tolerance_khz (xc_interface *xch, +uint32_t domid, +uint32_t *vtsc_tolerance_khz) +{ +int rc; +DECLARE_DOMCTL ; + domctl.cmd = XEN_DOMCTL_get_vtsc_tolerance_khz; + domctl.domain = domid; + rc =do_domctl (xch, &domctl); +if ( rc ==0 ) + *vtsc_tolerance_khz = domctl.u.vtsc_tolerance_khz.tolerance; +return rc; +} + int xc_domain_set_tsc_info(xc_interface *xch, uint32_t domid, uint32_t tsc_mode,
tools/libxc/xc_sr_common_x86.c +55 -1
73 lines not on this page · raw message
tools/libxc/xc_sr_common_x86.h +8 -3
@@ -4,10 +4,15 @@#include "xc_sr_common.h" /* -* Obtains a domains TSC information from Xen and writes a TSC_INFO record -* into the stream. +* Obtains a domains TSC information from Xen and writes a TSC_INFO +* and a record VTSC_TOLERANCE_KHZ into the stream. */ -int write_tsc_info (struct xc_sr_context *ctx); +int write_tsc (struct xc_sr_context *ctx); + +/* +* Parses a VTSC_TOLERANCE_KHZ record and applies the result to the domain. +*/ +int handle_vtsc_tolerance_khz (struct xc_sr_context *ctx,struct xc_sr_record *rec);/* * Parses a TSC_INFO record and applies the result to the domain.
tools/libxc/xc_sr_restore_x86_hvm.c +3 -0
@@ -178,6 +178,9 @@ static int x86_hvm_process_record(struct xc_sr_context *ctx,case REC_TYPE_HVM_PARAMS :return handle_hvm_params (ctx, rec); +case REC_TYPE_VTSC_TOLERANCE_KHZ : +return handle_vtsc_tolerance_khz (ctx, rec); +default :return RECORD_NOT_PROCESSED ; }
tools/libxc/xc_sr_restore_x86_pv.c +3 -0
@@ -1079,6 +1079,9 @@ static int x86_pv_process_record(struct xc_sr_context *ctx,case REC_TYPE_TSC_INFO :return handle_tsc_info (ctx, rec); +case REC_TYPE_VTSC_TOLERANCE_KHZ : +return handle_vtsc_tolerance_khz (ctx, rec); +default :return RECORD_NOT_PROCESSED ; }
tools/libxc/xc_sr_save_x86_hvm.c +1 -1
@@ -195,7 +195,7 @@ static int x86_hvm_end_of_checkpoint(struct xc_sr_context *ctx)int rc;/* Write the TSC record. */ - rc =write_tsc_info (ctx); + rc =write_tsc (ctx);if ( rc )return rc;
tools/libxc/xc_sr_save_x86_pv.c +1 -1
@@ -1104,7 +1104,7 @@ static int x86_pv_end_of_checkpoint(struct xc_sr_context *ctx) {int rc; - rc =write_tsc_info (ctx); + rc =write_tsc (ctx);if ( rc )return rc;
tools/libxc/xc_sr_stream_format.h +9 -0
@@ -76,6 +76,7 @@ struct xc_sr_rhdr#define REC_TYPE_VERIFY 0x0000000dU#define REC_TYPE_CHECKPOINT 0x0000000eU#define REC_TYPE_CHECKPOINT_DIRTY_PFN_LIST 0x0000000fU +#define REC_TYPE_VTSC_TOLERANCE_KHZ 0x00000010U#define REC_TYPE_OPTIONAL 0x80000000U
@@ -131,6 +132,14 @@ struct xc_sr_rec_hvm_params_entryuint64_t value; }; +/* VTSC_TOLERANCE_KHZ */ +struct xc_sr_rec_vtsc_tolerance_khz +{ +uint32_t tolerance; +uint32_t _res1; +}; + +struct xc_sr_rec_hvm_params {uint32_t count;
tools/libxl/libxl.h +6 -0
@@ -347,6 +347,12 @@#define LIBXL_HAVE_BUILDINFO_BOOTLOADER 1#define LIBXL_HAVE_BUILDINFO_BOOTLOADER_ARGS 1 +/* +* LIBXL_HAVE_VTSC_TOLERANCE_KHZ indicates that libxl_domain_build_info +* has the vtsc_tolerance_khz field. +*/ +#define LIBXL_HAVE_VTSC_TOLERANCE_KHZ 1 +/* * libxl ABI compatibility *
tools/libxl/libxl_types.idl +1 -0
@@ -466,6 +466,7 @@ libxl_domain_build_info = Struct("domain_build_info",[ ("vcpu_soft_affinity", Array(libxl_bitmap, "num_vcpu_soft_affinity")), ("numa_placement", libxl_defbool), ("tsc_mode", libxl_tsc_mode), + ("vtsc_tolerance_khz", uint32), ("max_memkb", MemKB), ("target_memkb", MemKB), ("video_memkb", MemKB),
tools/libxl/libxl_x86.c +2 -0
@@ -317,6 +317,8 @@ int libxl__arch_domain_create(libxl__gc *gc, libxl_domain_config *d_config,default :abort (); } +xc_domain_set_vtsc_tolerance_khz (ctx->xch, domid, + d_config->b_info.vtsc_tolerance_khz);xc_domain_set_tsc_info (ctx->xch, domid, tsc_mode,0 ,0 ,0 );if (libxl_defbool_val (d_config->b_info.disable_migrate))xc_domain_disable_migrate (ctx->xch, domid);
tools/misc/Makefile +4 -0
@@ -36,6 +36,7 @@ INSTALL_SBIN += $(INSTALL_SBIN-y) # Everything to be installed in a private bin/ INSTALL_PRIVBIN += xenpvnetboot +INSTALL_PRIVBIN += xen-vtsc # Everything to be installed TARGETS_ALL := $(INSTALL_BIN) $(INSTALL_SBIN) $(INSTALL_PRIVBIN)
@@ -112,6 +113,9 @@ xen-livepatch: xen-livepatch.o xen-diag: xen-diag.o $(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS) +xen-vtsc: xen-vtsc.o + $(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS) + xen-lowmemd: xen-lowmemd.o $(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenevtchn) $(LDLIBS_libxenctrl) $(LDLIBS_libxenstore) $(APPEND_LDFLAGS)
tools/misc/xen-vtsc.c new file +108 -0
108 lines not on this page · raw message
tools/python/xen/migration/libxc.py +24 -0
@@ -65,6 +65,7 @@ REC_TYPE_x86_pv_vcpu_msrs = 0x0000000c REC_TYPE_verify = 0x0000000d REC_TYPE_checkpoint = 0x0000000e REC_TYPE_checkpoint_dirty_pfn_list = 0x0000000f +REC_TYPE_vtsc_tolerance_khz = 0x00000010 rec_type_to_str = { REC_TYPE_end : "End",
@@ -83,6 +84,7 @@ rec_type_to_str = { REC_TYPE_verify : "Verify", REC_TYPE_checkpoint : "Checkpoint", REC_TYPE_checkpoint_dirty_pfn_list : "Checkpoint dirty pfn list" + REC_TYPE_vtsc_tolerance_khz : "vTSC tolerance" } # page_data
@@ -120,6 +122,9 @@ TSC_INFO_FORMAT = "IIQII" HVM_PARAMS_ENTRY_FORMAT = "QQ" HVM_PARAMS_FORMAT = "II" +# vtsc_tolerance_khz +VTSC_TOLERANCE_KHZ_FORMAT = "II" + class VerifyLibxc(VerifyBase): """ Verify a Libxc v2 stream """
@@ -424,6 +429,23 @@ class VerifyLibxc(VerifyBase): """ checkpoint dirty pfn list """ raise RecordError("Found checkpoint dirty pfn list record in stream") + def verify_record_vtsc_tolerance_khz(self, content): + """ tsc info record """ + + sz = calcsize(VTSC_TOLERANCE_KHZ_FORMAT) + + if len(content) != sz: + raise RecordError("Length should be %u bytes" % (sz, )) + + tolerance, res1 = unpack(VTSC_TOLERANCE_KHZ_FORMAT, content) + + if res1 != 0: + raise StreamError("Reserved bits set in VTSC_TOLERANCE_KHZ: 0x%08x" + % (res1, )) + + self.info(" Tolerance %u" % (tolerance)) + + record_verifiers = { REC_TYPE_end:
@@ -466,4 +488,6 @@ record_verifiers = { VerifyLibxc.verify_record_checkpoint, REC_TYPE_checkpoint_dirty_pfn_list: VerifyLibxc.verify_record_checkpoint_dirty_pfn_list, + REC_TYPE_vtsc_tolerance_khz: + VerifyLibxc.verify_record_vtsc_tolerance_khz, }
tools/python/xen/migration/tests.py +1 -0
@@ -26,6 +26,7 @@ class TestLibxc(unittest.TestCase): (libxc.TSC_INFO_FORMAT, 24), (libxc.HVM_PARAMS_ENTRY_FORMAT, 16), (libxc.HVM_PARAMS_FORMAT, 8), + (libxc.VTSC_TOLERANCE_KHZ_FORMAT, 24), ): self.assertEqual(calcsize(fmt), sz)
tools/xl/xl_parse.c +3 -0
@@ -1126,6 +1126,9 @@ void parse_config_data(const char *config_source, } } +if (!xlu_cfg_get_long (config,"vtsc_tolerance_khz" , &l,0 )) + b_info->vtsc_tolerance_khz = l; +if (!xlu_cfg_get_long (config,"rtc_timeoffset" , &l,0 )) b_info->rtc_timeoffset = l;
xen/arch/x86/domctl.c +8 -0
@@ -959,6 +959,14 @@ long arch_do_domctl( }break ; +case XEN_DOMCTL_get_vtsc_tolerance_khz: + domctl->u.vtsc_tolerance_khz.tolerance = d->arch.vtsc_tolerance_khz; +break ; + +case XEN_DOMCTL_set_vtsc_tolerance_khz: + d->arch.vtsc_tolerance_khz = domctl->u.vtsc_tolerance_khz.tolerance; +break ; +case XEN_DOMCTL_suppress_spurious_page_faults: d->arch.suppress_spurious_page_faults =1 ;break ;
xen/arch/x86/time.c +22 -1
@@ -2132,6 +2132,7 @@ void tsc_set_info(struct domain *d,switch ( d->arch.tsc_mode = tsc_mode ) { +bool disable_vtsc;bool enable_tsc_scaling;case TSC_MODE_DEFAULT :
@@ -2147,8 +2148,26 @@ void tsc_set_info(struct domain *d,* When a guest is created, gtsc_khz is passed in as zero, making * d->arch.tsc_khz == cpu_khz. Thus no need to check incarnation. */ + disable_vtsc = d->arch.tsc_khz == cpu_khz; + +if ( tsc_mode ==TSC_MODE_DEFAULT && gtsc_khz && + d->arch.vtsc_tolerance_khz ) + { +uint32_t khz_diff; + + khz_diff = cpu_khz > gtsc_khz ? + cpu_khz - gtsc_khz : gtsc_khz - cpu_khz; + disable_vtsc = khz_diff <= d->arch.vtsc_tolerance_khz; + +printk (XENLOG_G_INFO "%s: d%u: host has %lu kHz," +" domU expects %u kHz," +" difference of %u is %s tolerance of %u\n" , + __func__, d->domain_id, cpu_khz, gtsc_khz, khz_diff, + disable_vtsc ?"within" :"outside" , + d->arch.vtsc_tolerance_khz); + }if ( tsc_mode ==TSC_MODE_DEFAULT &&host_tsc_is_safe () && - (d->arch.tsc_khz == cpu_khz || + (disable_vtsc || (is_hvm_domain (d) &&hvm_get_tsc_scaling_ratio (d->arch.tsc_khz))) ) {
@@ -2237,6 +2256,8 @@ static void dump_softtsc(unsigned char key)printk (",ofs=%#" PRIx64, d->arch.vtsc_offset);if ( d->arch.tsc_khz )printk (",khz=%" PRIu32, d->arch.tsc_khz); +if ( d->arch.vtsc_tolerance_khz ) +printk (",tol=%" PRIu16, d->arch.vtsc_tolerance_khz);if ( d->arch.incarnation )printk (",inc=%" PRIu32, d->arch.incarnation);#if !defined(NDEBUG ) || defined(CONFIG_PERF_COUNTERS )
xen/include/asm-x86/domain.h +1 -0
@@ -374,6 +374,7 @@ struct arch_domainuint64_t vtsc_offset;/* adjustment for save/restore/migrate */ uint32_t tsc_khz;/* cached guest khz for certain emulated or hardware TSC scaling cases */ +uint32_t vtsc_tolerance_khz;/* domU handles that much jitter in khz value */ struct time_scale vtsc_to_ns;/* scaling for certain emulated or hardware TSC scaling cases */ struct time_scale ns_to_vtsc;/* scaling for certain emulated or
xen/include/public/domctl.h +10 -0
@@ -706,6 +706,13 @@ struct xen_domctl_tsc_info {uint64_aligned_t elapsed_nsec; }; +/* XEN_DOMCTL_get_vtsc_tolerance_khz */ +/* XEN_DOMCTL_set_vtsc_tolerance_khz */ +struct xen_domctl_vtsc_tolerance_khz { +/* IN/OUT */ +uint32_t tolerance; +}; +/* XEN_DOMCTL_gdbsx_guestmemio guest mem io */ struct xen_domctl_gdbsx_memio {/* IN */
@@ -1172,6 +1179,8 @@ struct xen_domctl {#define XEN_DOMCTL_soft_reset 79#define XEN_DOMCTL_set_gnttab_limits 80#define XEN_DOMCTL_vuart_op 81 +#define XEN_DOMCTL_get_vtsc_tolerance_khz 82 +#define XEN_DOMCTL_set_vtsc_tolerance_khz 83#define XEN_DOMCTL_gdbsx_guestmemio 1000#define XEN_DOMCTL_gdbsx_pausevcpu 1001#define XEN_DOMCTL_gdbsx_unpausevcpu 1002
@@ -1199,6 +1208,7 @@ struct xen_domctl {struct xen_domctl_settimeoffset settimeoffset;struct xen_domctl_disable_migrate disable_migrate;struct xen_domctl_tsc_info tsc_info; +struct xen_domctl_vtsc_tolerance_khz vtsc_tolerance_khz;struct xen_domctl_hvmcontext hvmcontext;struct xen_domctl_hvmcontext_partial hvmcontext_partial;struct xen_domctl_address_size address_size;
xen/xsm/flask/hooks.c +2 -0
@@ -718,9 +718,11 @@ static int flask_domctl(struct domain *d, int cmd)return current_has_perm (d,SECCLASS_DOMAIN2 ,DOMAIN2__SET_CPUID );case XEN_DOMCTL_gettscinfo: +case XEN_DOMCTL_get_vtsc_tolerance_khz:return current_has_perm (d,SECCLASS_DOMAIN2 ,DOMAIN2__GETTSC );case XEN_DOMCTL_settscinfo: +case XEN_DOMCTL_set_vtsc_tolerance_khz:return current_has_perm (d,SECCLASS_DOMAIN2 ,DOMAIN2__SETTSC );case XEN_DOMCTL_audit_p2m:
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 7 lines · Olaf Hering
Add an option to control when vTSC emulation will be activated for a domU with tsc_mode=default. Without such option each TSC access from domU will be emulated, which causes a significant perfomance drop for workloads that make use of rdtsc.
Add a new domctl
XEN_DOMCTL_set_vtsc_tolerance_khzto adjust the tolerance value of a running domU that is supposed to be migrated.
Please can we not proliferate the domctls.
This looks like it should be part of the set_tsc_info hypercall, not a separate hypercall.
Quoted 5 lines
One option to avoid the TSC option is to run domUs with tsc_mode=native. This has the drawback that migrating a domU from a "2.3GHz" class host to a "2.4GHz" class host may change the rate at wich the TSC counter increases, the domU may not be prepared for that.
Given that you identify this specifically, why do you think this change is safe or sensible in the first place? If using the options here, time will definitely drift in the VM after migrate.
Quoted 8 lines
With this option the host admin can decide how a domU should behave when it is migrated across systems of the same class. Since there is always some jitter when Xen calibrates the
cpu_khzvalue, all hosts of the same class will most likely have slightly different values. As a result vTSC emulation is unavoidable. Data collected during the incident which triggered this change showed a jitter of up to 200 KHz across systems of the same class.
We should see about using better sources of information. For one, many Intel systems actually expose the TSC frequency in the bottom of the PLATFORM_INFO MSR, although this isn't architectural, and has been replaced with CPUID information in Skylake.
Quoted 10 lines
A new utility is added which allows to adjust the
vtsc_tolerance_khzvalue for running domUs. This is useful to avoid emulation for domUs that are already running and which can not be restarted.The ordering of records sent during migration is important. The value of
vtsc_tolerance_khzmust be known by the receiving host before configuring TSC, because this is the place where the decision of vTSC emulation is made. Therefore the existingwrite_tsc_infofunction is modified to enforce that ordering.
Right, in which case this must be part of set_tsc_info and the TSC record. We already have too many problems with co-dependent ordering, and I'm trying to remove them.
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Am Tue, 6 Mar 2018 11:07:54 +0000
schrieb Andrew Cooper <[email protected]>:
Quoted 2 lines
This looks like it should be part of the
set_tsc_infohypercall, not a separate hypercall.
How would one adjust the value at runtime with set_tsc_info of a running domU? It seems to me set_tsc_info does way more than just applying some values, it uses runtime data such as get_s_time.
Olaf
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Am Tue, 6 Mar 2018 11:07:54 +0000
schrieb Andrew Cooper <[email protected]>:
Quoted 8 lines
Quoted 4 lines
One option to avoid the TSC option is to run domUs with tsc_mode=native. This has the drawback that migrating a domU from a "2.3GHz" class host to a "2.4GHz" class host may change the rate at wich the TSC counter increases, the domU may not be prepared for that.
Given that you identify this specifically, why do you think this change is safe or sensible in the first place? If using the options here, time will definitely drift in the VM after migrate.
This option is a variant of tsc_mode=default for different hosts, and tsc_mode=native for identical hosts. I think the tsc_mode can not be adjusted for running domUs. So it gives some control to the host admin about how to handle the drift.
Olaf
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Am Tue, 6 Mar 2018 11:07:54 +0000
schrieb Andrew Cooper <[email protected]>:
Quoted 4 lines
We should see about using better sources of information. For one, many Intel systems actually expose the TSC frequency in the bottom of the
PLATFORM_INFOMSR, although this isn't architectural, and has been replaced with CPUID information in Skylake.
I'm not familiar with that.
Is it what arch/x86/kernel/tsc_msr.c:cpu_khz_from_msr() gets from MSR_PLATFORM_INFO + MSR_FSB_FREQ?
Olaf
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 10 lines
Quoted 1 line
Quoted 1 line · On 06.03.18 at 12:07, <[email protected]> wrote:
Quoted 7 lines · Olaf Hering
With this option the host admin can decide how a domU should behave when it is migrated across systems of the same class. Since there is always some jitter when Xen calibrates the
cpu_khzvalue, all hosts of the same class will most likely have slightly different values. As a result vTSC emulation is unavoidable. Data collected during the incident which triggered this change showed a jitter of up to 200 KHz across systems of the same class.
Quoted 4 lines
We should see about using better sources of information. For one, many Intel systems actually expose the TSC frequency in the bottom of the
PLATFORM_INFOMSR, although this isn't architectural, and has been replaced with CPUID information in Skylake.
The question is how precise this information is - I can't imagine it is any better than the nominal clock, which then may be as far off the actual one as the jitter Olaf is talking about.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Am Tue, 6 Mar 2018 11:07:54 +0000
schrieb Andrew Cooper <[email protected]>:
Quoted 5 lines
Quoted 2 lines
Add a new domctl
XEN_DOMCTL_set_vtsc_tolerance_khzto adjust the tolerance value of a running domU that is supposed to be migrated.Please can we not proliferate the domctls.
This looks like it should be part of theset_tsc_infohypercall, not a separate hypercall.
How should this approach be implemented? To me it looks like set_tsc_info should be called only once for a not-yet-active domU, but I'm not familiar with timekeeping.
If it can be just called once, and there should be a way to adjust the tolerance value, then XEN_DOMCTL_settscinfo would need to be changed to have subcommands.
Olaf
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
So, how do we address the performance regression fixed by this patch?
Olaf
Am Mon, 12 Mar 2018 10:15:08 +0100
schrieb Olaf Hering <[email protected]>:
Quoted 9 lines
Am Tue, 6 Mar 2018 11:07:54 +0000
schrieb Andrew Cooper <[email protected]>:Quoted 5 lines
Quoted 2 lines
Add a new domctl
XEN_DOMCTL_set_vtsc_tolerance_khzto adjust the tolerance value of a running domU that is supposed to be migrated.Please can we not proliferate the domctls.
This looks like it should be part of theset_tsc_infohypercall, not a separate hypercall.
Quoted 6 lines
How should this approach be implemented? To me it looks like
set_tsc_infoshould be called only once for a not-yet-active domU, but I'm not familiar with timekeeping.
If it can be just called once, and there should be a way to adjust the tolerance value, thenXEN_DOMCTL_settscinfowould need to be changed to have subcommands.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel