x86/cpuid: Untangle Invariant TSC handling
PATCH 1/1 x86/cpuid: Untangle Invariant TSC handling
ITSC being visible to the guest is currently implicit with the toolstack unconditionally asking for it, and Xen clipping it based on the vTSC and/or XEN_DOMCTL_disable_migrate settings.
This is problematic for several reasons.
First, the implicit vTSC behaviour manifests as a real bug on migration to a host with a different frequency, with ITSC but without TSC scaling capabilities, whereby the ITSC feature becomes advertised to the guest. ITSC will disappear again if the guest migrates to server with the same frequency as the original, or to one with TSC scaling support.
Secondly, disallowing ITSC unless the guest doesn't migrate is conceptually wrong. It is common to have migration pools of identical hardware, at which point the TSC frequency is the same, and more modern hardware has TSC scaling support anyway. In both cases, it is safe to advertise ITSC and migrate the guest.
Remove all implicit logic logic in Xen, and make ITSC part of the max CPUID policies for guests. Plumb an itsc parameter into xc_cpuid_apply_policy() and have libxl__cpuid_legacy() fill in the two cases where it can reasonably expect ITSC to be safe for the guest to see.
This is a behaviour change for TSC_MODE_NATIVE, where the ITSC will now reliably not appear, and for the case where the user explicitly requests ITSC, in which case it will appear even if the guest isn't marked as nomigrate.
---
CC: Jan Beulich <[email protected]>
CC: Wei Liu <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Ian Jackson <[email protected]>
CC: Anthony PERARD <[email protected]>
--- tools/libxc/include/xenctrl.h | 4 ++-- tools/libxc/xc_cpuid_x86.c | 12 ++++++------ tools/libxl/libxl_cpuid.c | 18 +++++++++++++++++- xen/arch/x86/cpuid.c | 8 -------- xen/arch/x86/time.c | 2 -- xen/include/public/arch-x86/cpufeatureset.h | 2 +- 6 files changed, 26 insertions(+), 20 deletions(-)
tools/libxc/include/xenctrl.h +2 -2
@@ -1802,12 +1802,12 @@ int xc_cpuid_set(xc_interface *xch,* Make adjustments to the CPUID settings for a domain. * * Either pass a full new @featureset (and @nr_features), or adjust individual -* features (@pae). +* features (@pae, @itsc). */ int xc_cpuid_apply_policy (xc_interface *xch,uint32_t domid,const uint32_t *featureset, -unsigned int nr_features,bool pae); +unsigned int nr_features,bool pae,bool itsc);int xc_mca_op (xc_interface *xch,struct xen_mc *mc);int xc_mca_op_inject_v2 (xc_interface *xch,unsigned int flags,xc_cpumap_t cpumap,unsigned int nr_cpus);
tools/libxc/xc_cpuid_x86.c +6 -6
@@ -438,7 +438,7 @@ int xc_cpuid_set(int xc_cpuid_apply_policy (xc_interface *xch,uint32_t domid,const uint32_t *featureset,unsigned int nr_features, -bool pae) +bool pae,bool itsc) {int rc;xc_dominfo_t di;
@@ -534,6 +534,8 @@ int xc_cpuid_apply_policy(xc_interface *xch, uint32_t domid, }else { + p->extd.itsc = itsc; +if ( di.hvm ) p->basic.pae = pae; }
@@ -621,12 +623,10 @@ int xc_cpuid_apply_policy(xc_interface *xch, uint32_t domid, }/* -* These settings are necessary to cause earlier HVM_PARAM_NESTEDHVM / -* XEN_DOMCTL_disable_migrate settings to be reflected correctly in -* CPUID. Xen will discard these bits if configuration hasn't been -* set for the domain. +* These settings are necessary to cause earlier HVM_PARAM_NESTEDHVM +* to be reflected correctly in CPUID. Xen will discard these bits if +* configuration hasn't been set for the domain. */ - p->extd.itsc = true; p->basic.vmx = true; p->extd.svm = true; }
tools/libxl/libxl_cpuid.c +17 -1
@@ -418,6 +418,7 @@ void libxl__cpuid_legacy(libxl_ctx *ctx, uint32_t domid,int i;char *cpuid_res[4 ];bool pae = true; +bool itsc;/* * For PV guests, PAE is Xen-controlled (it is the 'p' that differentiates
@@ -432,7 +433,22 @@ void libxl__cpuid_legacy(libxl_ctx *ctx, uint32_t domid,if (info->type ==LIBXL_DOMAIN_TYPE_HVM ) pae =libxl_defbool_val (info->u.hvm.pae); -xc_cpuid_apply_policy (ctx->xch, domid,NULL ,0 , pae); +/* +* Advertising Invariant TSC to a guest means that the TSC frequency won't +* change at any point in the future. +* +* We do not have enough information about potential migration +* destinations to know whether advertising ITSC is safe, but if the guest +* isn't going to migrate, then the current hardware is all that matters. +* +* Alternatively, an internal property of vTSC is that the values read are +* invariant. Advertise ITSC when we know the domain will have emualted +* TSC everywhere it goes. +*/ + itsc = (libxl_defbool_val (info->disable_migrate) || + info->tsc_mode ==LIBXL_TSC_MODE_ALWAYS_EMULATE ); + +xc_cpuid_apply_policy (ctx->xch, domid,NULL ,0 , pae, itsc);if (!cpuid)return ;
xen/arch/x86/cpuid.c +0 -8
@@ -594,14 +594,6 @@ void recalculate_cpuid_policy(struct domain *d) }/* -* ITSC is masked by default (so domains are safe to migrate), but a -* toolstack which has configured disable_migrate or vTSC for a domain may -* safely select it, and needs a way of doing so. -*/ -if ( cpu_has_itsc && (d->disable_migrate || d->arch.vtsc) ) -__set_bit (X86_FEATURE_ITSC , max_fs); - -/* * On hardware with MSR_TSX_CTRL, the admin may have elected to disable * TSX and hide the feature bits. Migrating-in VMs may have been booted * pre-mitigation when the TSX features were visbile.
xen/arch/x86/time.c +0 -2
@@ -2380,8 +2380,6 @@ int tsc_set_info(struct domain *d, } } -recalculate_cpuid_policy (d); -return 0 ; }
xen/include/public/arch-x86/cpufeatureset.h +1 -1
@@ -241,7 +241,7 @@ XEN_CPUFEATURE(RDPID, 6*32+22) /*A RDPID instruction */XEN_CPUFEATURE (CLDEMOTE ,6 *32 +25 )/*A CLDEMOTE instruction */ /* AMD-defined CPU features, CPUID level 0x80000007.edx, word 7 */ -XEN_CPUFEATURE(ITSC, 7*32+ 8) /* Invariant TSC */ +XEN_CPUFEATURE(ITSC, 7*32+ 8) /*a Invariant TSC */XEN_CPUFEATURE (EFRO ,7 *32 +10 )/* APERF/MPERF Read Only interface */ /* AMD-defined CPU features, CPUID level 0x80000008.ebx, word 8 */
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 21 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Andrew Cooper
Sent: 03 March 2020 18:25
To: Xen-devel <[email protected]>
Cc: Wei Liu <[email protected]>; Andrew Cooper <[email protected]>; Jan Beulich <[email protected]>; Anthony PERARD <[email protected]>; Ian Jackson <[email protected]>; Roger Pau Monné <[email protected]>
Subject: [Xen-devel] [PATCH] x86/cpuid: Untangle Invariant TSC handlingITSC being visible to the guest is currently implicit with the toolstack unconditionally asking for it, and Xen clipping it based on the vTSC and/or
XEN_DOMCTL_disable_migratesettings.This is problematic for several reasons.
First, the implicit vTSC behaviour manifests as a real bug on migration to a host with a different frequency, with ITSC but without TSC scaling capabilities, whereby the ITSC feature becomes advertised to the guest. ITSC will disappear again if the guest migrates to server with the same frequency as the original, or to one with TSC scaling support.
Quoted 7 lines
Secondly, disallowing ITSC unless the guest doesn't migrate is conceptually wrong. It is common to have migration pools of identical hardware, at which point the TSC frequency is the same, and more modern hardware has TSC scaling support anyway. In both cases, it is safe to advertise ITSC and migrate the guest.
Remove all implicit logic logic in Xen, and make ITSC part of the max CPUID
One too many 'logic's there.
Quoted 8 lines
policies for guests. Plumb an itsc parameter into
xc_cpuid_apply_policy()and havelibxl__cpuid_legacy()fill in the two cases where it can reasonably expect ITSC to be safe for the guest to see.This is a behaviour change for
TSC_MODE_NATIVE, where the ITSC will now reliably not appear, and for the case where the user explicitly requests ITSC, in which case it will appear even if the guest isn't marked as nomigrate.
Does this mean a guest that would have seen ITSC on 4.13 may now no longer see it in 4.14 or is the TSC_MODE_NATIVE case just the one where the feature may erroneously appear after migration?
Paul
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 21 lines · Paul
Quoted 21 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Andrew Cooper
Sent: 03 March 2020 18:25
To: Xen-devel <[email protected]>
Cc: Wei Liu <[email protected]>; Andrew Cooper <[email protected]>; Jan Beulich <[email protected]>; Anthony PERARD <[email protected]>; Ian Jackson <[email protected]>; Roger Pau Monné <[email protected]>
Subject: [Xen-devel] [PATCH] x86/cpuid: Untangle Invariant TSC handlingITSC being visible to the guest is currently implicit with the toolstack unconditionally asking for it, and Xen clipping it based on the vTSC and/or
XEN_DOMCTL_disable_migratesettings.This is problematic for several reasons.
First, the implicit vTSC behaviour manifests as a real bug on migration to a host with a different frequency, with ITSC but without TSC scaling capabilities, whereby the ITSC feature becomes advertised to the guest. ITSC will disappear again if the guest migrates to server with the same frequency as the original, or to one with TSC scaling support.
Quoted 8 lines
Quoted 7 lines
Secondly, disallowing ITSC unless the guest doesn't migrate is conceptually wrong. It is common to have migration pools of identical hardware, at which point the TSC frequency is the same, and more modern hardware has TSC scaling support anyway. In both cases, it is safe to advertise ITSC and migrate the guest.
Remove all implicit logic logic in Xen, and make ITSC part of the max CPUID
One too many 'logic's there.
Oops.
Quoted 10 lines
Quoted 9 lines
policies for guests. Plumb an itsc parameter into
xc_cpuid_apply_policy()and havelibxl__cpuid_legacy()fill in the two cases where it can reasonably expect ITSC to be safe for the guest to see.This is a behaviour change for
TSC_MODE_NATIVE, where the ITSC will now reliably not appear, and for the case where the user explicitly requests ITSC, in which case it will appear even if the guest isn't marked as nomigrate.Does this mean a guest that would have seen ITSC on 4.13 may now no longer see it in 4.14 or is the
TSC_MODE_NATIVEcase just the one where the feature may erroneously appear after migration?
In general, guests don't get to see ITSC at all, even before this change. This is something which needs working on, but it is only a tractable problem in a multi-host toolstack.
After this change, the TSC_MODE_NATIVE case will now not see a metastable ITSC feature depending on the properties of the host it happens to be on. It will default to consistently 0, unless overridden by the toolstack.
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 59 lines
-----Original Message-----
From: Andrew Cooper <[email protected]>
Sent: 04 March 2020 17:31
To: Durrant, Paul <[email protected]>; Xen-devel <[email protected]>
Cc: Wei Liu <[email protected]>; Jan Beulich <[email protected]>; Anthony PERARD <[email protected]>; Ian Jackson <[email protected]>; Roger Pau Monné <[email protected]>
Subject: RE: [EXTERNAL][Xen-devel] [PATCH] x86/cpuid: Untangle Invariant TSC handlingCAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 5 lines · Paul
Quoted 5 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Andrew Cooper
Sent: 03 March 2020 18:25
To: Xen-devel <[email protected]>
Cc: Wei Liu <[email protected]>; Andrew Cooper <[email protected]>; Jan BeulichQuoted 16 lines
Quoted 16 lines
Anthony PERARD <[email protected]>; Ian Jackson <[email protected]>; Roger Pau Monné <[email protected]>
Subject: [Xen-devel] [PATCH] x86/cpuid: Untangle Invariant TSC handlingITSC being visible to the guest is currently implicit with the toolstack unconditionally asking for it, and Xen clipping it based on the vTSC and/or
XEN_DOMCTL_disable_migratesettings.This is problematic for several reasons.
First, the implicit vTSC behaviour manifests as a real bug on migration to a host with a different frequency, with ITSC but without TSC scaling capabilities, whereby the ITSC feature becomes advertised to the guest. ITSC will disappear again if the guest migrates to server with the same frequency as the original, or to one with TSC scaling support.
Quoted 8 lines
Quoted 7 lines
Secondly, disallowing ITSC unless the guest doesn't migrate is conceptually wrong. It is common to have migration pools of identical hardware, at which point the TSC frequency is the same, and more modern hardware has TSC scaling support anyway. In both cases, it is safe to advertise ITSC and migrate the guest.
Remove all implicit logic logic in Xen, and make ITSC part of the max CPUID
One too many 'logic's there.
Oops.
Quoted 10 lines
Quoted 9 lines
policies for guests. Plumb an itsc parameter into
xc_cpuid_apply_policy()and havelibxl__cpuid_legacy()fill in the two cases where it can reasonably expect ITSC to be safe for the guest to see.This is a behaviour change for
TSC_MODE_NATIVE, where the ITSC will now reliably not appear, and for the case where the user explicitly requests ITSC, in which case it will appear even if the guest isn't marked as nomigrate.Does this mean a guest that would have seen ITSC on 4.13 may now no longer see it in 4.14 or is the
TSC_MODE_NATIVEcase just the one where the feature may erroneously appear after migration?
Quoted 8 lines
In general, guests don't get to see ITSC at all, even before this change. This is something which needs working on, but it is only a tractable problem in a multi-host toolstack.
After this change, the
TSC_MODE_NATIVEcase will now not see a metastable ITSC feature depending on the properties of the host it happens to be on. It will default to consistently 0, unless overridden by the toolstack.
Ok, as long guests running on an older Xen won't see a stable ITSC disappear when moved to a newer Xen then there should be no problem here.
Paul
Quoted 2 lines
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 12 lines · Andrew Cooper
ITSC being visible to the guest is currently implicit with the toolstack unconditionally asking for it, and Xen clipping it based on the vTSC and/or
XEN_DOMCTL_disable_migratesettings.This is problematic for several reasons.
First, the implicit vTSC behaviour manifests as a real bug on migration to a host with a different frequency, with ITSC but without TSC scaling capabilities, whereby the ITSC feature becomes advertised to the guest. ITSC will disappear again if the guest migrates to server with the same frequency as the original, or to one with TSC scaling support.
Quoted 3 lines
Secondly, disallowing ITSC unless the guest doesn't migrate is conceptually wrong. It is common to have migration pools of identical hardware, at which point the TSC frequency is the same,
This statement is too broad: Pools of identical hardware may have the same nominal frequencies, but two distinct systems are hardly ever going to have the exact same actual (measured or even real) frequencies. Recall Olaf's vTSC-tolerance patch that still hasn't landed anywhere?
Quoted 4 lines
and more modern hardware has TSC scaling
support anyway. In both cases, it is safe to advertise ITSC and migrate the guest.
Quoted 8 lines
Remove all implicit logic logic in Xen, and make ITSC part of the max CPUID policies for guests. Plumb an itsc parameter into
xc_cpuid_apply_policy()and havelibxl__cpuid_legacy()fill in the two cases where it can reasonably expect ITSC to be safe for the guest to see.This is a behaviour change for
TSC_MODE_NATIVE, where the ITSC will now reliably not appear, and for the case where the user explicitly requests ITSC, in which case it will appear even if the guest isn't marked as nomigrate.
How sensible is it to allow the user to request something like ITSC with no respective support underneath? Shouldn't we translate such a request into enabling vTSC if there's no ITSC on the platform? Actually looking at the change to libxl__cpuid_legacy() I wonder whether you don't instead mean "requests vTSC" here.
Quoted 1 line
Signed-off-by Andrew Cooper <[email protected]>
Assuming I understand the tools side changes correctly, hypervisor side
Quoted 21 lines
tools/libxl/libxl_cpuid.c +7
-1@@ -418,6 +418,7 @@ void libxl__cpuid_legacy(libxl_ctx *ctx, uint32_t domid,int i;char *cpuid_res[4 ];bool pae = true; +bool itsc;/* * For PV guests, PAE is Xen-controlled (it is the 'p' that differentiates @@ -432,7 +433,22 @@ void libxl__cpuid_legacy(libxl_ctx *ctx, uint32_t domid,if (info->type ==LIBXL_DOMAIN_TYPE_HVM ) pae =libxl_defbool_val (info->u.hvm.pae); -xc_cpuid_apply_policy (ctx->xch, domid,NULL ,0 , pae); +/* +* Advertising Invariant TSC to a guest means that the TSC frequency won't +* change at any point in the future. +* +* We do not have enough information about potential migration +* destinations to know whether advertising ITSC is safe, but if the guest
tools/libxl/libxl_cpuid.c void libxl__cpuid_legacy(libxl_ctx *ctx, uint32_t domid, · line 451
Quoted 10 lines
tools/libxl/libxl_cpuid.c +10
-0+* isn't going to migrate, then the current hardware is all that matters. +* +* Alternatively, an internal property of vTSC is that the values read are +* invariant. Advertise ITSC when we know the domain will have emualted +* TSC everywhere it goes. +*/ + itsc = (libxl_defbool_val (info->disable_migrate) || + info->tsc_mode ==LIBXL_TSC_MODE_ALWAYS_EMULATE ); + +xc_cpuid_apply_policy (ctx->xch, domid,NULL ,0 , pae, itsc);
What's the implication of this on non- or partly-libxl-based tool stacks? Won't a change like this be needed there, too? In particular, is libvirt using this function, such that we won't have a perceived regression again?
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 13 lines · Jan Beulich
Quoted 12 lines · Andrew Cooper
ITSC being visible to the guest is currently implicit with the toolstack unconditionally asking for it, and Xen clipping it based on the vTSC and/or
XEN_DOMCTL_disable_migratesettings.This is problematic for several reasons.
First, the implicit vTSC behaviour manifests as a real bug on migration to a host with a different frequency, with ITSC but without TSC scaling capabilities, whereby the ITSC feature becomes advertised to the guest. ITSC will disappear again if the guest migrates to server with the same frequency as the original, or to one with TSC scaling support.
Quoted 6 lines
Quoted 3 lines
Secondly, disallowing ITSC unless the guest doesn't migrate is conceptually wrong. It is common to have migration pools of identical hardware, at which point the TSC frequency is the same,
This statement is too broad: Pools of identical hardware may have the same nominal frequencies, but two distinct systems are hardly ever going to have the exact same actual (measured or even real) frequencies.
There is no such thing as truly invariant TSC. Even with the best hardware in the world, the reference frequency will change based on physical properties of the surroundings, including things like ambient temperature. i.e. even a single server, sitting in a datacenter is likely to see a fractional change in frequency across a 24h period.
What matters is the error margins, and how long until it manifests as a noticeable difference.
Quoted 1 line
Recall Olaf's vTSC-tolerance patch that still hasn't landed anywhere?
This is a different problem. Even on the same system, errors in Xen's frequency calculations can differ by several hundred kHz (iirc), boot to boot, making it quite useless for answering the question "am I running at the frequency the guest saw before?", which is how we just whether to intercept TSC accesses or not.
There are things which can be done about this, such as using frequency data provided by the CPU directly (rather than correlating it with a separate timesource). At that point, the only difference between two identical systems will be the variability in the reference clock, and PLL circuitry which ultimately multiplies it up from 19.2/25/100 MHz to the 1-3.5GHz typically encountered for core frequencies.
Quoted 10 lines
Quoted 10 lines
and more modern hardware has TSC scaling
support anyway. In both cases, it is safe to advertise ITSC and migrate the guest.Remove all implicit logic logic in Xen, and make ITSC part of the max CPUID policies for guests. Plumb an itsc parameter into
xc_cpuid_apply_policy()and havelibxl__cpuid_legacy()fill in the two cases where it can reasonably expect ITSC to be safe for the guest to see.
Quoted 5 lines
Quoted 3 lines
This is a behaviour change for
TSC_MODE_NATIVE, where the ITSC will now reliably not appear, and for the case where the user explicitly requests ITSC, in which case it will appear even if the guest isn't marked as nomigrate.How sensible is it to allow the user to request something like ITSC with no respective support underneath?
Right now, Xen will ignore ITSC if the hardware isn't capable, just like any other missing feature flag.
When we get the policy auditing logic in better shape, I intend to reject requests which can't be fulfilled.
Quoted 2 lines
Shouldn't we translate such a request
into enabling vTSC if there's no ITSC on the platform?
No, because a) doing things implicitly like this is the root of far too many bugs, this patch included, and b) it probably isn't what the user wants.
The reason to play around with TSC settings will ultimately to be try and avoid intercepting RDTSC, because the performance hit from interception dominates most other factors.
Quoted 3 lines
Actually looking
at the change tolibxl__cpuid_legacy()I wonder whether you don't instead mean "requests vTSC" here.
I don't see how you come to that conclusion. It is two separate cases where the toolstack can reasonably expect the guest-observed frequency not to differ.
Quoted 5 lines
Quoted 2 lines
Signed-off-by Andrew Cooper <[email protected]>Assuming I understand the tools side changes correctly, hypervisor side
Reviewed-by Jan Beulich <[email protected]>
Thanks, but the above confusion wants resolving first.
Quoted 28 lines
Quoted 28 lines
tools/libxl/libxl_cpuid.c +13
-1@@ -418,6 +418,7 @@ void libxl__cpuid_legacy(libxl_ctx *ctx, uint32_t domid,int i;char *cpuid_res[4 ];bool pae = true; +bool itsc;/* * For PV guests, PAE is Xen-controlled (it is the 'p' that differentiates @@ -432,7 +433,22 @@ void libxl__cpuid_legacy(libxl_ctx *ctx, uint32_t domid,if (info->type ==LIBXL_DOMAIN_TYPE_HVM ) pae =libxl_defbool_val (info->u.hvm.pae); -xc_cpuid_apply_policy (ctx->xch, domid,NULL ,0 , pae); +/* +* Advertising Invariant TSC to a guest means that the TSC frequency won't +* change at any point in the future. +* +* We do not have enough information about potential migration +* destinations to know whether advertising ITSC is safe, but if the guest +* isn't going to migrate, then the current hardware is all that matters. +* +* Alternatively, an internal property of vTSC is that the values read are +* invariant. Advertise ITSC when we know the domain will have emualted +* TSC everywhere it goes. +*/
Quoted 8 lines
tools/libxl/libxl_cpuid.c void libxl__cpuid_legacy(libxl_ctx *ctx, uint32_t domid, · line 451
Quoted 4 lines
tools/libxl/libxl_cpuid.c +4
-0+ itsc = (libxl_defbool_val (info->disable_migrate) || + info->tsc_mode ==LIBXL_TSC_MODE_ALWAYS_EMULATE ); + +xc_cpuid_apply_policy (ctx->xch, domid,NULL ,0 , pae, itsc);What's the implication of this on non- or partly-libxl-based tool stacks? Won't a change like this be needed there, too? In particular, is libvirt using this function, such that we won't have a perceived regression again?
This function is private to libxl, and used consistently during domain building operations.
c/s dacb80f9757c0 covered moving the predecessor from technically being part of the public API to being private, because it a) wasn't acutally used externally, and b) couldn't be used correctly by an external caller.
Longer term, I do expect multi-host toolstacks to be able to obtain enough information to sensibly decide whether ITSC can safely be advertised, based on nominally identical frequencies across the resource pool, availability of TSC scaling support, and/or an accepted tolerance.
I don't know exactly how this will look yet - there are higher priority activities with CPUID and MSR handling right now.
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 30 lines · Andrew Cooper
Quoted 13 lines · Jan Beulich
Quoted 12 lines · Andrew Cooper
ITSC being visible to the guest is currently implicit with the toolstack unconditionally asking for it, and Xen clipping it based on the vTSC and/or
XEN_DOMCTL_disable_migratesettings.This is problematic for several reasons.
First, the implicit vTSC behaviour manifests as a real bug on migration to a host with a different frequency, with ITSC but without TSC scaling capabilities, whereby the ITSC feature becomes advertised to the guest. ITSC will disappear again if the guest migrates to server with the same frequency as the original, or to one with TSC scaling support.
Quoted 6 lines
Quoted 3 lines
Secondly, disallowing ITSC unless the guest doesn't migrate is conceptually wrong. It is common to have migration pools of identical hardware, at which point the TSC frequency is the same,
This statement is too broad: Pools of identical hardware may have the same nominal frequencies, but two distinct systems are hardly ever going to have the exact same actual (measured or even real) frequencies.
There is no such thing as truly invariant TSC. Even with the best hardware in the world, the reference frequency will change based on physical properties of the surroundings, including things like ambient temperature. i.e. even a single server, sitting in a datacenter is likely to see a fractional change in frequency across a 24h period.
What matters is the error margins, and how long until it manifests as a noticeable difference.
Quoted 7 lines
Quoted 1 line
Recall Olaf's vTSC-tolerance patch that still hasn't landed anywhere?
This is a different problem. Even on the same system, errors in Xen's frequency calculations can differ by several hundred kHz (iirc), boot to boot, making it quite useless for answering the question "am I running at the frequency the guest saw before?", which is how we just whether to intercept TSC accesses or not.
But that's why I've said "too broad": Right now pools of identical hardware will not look to us as if they all had the same freq.
Quoted 6 lines
There are things which can be done about this, such as using frequency data provided by the CPU directly (rather than correlating it with a separate timesource). At that point, the only difference between two identical systems will be the variability in the reference clock, and PLL circuitry which ultimately multiplies it up from 19.2/25/100 MHz to the 1-3.5GHz typically encountered for core frequencies.
Right. The question just is how large the error margin is from the nominal frequency reported via CPUID leaves 15/16 and the actual frequency. If it's no worse than the differences we observe from our "measurement", then yes, we could and perhaps should use that data if available.
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
and more modern hardware has TSC scaling
support anyway. In both cases, it is safe to advertise ITSC and migrate the guest.Remove all implicit logic logic in Xen, and make ITSC part of the max CPUID policies for guests. Plumb an itsc parameter into
xc_cpuid_apply_policy()and havelibxl__cpuid_legacy()fill in the two cases where it can reasonably expect ITSC to be safe for the guest to see.This is a behaviour change for
TSC_MODE_NATIVE, where the ITSC will now reliably not appear, and for the case where the user explicitly requests ITSC, in which case it will appear even if the guest isn't marked as nomigrate.
Quoted 8 lines
Quoted 2 lines
How sensible is it to allow the user to request something like ITSC with no respective support underneath?
Right now, Xen will ignore ITSC if the hardware isn't capable, just like any other missing feature flag.
When we get the policy auditing logic in better shape, I intend to reject requests which can't be fulfilled.
Okay, good to know. I wonder though how well we'll be able to express in the eventual user visible error message which of the settings was actually refused.
Quoted 11 lines
Quoted 2 lines
Shouldn't we translate such a request
into enabling vTSC if there's no ITSC on the platform?No, because a) doing things implicitly like this is the root of far too many bugs, this patch included, and b) it probably isn't what the user wants.
The reason to play around with TSC settings will ultimately to be try and avoid intercepting RDTSC, because the performance hit from interception dominates most other factors.
Quoted 7 lines
Quoted 3 lines
Actually looking
at the change tolibxl__cpuid_legacy()I wonder whether you don't instead mean "requests vTSC" here.I don't see how you come to that conclusion. It is two separate cases where the toolstack can reasonably expect the guest-observed frequency not to differ.
Looking at this hunk
@@ -432,7 +433,22 @@ void libxl__cpuid_legacy(libxl_ctx *ctx, uint32_t domid,
if (info->type == LIBXL_DOMAIN_TYPE_HVM)
pae = libxl_defbool_val(info->u.hvm.pae);- xc_cpuid_apply_policy(ctx->xch, domid, NULL, 0, pae);
+ /*
+ * Advertising Invariant TSC to a guest means that the TSC frequency won't + * change at any point in the future. + * + * We do not have enough information about potential migration + * destinations to know whether advertising ITSC is safe, but if the guest + * isn't going to migrate, then the current hardware is all that matters. + * + * Alternatively, an internal property of vTSC is that the values read are + * invariant. Advertise ITSC when we know the domain will have emualted + * TSC everywhere it goes. + */
+ itsc = (libxl_defbool_val(info->disable_migrate) ||
+ info->tsc_mode == LIBXL_TSC_MODE_ALWAYS_EMULATE);
+
+ xc_cpuid_apply_policy(ctx->xch, domid, NULL, 0, pae, itsc);
I see the check of ->tsc_mode, which aiui is a request to enable vTSC unconditionally. This plus "no-migrate" get translated to enabling of ITSC.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 30 lines · Jan Beulich
Quoted 29 lines · Andrew Cooper
Quoted 13 lines · Jan Beulich
Quoted 12 lines · Andrew Cooper
ITSC being visible to the guest is currently implicit with the toolstack unconditionally asking for it, and Xen clipping it based on the vTSC and/or
XEN_DOMCTL_disable_migratesettings.This is problematic for several reasons.
First, the implicit vTSC behaviour manifests as a real bug on migration to a host with a different frequency, with ITSC but without TSC scaling capabilities, whereby the ITSC feature becomes advertised to the guest. ITSC will disappear again if the guest migrates to server with the same frequency as the original, or to one with TSC scaling support.
Quoted 6 lines
Quoted 3 lines
Secondly, disallowing ITSC unless the guest doesn't migrate is conceptually wrong. It is common to have migration pools of identical hardware, at which point the TSC frequency is the same,
This statement is too broad: Pools of identical hardware may have the same nominal frequencies, but two distinct systems are hardly ever going to have the exact same actual (measured or even real) frequencies.
There is no such thing as truly invariant TSC. Even with the best hardware in the world, the reference frequency will change based on physical properties of the surroundings, including things like ambient temperature. i.e. even a single server, sitting in a datacenter is likely to see a fractional change in frequency across a 24h period.
What matters is the error margins, and how long until it manifests as a noticeable difference.
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Recall Olaf's vTSC-tolerance patch that still hasn't landed anywhere?
This is a different problem. Even on the same system, errors in Xen's frequency calculations can differ by several hundred kHz (iirc), boot to boot, making it quite useless for answering the question "am I running at the frequency the guest saw before?", which is how we just whether to intercept TSC accesses or not.
But that's why I've said "too broad": Right now pools of identical hardware will not look to us as if they all had the same freq.
The statement is about the hardware.
Xen's (mis)measurements is just another bug in the mix, needing fixing, and not related to the paragraph.
Quoted 11 lines
Quoted 6 lines
There are things which can be done about this, such as using frequency data provided by the CPU directly (rather than correlating it with a separate timesource). At that point, the only difference between two identical systems will be the variability in the reference clock, and PLL circuitry which ultimately multiplies it up from 19.2/25/100 MHz to the 1-3.5GHz typically encountered for core frequencies.
Right. The question just is how large the error margin is from the nominal frequency reported via CPUID leaves 15/16 and the actual frequency. If it's no worse than the differences we observe from our "measurement", then yes, we could and perhaps should use that data if available.
I can't locate (even via backchannels) any written guarantee on error margins, but from what I gather, it is in practice rather more accurate than Xen's error margins.
CPUID leaves 15/16 are far from perfect - see the steady stream of corrections making their way into Linux. The most recent issue I saw was that 15/16 has no compensation for overclocking settings in the K-sku processors. Either way, there are systems now in Linux where the TSC is the sole clocksource, and the stability seems to be ok now.
In addition to the logic Linux currently uses, the TSC frequency can be obtained for Nehalem thru Broadwell in a similar way to the existing Atom logic, and for AMD, the TSC frequency can be obtained directly from the P0 frequency control MSR, which is in the BKDG/PPR and available from at least Fam10h onwards (and we really don't care about K8 these days).
If we end up with a measured TSC frequency which is very close to what the model-specific logic thinks the actual TSC frequency is, then going with the model specific version seems like a much better bet - in particular, it should make most systems come in with a nice round number.
Obviously, the first step towards this is to build the model specific logic and at least report it on boot, so we can then see what the differences are in practice.
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 4 lines
and more modern hardware has TSC scaling
support anyway. In both cases, it is safe to advertise ITSC and migrate the guest.Quoted 8 lines
Remove all implicit logic logic in Xen, and make ITSC part of the max CPUID policies for guests. Plumb an itsc parameter into
xc_cpuid_apply_policy()and havelibxl__cpuid_legacy()fill in the two cases where it can reasonably expect ITSC to be safe for the guest to see.This is a behaviour change for
TSC_MODE_NATIVE, where the ITSC will now reliably not appear, and for the case where the user explicitly requests ITSC, in which case it will appear even if the guest isn't marked as nomigrate.How sensible is it to allow the user to request something like ITSC with no respective support underneath?
Quoted 8 lines
Quoted 5 lines
Right now, Xen will ignore ITSC if the hardware isn't capable, just like any other missing feature flag.
When we get the policy auditing logic in better shape, I intend to reject requests which can't be fulfilled.
Okay, good to know. I wonder though how well we'll be able to express in the eventual user visible error message which of the settings was actually refused.
That is still very much TBD, but even the current "There was some problem with leaf $X, subleaf $Y and MSR $Z" is far better than nothing.
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
Shouldn't we translate such a request
into enabling vTSC if there's no ITSC on the platform?No, because a) doing things implicitly like this is the root of far too many bugs, this patch included, and b) it probably isn't what the user wants.
The reason to play around with TSC settings will ultimately to be try and avoid intercepting RDTSC, because the performance hit from interception dominates most other factors.
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Actually looking
at the change tolibxl__cpuid_legacy()I wonder whether you don't instead mean "requests vTSC" here.I don't see how you come to that conclusion. It is two separate cases where the toolstack can reasonably expect the guest-observed frequency not to differ.
Looking at this hunk
Ok. There are ...
tools/libxl/libxl_cpuid.c void libxl__cpuid_legacy(libxl_ctx *ctx, uint32_t domid, · line 442
Quoted 13 lines
tools/libxl/libxl_cpuid.c +7
-1@@ -432,7 +433,22 @@ void libxl__cpuid_legacy(libxl_ctx *ctx, uint32_t domid,if (info->type ==LIBXL_DOMAIN_TYPE_HVM ) pae =libxl_defbool_val (info->u.hvm.pae); -xc_cpuid_apply_policy (ctx->xch, domid,NULL ,0 , pae); +/* +* Advertising Invariant TSC to a guest means that the TSC frequency won't +* change at any point in the future. +* +* We do not have enough information about potential migration +* destinations to know whether advertising ITSC is safe, but if the guest +* isn't going to migrate, then the current hardware is all that matters.
... 1, or ...
tools/libxl/libxl_cpuid.c void libxl__cpuid_legacy(libxl_ctx *ctx, uint32_t domid, · line 446
Quoted 4 lines
tools/libxl/libxl_cpuid.c +4
-0+* +* Alternatively, an internal property of vTSC is that the values read are +* invariant. Advertise ITSC when we know the domain will have emualted +* TSC everywhere it goes.
... 2 orthogonal cases described, where xl/libxl in its current form can determine that ITSC is safe to advertise.
Quoted 8 lines
tools/libxl/libxl_cpuid.c +5
-0+*/ + itsc = (libxl_defbool_val (info->disable_migrate) || + info->tsc_mode ==LIBXL_TSC_MODE_ALWAYS_EMULATE ); + +xc_cpuid_apply_policy (ctx->xch, domid,NULL ,0 , pae, itsc);I see the check of ->tsc_mode, which aiui is a request to enable vTSC unconditionally.
vTSC in Xen is not !!tsc_mode.
In particular, libxl cannot (currently) determine whether TSC_MODE_NATIVE will result in suitable invariant properties inside the guest, because amongst other things, it depends on where the VM might migrate to in the future.
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 46 lines · Andrew Cooper
Quoted 9 lines · Jan Beulich
Quoted 7 lines · Andrew Cooper
Quoted 3 lines · Jan Beulich
Actually looking
at the change tolibxl__cpuid_legacy()I wonder whether you don't instead mean "requests vTSC" here.I don't see how you come to that conclusion. It is two separate cases where the toolstack can reasonably expect the guest-observed frequency not to differ.
Looking at this hunk
Ok. There are ...
tools/libxl/libxl_cpuid.c void libxl__cpuid_legacy(libxl_ctx *ctx, uint32_t domid, · line 442
Quoted 13 lines
tools/libxl/libxl_cpuid.c +7
-1@@ -432,7 +433,22 @@ void libxl__cpuid_legacy(libxl_ctx *ctx, uint32_t domid,if (info->type ==LIBXL_DOMAIN_TYPE_HVM ) pae =libxl_defbool_val (info->u.hvm.pae); -xc_cpuid_apply_policy (ctx->xch, domid,NULL ,0 , pae); +/* +* Advertising Invariant TSC to a guest means that the TSC frequency won't +* change at any point in the future. +* +* We do not have enough information about potential migration +* destinations to know whether advertising ITSC is safe, but if the guest +* isn't going to migrate, then the current hardware is all that matters. ... 1, or ...
tools/libxl/libxl_cpuid.c void libxl__cpuid_legacy(libxl_ctx *ctx, uint32_t domid, · line 446
Quoted 4 lines
tools/libxl/libxl_cpuid.c +4
-0+* +* Alternatively, an internal property of vTSC is that the values read are +* invariant. Advertise ITSC when we know the domain will have emualted +* TSC everywhere it goes. ... 2 orthogonal cases described, where xl/libxl in its current form can determine that ITSC is safe to advertise.
Quoted 8 lines
tools/libxl/libxl_cpuid.c +5
-0+*/ + itsc = (libxl_defbool_val (info->disable_migrate) || + info->tsc_mode ==LIBXL_TSC_MODE_ALWAYS_EMULATE ); + +xc_cpuid_apply_policy (ctx->xch, domid,NULL ,0 , pae, itsc);I see the check of ->tsc_mode, which aiui is a request to enable vTSC unconditionally.
Quoted 6 lines
vTSC in Xen is not !!tsc_mode.
In particular, libxl cannot (currently) determine whether
TSC_MODE_NATIVEwill result in suitable invariant properties inside the guest, because amongst other things, it depends on where the VM might migrate to in the future.
And I didn't say anything like this. What I did say is that TSC_MODE_ALWAYS_EMULATE is a request to enable vTSC. I didn't exclude there being other cases where vTSC would get enabled even if the tool stack didn't explicitly as for it.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel