Guest TSC and Xen (Intel and AMD feedback please)
Various versions of Linux under various circumstances select TSC as the primary clocksource for the kernel. This is especially true for uniprocessor kernels, but also in some cases for multiprocessor kernels. In most cases, this is because a processor bit (tsc_invariant? constant_tsc?) is passed through directly from the hardware via Xen and tested by the hvm guest and the result implies that the TSC is "stable".
I'd like to propose that, for a Xen hvm guest, TSC should NEVER be considered stable. For at least these reasons:
1) Often this test is done once at guest boot; if the guest migrates to another machine without the bit set, time will be erratic.
2) I *think* that in some cases even within the same system, TSC values will skew somewhat. Since a uniprocessor guest will often be rescheduled to a different pcpu by Xen, the underlying tsc may appear erratic.
Comments (especially from Intel and AMD)?
If agreed, could Intel and AMD provide patches so that hvm reads of the bits return "false"? Or will this cause other problems?
Another alternative would be to trap all rdtsc's and emulate them but this probably will not be easy and may have significant performance implications. But perhaps it should be an option?
Dan
===================================
Thanks... for the memory
I really could use more / My throughput's on the floor The balloon is flat / My swap disk's fat / I've OOM's in store Overcommitted so much
(with apologies to the late great Bob Hope)
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Quoted 4 lines · Dan Magenheimer
Another alternative would be to trap all rdtsc's and emulate them but this probably will not be easy and may have significant performance implications. But perhaps it should be an option?
I think this would be useful, even if just for diagnosing guest time issues.
-- Keir
Quoted 11 lines
From: Dan Magenheimer
Sent: 2008年7月2日 1:33Various versions of Linux under various circumstances select TSC as the primary clocksource for the kernel. This is especially true for uniprocessor kernels, but also in some cases for multiprocessor kernels. In most cases, this is because a processor bit (
tsc_invariant?constant_tsc?) is passed through directly from the hardware via Xen and tested by the hvm guest and the result implies that the TSC is "stable".
constant_tsc is one of the factors affecting tsc stability, and the keypoint is that one internal variable 'tsc_unstable' is initialized to zero. That means, kernel considers TSC as stable by default, and you need trigger some factors upon which guest kernel'd like to mark tsc as unstable when checking them. However by far none of those factors seem appliable to all circumstances.
For example, even by hidding constant_tsc bit from cpuid, UP guest will still consider tsc as stable, and so does 32bit SMP guest running on Intel processors.
There's one proposal in KVM side, by exposing a faked C2 states into guest. ACPI processor idle path will mark TSC as unstable when trying to enter C2. However that requires guest to enable ACPI processor support, and need some tricky logic to further mark C2 invalid to avoid performance influence after purpose is achieved...
When saying this, I'm not opposing your proposal since it does be a light option for some type of guests. My point is just that no reliabe option to mark tsc as unstable, and whether we have done all what can be done in Xen side to tackle this issue... Sorry that I didn't follow up your long discussion in this area before and thus may lose some context.
Quoted 7 lines
I'd like to propose that, for a Xen hvm guest, TSC should NEVER be considered stable. For at least these reasons:
1) Often this test is done once at guest boot; if the guest migrates to another machine without the bit set, time will be erratic.
Since it's 'if', we should treat it as an option, instead of 'NEVER', right? User may not have migration requirement in some cases, and or the migratable backups are with same bits exactly... :-)
Quoted 4 lines
2) I *think* that in some cases even within the same system, TSC values will skew somewhat. Since a uniprocessor guest will often be rescheduled to a different pcpu by Xen, the underlying tsc may appear erratic.
Can't be the tsc kept monotonic to guest, with some tweak on tsc offset? Also Xen is always trying to sync tsc among cores. As long as you don't run on a box with multiple bus crystals, or a box without constant tsc upon freq change, the tsc drift among cores should be negligible considering the overhead of migration. Then for cases out of 'as long as', we should mark TSC as unstable, still as an option.
Quoted 6 lines
Comments (especially from Intel and AMD)?
If agreed, could Intel and AMD provide patches so that hvm reads of the bits return "false"? Or will this cause other problems?
I vaguely recalled some posts about CPUID virtualization framework. There's no need to a seperate category, and it looks like this option can be aligned with that part?
Thanks,
Kevin
Quoted 6 lines
Another alternative would be to trap all rdtsc's and emulate them but this probably will not be easy and may have significant performance implications. But perhaps it should be an option?
Agree.
Thanks,
Kevin
Hi Kevin --
Quoted 11 lines
constant_tscis one of the factors affecting tsc stability, and the keypoint is that one internal variable 'tsc_unstable' is initialized to zero. That means, kernel considers TSC as stable by default, and you need trigger some factors upon which guest kernel'd like to mark tsc as unstable when checking them. However by far none of those factors seem appliable to all circumstances.For example, even by hidding
constant_tscbit from cpuid, UP guest will still consider tsc as stable, and so does 32bit SMP guest running on Intel processors.
Yes, looking over some Linux code, I see you are right. In one version of RH, if the processor vendor is Intel, it always assumes tsc is stable!
Quoted 3 lines
Since it's 'if', we should treat it as an option, instead of 'NEVER', right? User may not have migration requirement in some cases, and or the migratable backups are with same bits exactly... :-)
Yes, if it could be done, I agree an option would be better than 'NEVER'. But Keir is not too keen on more time-related options.
Quoted 5 lines
Quoted 4 lines
2) I *think* that in some cases even within the same system, TSC values will skew somewhat. Since a uniprocessor guest will often be rescheduled to a different pcpu by Xen, the underlying tsc may appear erratic.
Quoted 7 lines
Can't be the tsc kept monotonic to guest, with some tweak on tsc offset? Also Xen is always trying to sync tsc among cores. As long as you don't run on a box with multiple bus crystals, or a box without constant tsc upon freq change, the tsc drift among cores should be negligible considering the overhead of migration. Then for cases out of 'as long as', we should mark TSC as unstable, still as an option.
No Xen doesn't actally sync the tsc's, just maintains its own software offset variables. I suppose it could periodically check to make sure the tsc skew is within some reasonable value, but after the guest boots, it is probably too late.
Quoted 3 lines
I vaguely recalled some posts about CPUID virtualization framework. There's no need to a seperate category, and it looks like this option can be aligned with that part?
Yes, good point. I wasn't following that closely either but this could be covered by it.
Quoted 6 lines
Quoted 4 lines
Another alternative would be to trap all rdtsc's and emulate them but this probably will not be easy and may have significant performance implications. But perhaps it should be an option?
Agree.
Is this something that you (or Intel in general) could look at? I would be happy to participate but I don't think I understand VT well enough. Once the trap occurs, I suppose Xen system time could be used as the virtual TSC, possibly scaled up.
Thanks,
Dan
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Quoted 11 lines
From: Dan Magenheimer [mailto:[email protected]]
Sent: 2008年7月3日 9:21Quoted 8 lines
Can't be the tsc kept monotonic to guest, with some tweak on tsc offset? Also Xen is always trying to sync tsc among cores. As long as you don't run on a box with multiple bus crystals, or a box without constant tsc upon freq change, the tsc drift among cores should be negligible considering the overhead of migration. Then for cases out of 'as long as', we should mark TSC as unstable, still as an option.
Quoted 4 lines
No Xen doesn't actally sync the tsc's, just maintains its own software offset variables. I suppose it could periodically check to make sure the tsc skew is within some reasonable value, but after the guest boots, it is probably too late.
You are right. Xen only sync TSCs at boot time and then calibrate TSC pace independently on each cpu. If some factors drag TSC in the middle, then we may observe larger drift. But now I'm not sure whether periodically sync among cpus is light enough, if taking a similar convergence algothrim as boot time. But still, if only TSC drift exists (caused by occasional events instead of a constantly increasing drift from freq difference), this monotonic can be easily exposed to the guest, like derived from xen system time to adjust TSC offset at migration...
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Another alternative would be to trap all rdtsc's and emulate them but this probably will not be easy and may have significant performance implications. But perhaps it should be an option?
Quoted 7 lines
Quoted 1 line
Agree.
Is this something that you (or Intel in general) could look at? I would be happy to participate but I don't think I understand VT well enough. Once the trap occurs, I suppose Xen system time could be used as the virtual TSC, possibly scaled up.
There should be tiny related to VT, as only turning on some bit to allow RDTSC trapping and then the rest stuff should be common how to handle it. We'll take a look, but can't commit the time due to other scheduled bandwidth. But if you'd like to jump in early we definitely can help with VT side.
Thanks,
Kevin
Quoted 11 lines
Quoted 5 lines
Is this something that you (or Intel in general) could look at? I would be happy to participate but I don't think I understand VT well enough. Once the trap occurs, I suppose Xen system time could be used as the virtual TSC, possibly scaled up.
There should be tiny related to VT, as only turning on some bit to allow RDTSC trapping and then the rest stuff should be common how to handle it. We'll take a look, but can't commit the time due to other scheduled bandwidth. But if you'd like to jump in early we definitely can help with VT side.
If you can post a patch with code that:
1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtsc instruction in any hvm domain causes a trap
3) the trap handler just does a physical rdtsc and returns
then I could probably take it from there.
Dan
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Quoted 15 lines
From: Dan Magenheimer [mailto:[email protected]]
Sent: 2008年7月5日 1:32Quoted 11 lines
Quoted 5 lines
Is this something that you (or Intel in general) could look at? I would be happy to participate but I don't think I understand VT well enough. Once the trap occurs, I suppose Xen system time could be used as the virtual TSC, possibly scaled up.
There should be tiny related to VT, as only turning on some bit to allow RDTSC trapping and then the rest stuff should be common how to handle it. We'll take a look, but can't commit the time due to other scheduled bandwidth. But if you'd like to jump in early we definitely can help with VT side.
Quoted 8 lines
If you can post a patch with code that:
1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtsc instruction in any hvm domain causes a trap
3) the trap handler just does a physical rdtsc and returnsthen I could probably take it from there.
OK, and Li in CC will take a look and bake a patch for you.
Thanks,
Kevin
Hi, Dan
I just add some code to produce vmexit. The function vmx_rdtsc_intercept() is not completed. It is needed to add some code to get the guest tsc.
Thanks
--Li
Quoted 29 lines
-----Original Message-----
From: Tian, Kevin
Sent: Tuesday, July 08, 2008 9:39 AM
To: '[email protected]'; Xen-Devel (E-mail)
Cc: Zhang, Li
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)Quoted 22 lines
From: Dan Magenheimer [mailto:[email protected]]
Sent: 2008年7月5日 1:32Quoted 11 lines
Quoted 5 lines
Is this something that you (or Intel in general) could look at? I would be happy to participate but I don't think I understand VT well enough. Once the trap occurs, I suppose Xen system time could be used as the virtual TSC, possibly scaled up.
There should be tiny related to VT, as only turning on some bit to allow RDTSC trapping and then the rest stuff should be common how to handle it. We'll take a look, but can't commit the time due to other scheduled bandwidth. But if you'd like to jump in early we definitely can help with VT side.
If you can post a patch with code that:
1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtsc instruction in any hvm domain causes a trap
3) the trap handler just does a physical rdtsc and returns
Quoted 6 lines
Quoted 1 line
then I could probably take it from there.
OK, and Li in CC will take a look and bake a patch for you.
Thanks,
Kevin
xen/arch/x86/hvm/vmx/vmcs.c +1 -0
@@ -89,6 +89,7 @@ min = (CPU_BASED_HLT_EXITING | CPU_BASED_INVLPG_EXITING | + CPU_BASED_RDTSC_EXITING | CPU_BASED_CR3_LOAD_EXITING | CPU_BASED_CR3_STORE_EXITING | CPU_BASED_MONITOR_EXITING |
xen/arch/x86/hvm/vmx/vmx.c +12 -0
@@ -1374,6 +1374,13 @@HVMTRACE_LONG_2D (INVLPG , curr,/*invlpga=*/ 0 ,TRC_PAR_LONG (vaddr));if (paging_invlpg (curr, vaddr) )vpid_sync_vcpu_gva (curr, vaddr); +} + +static void vmx_rdtsc_intercept (struct vcpu *vcpu) +{ +uint64_t tsc; +rdtscll (tsc); +return ; }#define CASE_SET_REG (REG , reg) \
@@ -2193,6 +2200,11 @@vmx_invlpg_intercept (exit_qualification);break ; } +case EXIT_REASON_RDTSC : + inst_len =__get_instruction_length (); +__update_guest_eip (inst_len); +vmx_rdtsc_intercept (v); +break ;case EXIT_REASON_VMCALL : {int rc;
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Hi, Dan
This seems not to be enough. I will cook another patch.
Quoted 49 lines
-----Original Message-----
From: Zhang, Li
Sent: Tuesday, July 08, 2008 10:28 AM
To: Tian, Kevin; '[email protected]'; 'Xen-Devel (E-mail)'
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)Hi, Dan
I just add some code to produce vmexit. The function
vmx_rdtsc_intercept()is not completed. It is needed to add some code to get the guest tsc.Thanks
--LiQuoted 35 lines
-----Original Message-----
From: Tian, Kevin
Sent: Tuesday, July 08, 2008 9:39 AM
To: '[email protected]'; Xen-Devel (E-mail)
Cc: Zhang, Li
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)Quoted 15 lines
From: Dan Magenheimer [mailto:[email protected]]
Sent: 2008年7月5日 1:32Quoted 11 lines
Quoted 5 lines
Is this something that you (or Intel in general) could look at? I would be happy to participate but I don't think I understand VT well enough. Once the trap occurs, I suppose Xen system time could be used as the virtual TSC, possibly scaled up.
There should be tiny related to VT, as only turning on some bit to allow RDTSC trapping and then the rest stuff should be common how to handle it. We'll take a look, but can't commit the time due to other scheduled bandwidth. But if you'd like to jump in early we definitely can help with VT side.
Quoted 8 lines
If you can post a patch with code that:
1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtsc instruction in any hvm domain causes a trap
3) the trap handler just does a physical rdtsc and returnsthen I could probably take it from there.
OK, and Li in CC will take a look and bake a patch for you.
Thanks,
Kevin
Thanks Kevin and Li --
A couple of questions:
1) If the EXITING flag to be set in vmcs.c is to be controlled by a xen boot option, would this work:
static int opt_softtsc = 0;boolean_param("softtsc", opt_softtsc);
:
min = /* original code */
if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;
2) In vmx_rdtsc_intercept(/* need regs as param*/) if we change the rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and then set regs->edx amd regs->eax from tmptsc, are we almost done?
3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets called to force the trap?
4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is there already code somewhere to emulate WRMSR of tsc?
Sorry I am not very familar with the details of the ia32 instruction set.
Thanks,
Dan
Quoted 49 lines
-----Original Message-----
From: Zhang, Li [mailto:[email protected]]
Sent: Monday, July 07, 2008 9:00 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)Hi, Dan
This seems not to be enough. I will cook another patch.Quoted 5 lines
-----Original Message-----
From: Zhang, Li
Sent: Tuesday, July 08, 2008 10:28 AM
To: Tian, Kevin; '[email protected]'; 'Xen-Devel (E-mail)'
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMDfeedback please)
Quoted 4 lines
Hi, Dan
I just add some code to produce vmexit. The function
vmx_rdtsc_intercept()Quoted 1 line
is not completed. It is needed to add some code to get the
guest tsc.
Quoted 10 lines
Thanks
--LiQuoted 6 lines
-----Original Message-----
From: Tian, Kevin
Sent: Tuesday, July 08, 2008 9:39 AM
To: '[email protected]'; Xen-Devel (E-mail)
Cc: Zhang, Li
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel andAMD feedback please)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
From: Dan Magenheimer [mailto:[email protected]]
Sent: 2008年7月5日 1:32Quoted 1 line
Quoted 1 line
Is this something that you (or Intel in general)
could look at?
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
I would be happy to participate but I don't think I
understand
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
VT well enough. Once the trap occurs, I suppose
Xen system time
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
could be used as the virtual TSC, possibly scaled up.
Quoted 23 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
There should be tiny related to VT, as only turning
on some bit to
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
allow RDTSC trapping and then the rest stuff should be common how to handle it. We'll take a look, but can't
commit the time due
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to other scheduled bandwidth. But if you'd like to
jump in early
Quoted 15 lines
Quoted 15 lines
Quoted 10 lines
Quoted 1 line
we definitely can help with VT side.
If you can post a patch with code that:
1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtsc instruction in any hvm domain causes a trap
3) the trap handler just does a physical rdtsc and returnsthen I could probably take it from there.
OK, and Li in CC will take a look and bake a patch for you.
Thanks,
Kevin
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Quoted 11 lines
From: Dan Magenheimer [mailto:[email protected]]
Sent: 2008年7月8日 11:341) If the EXITING flag to be set in
vmcs.cis to be controlled by a xen boot option, would this work:static int
opt_softtsc= 0;boolean_param("softtsc", opt_softtsc);
:
min = /* original code */
if (opt_softtsc) min |=CPU_BASED_RDTSC_EXITING;
yes.
Quoted 4 lines
2) In
vmx_rdtsc_intercept(/* need regs as param*/)if we change the rdtscll(tsc) to be tmptsc =hvm_get_guest_tsc()and then setregs->edxamdregs->eaxfrom tmptsc, are we almost done?
Yes, and similar example is under case EXIT_REASON_CPUID.
Quoted 3 lines
3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets called to force the trap?
My feeling is that both Linux and Windows allows user directly read to TSC, and thus CR4.TSD is unlikely to be set. But it needs confirmation. If you really want to trap such setting, hvm_set_cr4 is the point and you may further reference vmx_update_guest_cr if it's vmx specific (but I guess it should be common).
But I doubt the point to support such model, since two levels trapping (Xen trap, inject fault, guest kernel read, xen trap, ...) which adds much delay on Xen maintained guest tsc.
Quoted 3 lines
4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is there already code somewhere to emulate WRMSR of tsc?
I'm not sure about this one, and others may answer.
Thanks,
Kevin
Hi, Dan
This is the patch which I think it has been completed VT. But there is an instable issue. The reason may be from that this interception is not bound to timer mode yet.
Thanks
--Li
Quoted 27 lines
-----Original Message-----
From: Dan Magenheimer [mailto:[email protected]]
Sent: Tuesday, July 08, 2008 11:34 AM
To: Zhang, Li; Xen-Devel (E-mail)
Cc: Tian, Kevin; [email protected]
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)Thanks Kevin and Li --
A couple of questions:
1) If the EXITING flag to be set in
vmcs.cis to be controlled by a xen boot option, would this work:static int
opt_softtsc= 0;boolean_param("softtsc", opt_softtsc); :min = /* original code */
if (opt_softtsc) min |=CPU_BASED_RDTSC_EXITING;2) In
vmx_rdtsc_intercept(/* need regs as param*/)if we change the rdtscll(tsc) to be tmptsc =hvm_get_guest_tsc()and then setregs->edxamdregs->eaxfrom tmptsc, are we almost done?3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets called to force the trap?
Quoted 80 lines
4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is there already code somewhere to emulate WRMSR of tsc?
Sorry I am not very familar with the details of the ia32 instruction set.
Thanks,
DanQuoted 72 lines
-----Original Message-----
From: Zhang, Li [mailto:[email protected]]
Sent: Monday, July 07, 2008 9:00 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)Hi, Dan
This seems not to be enough. I will cook another patch.Quoted 5 lines
-----Original Message-----
From: Zhang, Li
Sent: Tuesday, July 08, 2008 10:28 AM
To: Tian, Kevin; '[email protected]'; 'Xen-Devel (E-mail)'
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMDfeedback please)
Quoted 4 lines
Hi, Dan
I just add some code to produce vmexit. The function
vmx_rdtsc_intercept()Quoted 1 line
is not completed. It is needed to add some code to get the
guest tsc.
Quoted 10 lines
Thanks
--LiQuoted 6 lines
-----Original Message-----
From: Tian, Kevin
Sent: Tuesday, July 08, 2008 9:39 AM
To: '[email protected]'; Xen-Devel (E-mail)
Cc: Zhang, Li
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel andAMD feedback please)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
From: Dan Magenheimer [mailto:[email protected]]
Sent: 2008年7月5日 1:32Quoted 1 line
Quoted 1 line
Is this something that you (or Intel in general)
could look at?
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
I would be happy to participate but I don't think I
understand
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
VT well enough. Once the trap occurs, I suppose
Xen system time
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
could be used as the virtual TSC, possibly scaled up.
There should be tiny related to VT, as only turning
on some bit to
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
allow RDTSC trapping and then the rest stuff should be common how to handle it. We'll take a look, but can't
commit the time due
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to other scheduled bandwidth. But if you'd like to
jump in early
Quoted 15 lines
Quoted 15 lines
Quoted 10 lines
Quoted 1 line
we definitely can help with VT side.
If you can post a patch with code that:
1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtsc instruction in any hvm domain causes a trap
3) the trap handler just does a physical rdtsc and returnsthen I could probably take it from there.
OK, and Li in CC will take a look and bake a patch for you.
Thanks,
Kevin
xen/arch/x86/hvm/vmx/vmcs.c +3 -0
@@ -40,6 +40,7 @@static int opt_vpid_enabled =1 ;boolean_param ("vpid" , opt_vpid_enabled); +int softtsc =1 ;/* Dynamic (run-time adjusted) execution control flags. */ u32 vmx_pin_based_exec_control__read_mostly ;
@@ -96,6 +97,8 @@CPU_BASED_MOV_DR_EXITING |CPU_BASED_ACTIVATE_IO_BITMAP | CPU_BASED_USE_TSC_OFFSETING); +if (softtsc) + min |=CPU_BASED_RDTSC_EXITING ; opt = (CPU_BASED_ACTIVATE_MSR_BITMAP |CPU_BASED_TPR_SHADOW |CPU_BASED_ACTIVATE_SECONDARY_CONTROLS );
xen/arch/x86/hvm/vmx/vmx.c +16 -0
@@ -1375,6 +1375,17 @@if (paging_invlpg (curr, vaddr) )vpid_sync_vcpu_gva (curr, vaddr); } + +static void vmx_rdtsc_intercept (struct cpu_user_regs *regs) +{ +uint64_t tsc; +struct vcpu *v = current; + + tsc =hvm_get_guest_tsc (v); + regs->eax = tsc &0xffff ; + regs->edx = tsc >>32 &0xffff ; +} +#define CASE_SET_REG (REG , reg) \ case VMX_CONTROL_REG_ACCESS_GPR_ ## REG: regs->reg = value; break
@@ -2192,6 +2203,11 @@vmx_invlpg_intercept (exit_qualification);break ; } +case EXIT_REASON_RDTSC : + inst_len =__get_instruction_length (); +__update_guest_eip (inst_len); +vmx_rdtsc_intercept (regs); +break ;case EXIT_REASON_VMCALL : {int rc;
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
In fact, the issue is from guest. In the guest's terminal, it shows that "your time source seems to be instable..."
Quoted 126 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Zhang, Li
Sent: Tuesday, July 08, 2008 2:49 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)Hi, Dan
This is the patch which I think it has been completed VT. But there is an instable issue. The reason may be from that this interception is not bound to timer mode yet.
Thanks
--LiQuoted 7 lines
-----Original Message-----
From: Dan Magenheimer [mailto:[email protected]]
Sent: Tuesday, July 08, 2008 11:34 AM
To: Zhang, Li; Xen-Devel (E-mail)
Cc: Tian, Kevin; [email protected]
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)Quoted 5 lines
Thanks Kevin and Li --
A couple of questions:
1) If the EXITING flag to be set in
vmcs.cis to be controlled by a xenboot
Quoted 5 lines
option, would this work:
static int
opt_softtsc= 0;boolean_param("softtsc", opt_softtsc); :Quoted 8 lines
min = /* original code */
if (opt_softtsc) min |=CPU_BASED_RDTSC_EXITING;2) In
vmx_rdtsc_intercept(/* need regs as param*/)if we change the rdtscll(tsc) to be tmptsc =hvm_get_guest_tsc()and then setregs->edxamdregs->eaxfrom tmptsc, are we almost done?3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets
called
Quoted 82 lines
to force the trap?
4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is there already code somewhere to emulate WRMSR of tsc?
Sorry I am not very familar with the details of the ia32 instruction set.
Thanks,
DanQuoted 72 lines
-----Original Message-----
From: Zhang, Li [mailto:[email protected]]
Sent: Monday, July 07, 2008 9:00 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)Hi, Dan
This seems not to be enough. I will cook another patch.Quoted 5 lines
-----Original Message-----
From: Zhang, Li
Sent: Tuesday, July 08, 2008 10:28 AM
To: Tian, Kevin; '[email protected]'; 'Xen-Devel (E-mail)'
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMDfeedback please)
Quoted 4 lines
Hi, Dan
I just add some code to produce vmexit. The function
vmx_rdtsc_intercept()Quoted 1 line
is not completed. It is needed to add some code to get the
guest tsc.
Quoted 10 lines
Thanks
--LiQuoted 6 lines
-----Original Message-----
From: Tian, Kevin
Sent: Tuesday, July 08, 2008 9:39 AM
To: '[email protected]'; Xen-Devel (E-mail)
Cc: Zhang, Li
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel andAMD feedback please)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
From: Dan Magenheimer [mailto:[email protected]]
Sent: 2008年7月5日 1:32Quoted 1 line
Quoted 1 line
Is this something that you (or Intel in general)
could look at?
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
I would be happy to participate but I don't think I
understand
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
VT well enough. Once the trap occurs, I suppose
Xen system time
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
could be used as the virtual TSC, possibly scaled up.
There should be tiny related to VT, as only turning
on some bit to
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
allow RDTSC trapping and then the rest stuff should be common how to handle it. We'll take a look, but can't
commit the time due
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to other scheduled bandwidth. But if you'd like to
jump in early
Quoted 15 lines
Quoted 15 lines
Quoted 10 lines
Quoted 1 line
we definitely can help with VT side.
If you can post a patch with code that:
1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtsc instruction in any hvm domain causes a trap
3) the trap handler just does a physical rdtsc and returnsthen I could probably take it from there.
OK, and Li in CC will take a look and bake a patch for you.
Thanks,
Kevin
Hi, Dan
Please ignore previous comments. The above attachment is OK. Sorry for that. :(
Quoted 138 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Zhang, Li
Sent: Tuesday, July 08, 2008 2:58 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)In fact, the issue is from guest. In the guest's terminal, it shows that "your time source seems to be instable..."
Quoted 8 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Zhang, Li
Sent: Tuesday, July 08, 2008 2:49 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)Quoted 15 lines
Hi, Dan
This is the patch which I think it has been completed VT. But there is an instable issue. The reason may be from that this interception is not bound to timer mode yet.
Thanks
--LiQuoted 6 lines
-----Original Message-----
From: Dan Magenheimer [mailto:[email protected]]
Sent: Tuesday, July 08, 2008 11:34 AM
To: Zhang, Li; Xen-Devel (E-mail)
Cc: Tian, Kevin; [email protected]
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedbackplease)
Quoted 17 lines
Quoted 6 lines
Thanks Kevin and Li --
A couple of questions:
1) If the EXITING flag to be set in
vmcs.cis to be controlled by a xenboot
Quoted 10 lines
option, would this work:
static int
opt_softtsc= 0;boolean_param("softtsc", opt_softtsc); :min = /* original code */
if (opt_softtsc) min |=CPU_BASED_RDTSC_EXITING;2) In
vmx_rdtsc_intercept(/* need regs as param*/)if we change the rdtscll(tsc) to be tmptsc =hvm_get_guest_tsc()and then setregs->edxamd
Quoted 10 lines
Quoted 3 lines
regs->eaxfrom tmptsc, are we almost done?3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets
called
Quoted 6 lines
to force the trap?
4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is there already code somewhere to emulate WRMSR of tsc?
Sorry I am not very familar with the details of the ia32 instruction
set.
Quoted 13 lines
Quoted 13 lines
Thanks,
DanQuoted 9 lines
-----Original Message-----
From: Zhang, Li [mailto:[email protected]]
Sent: Monday, July 07, 2008 9:00 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Hi, Dan
This seems not to be enough. I will cook another patch.Quoted 4 lines
-----Original Message-----
From: Zhang, Li
Sent: Tuesday, July 08, 2008 10:28 AM
To: Tian, Kevin; '[email protected]'; 'Xen-Devel(E-mail)'
Quoted 53 lines
Quoted 53 lines
Quoted 53 lines
Quoted 1 line
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
feedback please)
Quoted 4 lines
Hi, Dan
I just add some code to produce vmexit. The function
vmx_rdtsc_intercept()Quoted 1 line
is not completed. It is needed to add some code to get the
guest tsc.
Quoted 10 lines
Thanks
--LiQuoted 6 lines
-----Original Message-----
From: Tian, Kevin
Sent: Tuesday, July 08, 2008 9:39 AM
To: '[email protected]'; Xen-Devel (E-mail)
Cc: Zhang, Li
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel andAMD feedback please)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
From: Dan Magenheimer [mailto:[email protected]]
Sent: 2008年7月5日 1:32Quoted 1 line
Quoted 1 line
Is this something that you (or Intel in general)
could look at?
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
I would be happy to participate but I don't think I
understand
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
VT well enough. Once the trap occurs, I suppose
Xen system time
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
could be used as the virtual TSC, possibly scaled up.
There should be tiny related to VT, as only turning
on some bit to
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
allow RDTSC trapping and then the rest stuff should be common how to handle it. We'll take a look, but can't
commit the time due
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to other scheduled bandwidth. But if you'd like to
jump in early
Quoted 13 lines
Quoted 13 lines
Quoted 10 lines
Quoted 1 line
we definitely can help with VT side.
If you can post a patch with code that:
1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtsc instruction in any hvm domain causes a trap
3) the trap handler just does a physical rdtsc and returnsthen I could probably take it from there.
OK, and Li in CC will take a look and bake a patch for you.
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
Thanks,
Kevin_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
xen/arch/x86/hvm/vmx/vmcs.c +3 -0
@@ -40,6 +40,7 @@static int opt_vpid_enabled =1 ;boolean_param ("vpid" , opt_vpid_enabled); +int softtsc =1 ;/* Dynamic (run-time adjusted) execution control flags. */ u32 vmx_pin_based_exec_control__read_mostly ;
@@ -96,6 +97,8 @@CPU_BASED_MOV_DR_EXITING |CPU_BASED_ACTIVATE_IO_BITMAP | CPU_BASED_USE_TSC_OFFSETING); +if (softtsc) + min |=CPU_BASED_RDTSC_EXITING ; opt = (CPU_BASED_ACTIVATE_MSR_BITMAP |CPU_BASED_TPR_SHADOW |CPU_BASED_ACTIVATE_SECONDARY_CONTROLS );
xen/arch/x86/hvm/vmx/vmx.c +16 -0
@@ -1375,6 +1375,17 @@if (paging_invlpg (curr, vaddr) )vpid_sync_vcpu_gva (curr, vaddr); } + +static void vmx_rdtsc_intercept (struct cpu_user_regs *regs) +{ +uint64_t tsc; +struct vcpu *v = current; + + tsc =hvm_get_guest_tsc (v); + regs->eax = tsc &0xffffffff ; + regs->edx = tsc >>32 &0xffffffff ; +} +#define CASE_SET_REG (REG , reg) \ case VMX_CONTROL_REG_ACCESS_GPR_ ## REG: regs->reg = value; break
@@ -2192,6 +2203,11 @@vmx_invlpg_intercept (exit_qualification);break ; } +case EXIT_REASON_RDTSC : + inst_len =__get_instruction_length (); +__update_guest_eip (inst_len); +vmx_rdtsc_intercept (regs); +break ;case EXIT_REASON_VMCALL : {int rc;
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Hi Li,
Shouldn't this
+ regs->eax = tsc & 0xffff;
+ regs->edx = tsc >> 32 & 0xffff;
be
+ regs->eax = tsc & 0xffffffff;
+ regs->edx = tsc >> 32 & 0xffffffff;
Thanks,
Dave
Quoted 161 lines · Li
Hi, Dan
Please ignore previous comments. The above attachment is OK. Sorry for that. :(Quoted 156 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Zhang, Li
Sent: Tuesday, July 08, 2008 2:58 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)In fact, the issue is from guest. In the guest's terminal, it shows that "your time source seems to be instable..."
Quoted 8 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Zhang, Li
Sent: Tuesday, July 08, 2008 2:49 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)Quoted 15 lines
Hi, Dan
This is the patch which I think it has been completed VT. But there is an instable issue. The reason may be from that this interception is not bound to timer mode yet.
Thanks
--LiQuoted 6 lines
-----Original Message-----
From: Dan Magenheimer [mailto:[email protected]]
Sent: Tuesday, July 08, 2008 11:34 AM
To: Zhang, Li; Xen-Devel (E-mail)
Cc: Tian, Kevin; [email protected]
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedbackplease)
Quoted 17 lines
Quoted 6 lines
Thanks Kevin and Li --
A couple of questions:
1) If the EXITING flag to be set in
vmcs.cis to be controlled by a xenboot
Quoted 10 lines
option, would this work:
static int
opt_softtsc= 0;boolean_param("softtsc", opt_softtsc); :min = /* original code */
if (opt_softtsc) min |=CPU_BASED_RDTSC_EXITING;2) In
vmx_rdtsc_intercept(/* need regs as param*/)if we change the rdtscll(tsc) to be tmptsc =hvm_get_guest_tsc()and then setregs->edxamd
Quoted 10 lines
Quoted 3 lines
regs->eaxfrom tmptsc, are we almost done?3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets
called
Quoted 6 lines
to force the trap?
4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is there already code somewhere to emulate WRMSR of tsc?
Sorry I am not very familar with the details of the ia32 instruction
set.
Quoted 13 lines
Quoted 13 lines
Thanks,
DanQuoted 9 lines
-----Original Message-----
From: Zhang, Li [mailto:[email protected]]
Sent: Monday, July 07, 2008 9:00 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Hi, Dan
This seems not to be enough. I will cook another patch.Quoted 4 lines
-----Original Message-----
From: Zhang, Li
Sent: Tuesday, July 08, 2008 10:28 AM
To: Tian, Kevin; '[email protected]'; 'Xen-Devel(E-mail)'
Quoted 49 lines
Quoted 49 lines
Quoted 49 lines
Quoted 1 line
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
feedback please)
Quoted 4 lines
Hi, Dan
I just add some code to produce vmexit. The function
vmx_rdtsc_intercept()Quoted 1 line
is not completed. It is needed to add some code to get the
guest tsc.
Quoted 10 lines
Thanks
--LiQuoted 6 lines
-----Original Message-----
From: Tian, Kevin
Sent: Tuesday, July 08, 2008 9:39 AM
To: '[email protected]'; Xen-Devel (E-mail)
Cc: Zhang, Li
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel andAMD feedback please)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
From: Dan Magenheimer [mailto:[email protected]]
Sent: 2008年7月5日 1:32Quoted 1 line
Quoted 1 line
Is this something that you (or Intel in general)
could look at?
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
I would be happy to participate but I don't think I
understand
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
VT well enough. Once the trap occurs, I suppose
Xen system time
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
could be used as the virtual TSC, possibly scaled up.
There should be tiny related to VT, as only turning
on some bit to
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
allow RDTSC trapping and then the rest stuff should be common how to handle it. We'll take a look, but can't
commit the time due
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to other scheduled bandwidth. But if you'd like to
jump in early
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 1 line
we definitely can help with VT side.
If you can post a patch with code that:
1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtsc instruction in any hvm domain causes a trap
3) the trap handler just does a physical rdtsc and returnsQuoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
then I could probably take it from there.
OK, and Li in CC will take a look and bake a patch for you.
Thanks,
Kevin_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel------------------------------------------------------------------------
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
I would go for (uint32_t)tsc and (uint32_t)(tsc >> 32) as being even clearer.
-- Keir
Quoted 187 lines · Dave Winchell
Hi Li,
Shouldn't this
+
regs->eax= tsc &0xffff;
+regs->edx= tsc >> 32 &0xffff;be
+
regs->eax= tsc &0xffffffff;
+regs->edx= tsc >> 32 &0xffffffff;Thanks,
DaveQuoted 163 lines · Li
Hi, Dan
Please ignore previous comments. The above attachment is OK. Sorry for that. :(Quoted 157 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Zhang, Li
Sent: Tuesday, July 08, 2008 2:58 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)In fact, the issue is from guest. In the guest's terminal, it shows that "your
time source seems to be instable..."Quoted 8 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Zhang, Li
Sent: Tuesday, July 08, 2008 2:49 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)Quoted 15 lines
Hi, Dan
This is the patch which I think it has been completed VT. But there is an instable issue. The reason may be from that this interception is not bound to timer mode yet.
Thanks
--LiQuoted 6 lines
-----Original Message-----
From: Dan Magenheimer [mailto:[email protected]]
Sent: Tuesday, July 08, 2008 11:34 AM
To: Zhang, Li; Xen-Devel (E-mail)
Cc: Tian, Kevin; [email protected]
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedbackplease)
Quoted 17 lines
Quoted 6 lines
Thanks Kevin and Li --
A couple of questions:
1) If the EXITING flag to be set in
vmcs.cis to be controlled by a xenboot
Quoted 10 lines
option, would this work:
static int
opt_softtsc= 0;boolean_param("softtsc", opt_softtsc); :min = /* original code */
if (opt_softtsc) min |=CPU_BASED_RDTSC_EXITING;2) In
vmx_rdtsc_intercept(/* need regs as param*/)if we change the rdtscll(tsc) to be tmptsc =hvm_get_guest_tsc()and then setregs->edxamd
Quoted 10 lines
Quoted 3 lines
regs->eaxfrom tmptsc, are we almost done?3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets
called
Quoted 6 lines
to force the trap?
4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is there already code somewhere to emulate WRMSR of tsc?
Sorry I am not very familar with the details of the ia32 instruction
set.
Quoted 13 lines
Quoted 13 lines
Thanks,
DanQuoted 9 lines
-----Original Message-----
From: Zhang, Li [mailto:[email protected]]
Sent: Monday, July 07, 2008 9:00 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Hi, Dan
This seems not to be enough. I will cook another patch.Quoted 4 lines
-----Original Message-----
From: Zhang, Li
Sent: Tuesday, July 08, 2008 10:28 AM
To: Tian, Kevin; '[email protected]'; 'Xen-Devel(E-mail)'
Quoted 49 lines
Quoted 49 lines
Quoted 49 lines
Quoted 1 line
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
feedback please)
Quoted 4 lines
Hi, Dan
I just add some code to produce vmexit. The function
vmx_rdtsc_intercept()Quoted 1 line
is not completed. It is needed to add some code to get the
guest tsc.
Quoted 10 lines
Thanks
--LiQuoted 6 lines
-----Original Message-----
From: Tian, Kevin
Sent: Tuesday, July 08, 2008 9:39 AM
To: '[email protected]'; Xen-Devel (E-mail)
Cc: Zhang, Li
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel andAMD feedback please)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
From: Dan Magenheimer [mailto:[email protected]]
Sent: 2008年7月5日 1:32Quoted 1 line
Quoted 1 line
Is this something that you (or Intel in general)
could look at?
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
I would be happy to participate but I don't think I
understand
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
VT well enough. Once the trap occurs, I suppose
Xen system time
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
could be used as the virtual TSC, possibly scaled up.
There should be tiny related to VT, as only turning
on some bit to
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
allow RDTSC trapping and then the rest stuff should be common how to handle it. We'll take a look, but can't
commit the time due
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to other scheduled bandwidth. But if you'd like to
jump in early
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 1 line
we definitely can help with VT side.
If you can post a patch with code that:
1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtsc instruction in any hvm domain causes a trap
3) the trap handler just does a physical rdtsc and returnsQuoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
then I could probably take it from there.
OK, and Li in CC will take a look and bake a patch for you.
Thanks,
Kevin_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel------------------------------------------------------------------------
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Looks great! I am continuing to code and test. Some preliminary results:
1) Substituting hvm_get_guest_time() for hvm_get_guest_tsc() seems to just work! The hvm guest believes it is running on a 1000.185 MHz clock.
2) On my test box, a guest rdtsc without softtsc takes about 30ns. With softtsc, rdtsc takes about 1us.
3) On my user-land test program, I am seeing rare "Time goes backwards", even with hvm_get_guest_time(). I'm trying to track this down.
Dan
Quoted 167 lines
-----Original Message-----
From: Zhang, Li [mailto:[email protected]]
Sent: Tuesday, July 08, 2008 3:47 AM
To: Zhang, Li; [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)Hi, Dan
Please ignore previous comments. The above attachment is OK. Sorry for that. :(Quoted 3 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On BehalfOf Zhang, Li
Quoted 4 lines
Sent: Tuesday, July 08, 2008 2:58 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMDfeedback please)
Quoted 2 lines
In fact, the issue is from guest. In the guest's terminal,
it shows that "your
Quoted 5 lines
time source seems to be instable..."
Quoted 3 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] OnBehalf Of Zhang, Li
Quoted 4 lines
Quoted 4 lines
Sent: Tuesday, July 08, 2008 2:49 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel andAMD feedback please)
Quoted 4 lines
Quoted 4 lines
Hi, Dan
This is the patch which I think it has been completed
VT. But there is an
Quoted 1 line
Quoted 1 line
instable issue. The reason may be from that this
interception is not bound
Quoted 11 lines
Quoted 11 lines
to timer mode yet.
Thanks
--LiQuoted 6 lines
-----Original Message-----
From: Dan Magenheimer [mailto:[email protected]]
Sent: Tuesday, July 08, 2008 11:34 AM
To: Zhang, Li; Xen-Devel (E-mail)
Cc: Tian, Kevin; [email protected]
Subject: RE: [Xen-devel] Guest TSC and Xen (Inteland AMD feedback
Quoted 7 lines
please)
Quoted 6 lines
Quoted 6 lines
Thanks Kevin and Li --
A couple of questions:
1) If the EXITING flag to be set in
vmcs.cis to becontrolled by a xen
Quoted 10 lines
Quoted 10 lines
boot
Quoted 9 lines
option, would this work:
static int
opt_softtsc= 0;boolean_param("softtsc", opt_softtsc); :min = /* original code */
if (opt_softtsc) min |=CPU_BASED_RDTSC_EXITING;2) In
vmx_rdtsc_intercept(/* need regs as param*/)if we change the
Quoted 1 line
Quoted 1 line
Quoted 1 line
rdtscll(tsc) to be tmptsc =
hvm_get_guest_tsc()andthen set
regs->edxQuoted 4 lines
amd
Quoted 3 lines
Quoted 3 lines
regs->eaxfrom tmptsc, are we almost done?3) Des Linux or Windows use the CR4.TSD flag and, if
so, what code gets
Quoted 4 lines
Quoted 4 lines
called
Quoted 3 lines
to force the trap?
4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is
there already code
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
somewhere to emulate WRMSR of tsc?
Sorry I am not very familar with the details of the
ia32 instruction
Quoted 21 lines
set.
Quoted 20 lines
Quoted 20 lines
Thanks,
DanQuoted 16 lines
-----Original Message-----
From: Zhang, Li [mailto:[email protected]]
Sent: Monday, July 07, 2008 9:00 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)Hi, Dan
This seems not to be enough. I will cook another patch.Quoted 4 lines
-----Original Message-----
From: Zhang, Li
Sent: Tuesday, July 08, 2008 10:28 AM
To: Tian, Kevin; '[email protected]';'Xen-Devel
Quoted 2 lines
(E-mail)'
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Subject: RE: [Xen-devel] Guest TSC and Xen
(Intel and AMD
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
feedback please)
Quoted 4 lines
Hi, Dan
I just add some code to produce vmexit. The function
vmx_rdtsc_intercept()Quoted 1 line
is not completed. It is needed to add some code
to get the
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
guest tsc.
Quoted 10 lines
Thanks
--LiQuoted 6 lines
-----Original Message-----
From: Tian, Kevin
Sent: Tuesday, July 08, 2008 9:39 AM
To: '[email protected]'; Xen-Devel (E-mail)
Cc: Zhang, Li
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel andAMD feedback please)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
From: Dan Magenheimer
[mailto:[email protected]]
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Sent: 2008年7月5日 1:32
Quoted 1 line
Quoted 1 line
Is this something that you (or Intel in general)
could look at?
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
I would be happy to participate but I
don't think I
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
understand
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
VT well enough. Once the trap occurs, I suppose
Xen system time
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
could be used as the virtual TSC,
possibly scaled up.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
There should be tiny related to VT, as
only turning
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
on some bit to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
allow RDTSC trapping and then the rest
stuff should be common
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
how to handle it. We'll take a look, but can't
commit the time due
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to other scheduled bandwidth. But if you'd like to
jump in early
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
we definitely can help with VT side.
If you can post a patch with code that:
1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtscinstruction in any hvm
Quoted 19 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
domain causes a trap
3) the trap handler just does a physicalrdtsc and returns
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
then I could probably take it from there.
OK, and Li in CC will take a look and bake a
patch for you.
Quoted 9 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Thanks,
Kevin_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
PATCH 1/1 softtsc (was RE: Guest TSC and Xen (Intel and AMD feedback please))
The attached patch adds a new Xen boot option "softtsc" that causes all hvm domains to trap and emulate all rdtsc instructions and fetch Xen system time instead of the hardware tsc. As a result, the processor will appear to be running at 1000.xyz MHz (where xyz indicates some fraction that seems to vary slightly across different boots). I've also added a line of output to "xm debug-key t" to report softsc and tsc-related CPU features.
(Keir, though 3.3 functionality freeze is past, this is more of a diagnostic aid and defaults off, so I don't know if you would consider it for 3.3.)
Dan
P.S. Thanks to Li Zhang for coding the foundation for this patch. Li, do you want to provide a signed-off-by line too?
Quoted 182 lines
-----Original Message-----
From: Dan Magenheimer [mailto:[email protected]]
Sent: Tuesday, July 08, 2008 6:30 PM
To: 'Zhang, Li'; 'Xen-Devel (E-mail)'
Cc: 'Tian, Kevin'
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)Looks great! I am continuing to code and test. Some preliminary results:
1) Substituting
hvm_get_guest_time()forhvm_get_guest_tsc()seems to just work! The hvm
guest believes it is running on a 1000.185 MHz clock.2) On my test box, a guest rdtsc without softtsc takes about 30ns. With softtsc,
rdtsc takes about 1us.3) On my user-land test program, I am seeing rare "Time goes backwards", even with
hvm_get_guest_time(). I'm trying to track this down.Dan
Quoted 115 lines
-----Original Message-----
From: Zhang, Li [mailto:[email protected]]
Sent: Tuesday, July 08, 2008 3:47 AM
To: Zhang, Li; [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)Hi, Dan
Please ignore previous comments. The above attachment is OK. Sorry for that. :(Quoted 3 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On BehalfOf Zhang, Li
Quoted 4 lines
Sent: Tuesday, July 08, 2008 2:58 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMDfeedback please)
Quoted 2 lines
In fact, the issue is from guest. In the guest's terminal,
it shows that "your
Quoted 5 lines
time source seems to be instable..."
Quoted 3 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] OnBehalf Of Zhang, Li
Quoted 4 lines
Quoted 4 lines
Sent: Tuesday, July 08, 2008 2:49 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel andAMD feedback please)
Quoted 4 lines
Quoted 4 lines
Hi, Dan
This is the patch which I think it has been completed
VT. But there is an
Quoted 1 line
Quoted 1 line
instable issue. The reason may be from that this
interception is not bound
Quoted 11 lines
Quoted 11 lines
to timer mode yet.
Thanks
--LiQuoted 6 lines
-----Original Message-----
From: Dan Magenheimer [mailto:[email protected]]
Sent: Tuesday, July 08, 2008 11:34 AM
To: Zhang, Li; Xen-Devel (E-mail)
Cc: Tian, Kevin; [email protected]
Subject: RE: [Xen-devel] Guest TSC and Xen (Inteland AMD feedback
Quoted 7 lines
please)
Quoted 6 lines
Quoted 6 lines
Thanks Kevin and Li --
A couple of questions:
1) If the EXITING flag to be set in
vmcs.cis to becontrolled by a xen
Quoted 10 lines
Quoted 10 lines
boot
Quoted 9 lines
option, would this work:
static int
opt_softtsc= 0;boolean_param("softtsc", opt_softtsc); :min = /* original code */
if (opt_softtsc) min |=CPU_BASED_RDTSC_EXITING;2) In
vmx_rdtsc_intercept(/* need regs as param*/)if we change the
Quoted 1 line
Quoted 1 line
Quoted 1 line
rdtscll(tsc) to be tmptsc =
hvm_get_guest_tsc()andthen set
regs->edxQuoted 4 lines
amd
Quoted 3 lines
Quoted 3 lines
regs->eaxfrom tmptsc, are we almost done?3) Des Linux or Windows use the CR4.TSD flag and, if
so, what code gets
Quoted 4 lines
Quoted 4 lines
called
Quoted 3 lines
to force the trap?
4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is
there already code
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
somewhere to emulate WRMSR of tsc?
Sorry I am not very familar with the details of the
ia32 instruction
Quoted 21 lines
set.
Quoted 20 lines
Quoted 20 lines
Thanks,
DanQuoted 16 lines
-----Original Message-----
From: Zhang, Li [mailto:[email protected]]
Sent: Monday, July 07, 2008 9:00 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)Hi, Dan
This seems not to be enough. I will cook another patch.Quoted 4 lines
-----Original Message-----
From: Zhang, Li
Sent: Tuesday, July 08, 2008 10:28 AM
To: Tian, Kevin; '[email protected]';'Xen-Devel
Quoted 2 lines
(E-mail)'
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Subject: RE: [Xen-devel] Guest TSC and Xen
(Intel and AMD
Quoted 17 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
feedback please)
Quoted 4 lines
Hi, Dan
I just add some code to produce vmexit. The function
vmx_rdtsc_intercept()Quoted 1 line
is not completed. It is needed to add some code
to get the
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
guest tsc.
Quoted 8 lines
Thanks
--LiQuoted 4 lines
-----Original Message-----
From: Tian, Kevin
Sent: Tuesday, July 08, 2008 9:39 AM
To: '[email protected]';Xen-Devel (E-mail)
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Cc: Zhang, Li
Subject: RE: [Xen-devel] Guest TSC and Xen(Intel and
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
AMD feedback please)
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
From: Dan Magenheimer
[mailto:[email protected]]
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Sent: 2008年7月5日 1:32
Quoted 1 line
Quoted 1 line
Is this something that you (or Intel
in general)
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
could look at?
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
I would be happy to participate but I
don't think I
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
understand
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
VT well enough. Once the trap occurs,
I suppose
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Xen system time
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
could be used as the virtual TSC,
possibly scaled up.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
There should be tiny related to VT, as
Quoted 35 lines
Quoted 7 lines
only turning
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
on some bit to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
allow RDTSC trapping and then the rest
stuff should be common
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
how to handle it. We'll take a look, but can't
commit the time due
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to other scheduled bandwidth. But if
you'd like to
Quoted 27 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
jump in early
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
we definitely can help with VT side.
If you can post a patch with code that:
1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtscinstruction in any hvm
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
domain causes a trap
3) the trap handler just does a physicalrdtsc and returns
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
then I could probably take it from there.
OK, and Li in CC will take a look and bake a
patch for you.
Quoted 9 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Thanks,
Kevin_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
xen/arch/x86/cpu/amd.c +3 -1
@@ -327,8 +327,10 @@ static void __init init_amd(struct cpuinif (cpuid_eax (0x80000000 ) >=0x80000007 ) { c->x86_power =cpuid_edx (0x80000007 ); -if (c->x86_power & (1 <<8 )) +if (c->x86_power & (1 <<8 )) {set_bit (X86_FEATURE_CONSTANT_TSC , c->x86_capability); + constant_tsc =1 ; + } }#ifdef CONFIG_X86_HT
xen/arch/x86/cpu/common.c +2 -0
@@ -16,6 +16,8 @@#define tsc_disable 0#define disable_pse 0 + +int constant_tsc =0 ;static int cachesize_override__cpuinitdata =-1 ;static int disable_x86_fxsr__cpuinitdata ;
xen/arch/x86/cpu/intel.c +3 -1
@@ -184,8 +184,10 @@ static void __devinit init_intel(structif (c->x86 ==6 )set_bit (X86_FEATURE_P3 , c->x86_capability);if ((c->x86 ==0xf && c->x86_model >=0x03 ) || - (c->x86 ==0x6 && c->x86_model >=0x0e )) + (c->x86 ==0x6 && c->x86_model >=0x0e )) {set_bit (X86_FEATURE_CONSTANT_TSC , c->x86_capability); + constant_tsc =1 ; + }start_vmx (); }
xen/arch/x86/hvm/hvm.c +16 -1
@@ -56,6 +56,8 @@ int hvm_enabled __read_mostly;unsigned int opt_hvm_debug_level__read_mostly ;integer_param ("hvm_debug" , opt_hvm_debug_level); +int opt_softtsc =0 ; +boolean_param ("softtsc" , opt_softtsc);struct hvm_function_table hvm_funcs__read_mostly ;
@@ -148,7 +150,10 @@ u64 hvm_get_guest_tsc(struct vcpu *v) {u64 host_tsc; -rdtscll (host_tsc); +if ( opt_softtsc ) + host_tsc =hvm_get_guest_time (v); +else +rdtscll (host_tsc);return host_tsc + v->arch.hvm_vcpu.cache_tsc_offset; }
@@ -1640,6 +1645,16 @@ void hvm_cpuid(unsigned int input, unsig } } +void hvm_rdtsc_intercept (struct cpu_user_regs *regs) +{ +uint64_t tsc; +struct vcpu *v = current; + + tsc =hvm_get_guest_tsc (v); + regs->eax = (uint32_t )tsc; + regs->edx = (uint32_t )(tsc >>32 ); +} +int hvm_msr_read_intercept (struct cpu_user_regs *regs) {uint32_t ecx = regs->ecx;
xen/arch/x86/hvm/svm/svm.c +9 -0
@@ -52,6 +52,8 @@#include <asm/hap.h> u32 svm_feature_flags; + +extern int opt_softtsc;#define set_segment_register (name, value) \ asm volatile ( "movw %%ax ,%%" STR(name) "" : : "a" (value) )
@@ -1340,6 +1342,13 @@ asmlinkage void svm_vmexit_handler(struchvm_triple_fault ();break ; +case VMEXIT_RDTSC : +if ( opt_softtsc ) + { +hvm_rdtsc_intercept (regs); +break ; + } +/* else fall through */ case VMEXIT_RDTSCP :case VMEXIT_MONITOR :case VMEXIT_MWAIT :
xen/arch/x86/hvm/svm/vmcb.c +5 -0
@@ -39,6 +39,8 @@#include <xen/keyhandler.h> extern int svm_dbg_on; + +extern int opt_softtsc;#define IOPM_SIZE (12 * 1024)#define MSRPM_SIZE (8 * 1024)
@@ -165,6 +167,9 @@ static int construct_vmcb(struct vcpu *v/* TSC. */ vmcb->tsc_offset =0 ; +if ( opt_softtsc ) + vmcb->general1_intercepts |=GENERAL1_INTERCEPT_RDTSC ; +/* Guest EFER: *must* contain SVME or VMRUN will fail. */ vmcb->efer =EFER_SVME ;
xen/arch/x86/hvm/vmx/vmcs.c +3 -0
@@ -40,6 +40,7 @@static int opt_vpid_enabled =1 ;boolean_param ("vpid" , opt_vpid_enabled); +extern int opt_softtsc;/* Dynamic (run-time adjusted) execution control flags. */ u32 vmx_pin_based_exec_control__read_mostly ;
@@ -96,6 +97,8 @@ static void vmx_init_vmcs_config(void)CPU_BASED_MOV_DR_EXITING |CPU_BASED_ACTIVATE_IO_BITMAP | CPU_BASED_USE_TSC_OFFSETING); +if (opt_softtsc) + min |=CPU_BASED_RDTSC_EXITING ; opt = (CPU_BASED_ACTIVATE_MSR_BITMAP |CPU_BASED_TPR_SHADOW |CPU_BASED_ACTIVATE_SECONDARY_CONTROLS );
xen/arch/x86/hvm/vmx/vmx.c +5 -0
@@ -2202,6 +2202,11 @@ asmlinkage void vmx_vmexit_handler(strucvmx_invlpg_intercept (exit_qualification);break ; } +case EXIT_REASON_RDTSC : + inst_len =__get_instruction_length (); +__update_guest_eip (inst_len); +hvm_rdtsc_intercept (regs); +break ;case EXIT_REASON_VMCALL : {int rc;
xen/arch/x86/time.c +1 -1
@@ -68,7 +68,7 @@ static DEFINE_PER_CPU(struct cpu_time, c static DEFINE_PER_CPU(struct cpu_time, cpu_time);/* TSC is invariant on C state entry? */ -static bool_t tsc_invariant; +bool_t tsc_invariant;/* * We simulate a 32-bit platform timer from the 16-bit PIT ch2 counter.
xen/common/keyhandler.c +4 -0
@@ -251,6 +251,8 @@ static void read_clocks(unsigned char keunsigned int cpu =smp_processor_id (), min_cpu, max_cpu;u64 min, max, dif, difus;static DEFINE_SPINLOCK (lock); +extern int opt_softtsc, constant_tsc; +extern bool_t tsc_invariant;spin_lock (&lock);
@@ -284,6 +286,8 @@ static void read_clocks(unsigned char keprintk ("Min = %" PRIu64" ; Max = %" PRIu64" ; Diff = %" PRIu64" (%" PRIu64" microseconds)\n" , min, max, dif, difus); +printk ("softtsc = %d, constant_tsc = %d, tsc_invariant = %d\n" , + opt_softtsc, constant_tsc, (int )tsc_invariant); }extern void dump_runq (unsigned char key);
xen/include/asm-x86/cpufeature.h +2 -0
@@ -74,6 +74,8 @@#define X86_FEATURE_P3 (3*32+ 6)/* P3 */ #define X86_FEATURE_P4 (3*32+ 7)/* P4 */ #define X86_FEATURE_CONSTANT_TSC (3*32+ 8)/* TSC ticks at a constant rate */ + +extern int constant_tsc;/* Intel-defined CPU features, CPUID level 0x00000001 (ecx), word 4 */ #define X86_FEATURE_XMM3 (4*32+ 0)/* Streaming SIMD Extensions-3 */
xen/include/asm-x86/hvm/support.h +2 -0
@@ -125,6 +125,8 @@ void hvm_hlt(unsigned long rflags);void hvm_hlt (unsigned long rflags);void hvm_triple_fault (void ); +void hvm_rdtsc_intercept (struct cpu_user_regs *regs); +/* These functions all return X86EMUL return codes. */ int hvm_set_efer (uint64_t value);int hvm_set_cr0 (unsigned long value);
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
RE: [PATCH] softtsc (was RE: Guest TSC and Xen (Intel and AMD feedback please))
OK.
Thanks
Quoted 244 lines
-----Original Message-----
From: Dan Magenheimer [mailto:[email protected]]
Sent: Thursday, July 10, 2008 5:33 AM
To: [email protected]; Zhang, Li; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: [PATCH] softtsc (was RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please))The attached patch adds a new Xen boot option "softtsc" that causes all hvm domains to trap and emulate all rdtsc instructions and fetch Xen system time instead of the hardware tsc. As a result, the processor will appear to be running at 1000.xyz MHz (where xyz indicates some fraction that seems to vary slightly across different boots). I've also added a line of output to "xm debug-key t" to report softsc and tsc-related CPU features.
(Keir, though 3.3 functionality freeze is past, this is more of a diagnostic aid and defaults off, so I don't know if you would consider it for 3.3.)
Signed-off-by Dan Magenheimer <[email protected]>Dan
P.S. Thanks to Li Zhang for coding the foundation for this patch. Li, do you want to provide a signed-off-by line too?Quoted 217 lines
-----Original Message-----
From: Dan Magenheimer [mailto:[email protected]]
Sent: Tuesday, July 08, 2008 6:30 PM
To: 'Zhang, Li'; 'Xen-Devel (E-mail)'
Cc: 'Tian, Kevin'
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)Looks great! I am continuing to code and test. Some preliminary results:
1) Substituting
hvm_get_guest_time()forhvm_get_guest_tsc()seems to just work! The hvm
guest believes it is running on a 1000.185 MHz clock.2) On my test box, a guest rdtsc without softtsc takes about 30ns. With softtsc,
rdtsc takes about 1us.3) On my user-land test program, I am seeing rare "Time goes backwards", even with
hvm_get_guest_time(). I'm trying to track this down.Dan
Quoted 115 lines
-----Original Message-----
From: Zhang, Li [mailto:[email protected]]
Sent: Tuesday, July 08, 2008 3:47 AM
To: Zhang, Li; [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)Hi, Dan
Please ignore previous comments. The above attachment is OK. Sorry for that. :(Quoted 3 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On BehalfOf Zhang, Li
Quoted 4 lines
Sent: Tuesday, July 08, 2008 2:58 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMDfeedback please)
Quoted 2 lines
In fact, the issue is from guest. In the guest's terminal,
it shows that "your
Quoted 5 lines
time source seems to be instable..."
Quoted 3 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] OnBehalf Of Zhang, Li
Quoted 4 lines
Quoted 4 lines
Sent: Tuesday, July 08, 2008 2:49 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel andAMD feedback please)
Quoted 4 lines
Quoted 4 lines
Hi, Dan
This is the patch which I think it has been completed
VT. But there is an
Quoted 1 line
Quoted 1 line
instable issue. The reason may be from that this
interception is not bound
Quoted 11 lines
Quoted 11 lines
to timer mode yet.
Thanks
--LiQuoted 6 lines
-----Original Message-----
From: Dan Magenheimer [mailto:[email protected]]
Sent: Tuesday, July 08, 2008 11:34 AM
To: Zhang, Li; Xen-Devel (E-mail)
Cc: Tian, Kevin; [email protected]
Subject: RE: [Xen-devel] Guest TSC and Xen (Inteland AMD feedback
Quoted 7 lines
please)
Quoted 6 lines
Quoted 6 lines
Thanks Kevin and Li --
A couple of questions:
1) If the EXITING flag to be set in
vmcs.cis to becontrolled by a xen
Quoted 10 lines
Quoted 10 lines
boot
Quoted 9 lines
option, would this work:
static int
opt_softtsc= 0;boolean_param("softtsc", opt_softtsc); :min = /* original code */
if (opt_softtsc) min |=CPU_BASED_RDTSC_EXITING;2) In
vmx_rdtsc_intercept(/* need regs as param*/)if we change the
Quoted 1 line
Quoted 1 line
Quoted 1 line
rdtscll(tsc) to be tmptsc =
hvm_get_guest_tsc()andthen set
regs->edxQuoted 4 lines
amd
Quoted 3 lines
Quoted 3 lines
regs->eaxfrom tmptsc, are we almost done?3) Des Linux or Windows use the CR4.TSD flag and, if
so, what code gets
Quoted 4 lines
Quoted 4 lines
called
Quoted 3 lines
to force the trap?
4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is
there already code
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
somewhere to emulate WRMSR of tsc?
Sorry I am not very familar with the details of the
ia32 instruction
Quoted 21 lines
set.
Quoted 20 lines
Quoted 20 lines
Thanks,
DanQuoted 16 lines
-----Original Message-----
From: Zhang, Li [mailto:[email protected]]
Sent: Monday, July 07, 2008 9:00 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)Hi, Dan
This seems not to be enough. I will cook another patch.Quoted 4 lines
-----Original Message-----
From: Zhang, Li
Sent: Tuesday, July 08, 2008 10:28 AM
To: Tian, Kevin; '[email protected]';'Xen-Devel
Quoted 2 lines
(E-mail)'
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Subject: RE: [Xen-devel] Guest TSC and Xen
(Intel and AMD
Quoted 17 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
feedback please)
Quoted 4 lines
Hi, Dan
I just add some code to produce vmexit. The function
vmx_rdtsc_intercept()Quoted 1 line
is not completed. It is needed to add some code
to get the
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
guest tsc.
Quoted 8 lines
Thanks
--LiQuoted 4 lines
-----Original Message-----
From: Tian, Kevin
Sent: Tuesday, July 08, 2008 9:39 AM
To: '[email protected]';Xen-Devel (E-mail)
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Cc: Zhang, Li
Subject: RE: [Xen-devel] Guest TSC and Xen(Intel and
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
AMD feedback please)
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
From: Dan Magenheimer
[mailto:[email protected]]
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Sent: 2008年7月5日 1:32
Quoted 1 line
Quoted 1 line
Is this something that you (or Intel
in general)
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
could look at?
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
I would be happy to participate but I
don't think I
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
understand
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
VT well enough. Once the trap occurs,
I suppose
Quoted 13 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Xen system time
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
could be used as the virtual TSC,
possibly scaled up.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
There should be tiny related to VT, as
only turning
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
on some bit to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
allow RDTSC trapping and then the rest
stuff should be common
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
how to handle it. We'll take a look, but can't
commit the time due
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to other scheduled bandwidth. But if
you'd like to
Quoted 27 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
jump in early
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
we definitely can help with VT side.
If you can post a patch with code that:
1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtscinstruction in any hvm
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
domain causes a trap
3) the trap handler just does a physicalrdtsc and returns
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
then I could probably take it from there.
OK, and Li in CC will take a look and bake a
patch for you.
Quoted 9 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Thanks,
Kevin_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Re: [PATCH] softtsc (was RE: Guest TSC and Xen (Intel and AMD feedback please))
In reply to Dan MagenheimerQuoted 11 lines · Dan Magenheimer
The attached patch adds a new Xen boot option "softtsc" that causes all hvm domains to trap and emulate all rdtsc instructions and fetch Xen system time instead of the hardware tsc. As a result, the processor will appear to be running at 1000.xyz MHz (where xyz indicates some fraction that seems to vary slightly across different boots). I've also added a line of output to "xm debug-key t" to report softsc and tsc-related CPU features.
(Keir, though 3.3 functionality freeze is past, this is more of a diagnostic aid and defaults off, so I don't know if you would consider it for 3.3.)
Yes, I will take it, but have the following comments.
1. Why do you define new boolean 'constant_tsc'? Can you just usetest_bit(X86_FEATURE_CONSTANT_TSC)?
2. Your change in common/keyhandler.c breaks ia64. :-)
3. Your change to arch/x86/time.c looks unnecessary.
4. Should you catch SVM's RDTSCP vmexit as well as RDTSC?
-- Keir
RE: [PATCH] softtsc (was RE: Guest TSC and Xen (Intel and AMD feedback please))
Quoted 2 lines
Yes, I will take it, but have the following comments.
2. Your change incommon/keyhandler.cbreaks ia64. :-)
Oops! ;-) What's the best way to handle this? It would be unfortunate to lose valuable debug data just because its arch-dependent but I don't see any other arch-dependent code in keyhandler.c and I'll bet you don't want to start adding ifdef's nor introduce arch/xxx/keyhandler.c just for this.
Quoted 3 lines
1. Why do you define new boolean '
constant_tsc'? Can you just usetest_bit(X86_FEATURE_CONSTANT_TSC)?
3. Your change toarch/x86/time.clooks unnecessary.
I was thinking that the tests for these features should probably be abstracted (e.g. static inline in a header file or a global function), but wasn't sure about the best way to deal with the datatypes (e.g. struct cpuinfo_x86) so defaulted to global variables.
Both globals are simply for debug output in keyhandler.c so depending on the answer to (2) above, those patch-parts could just go away.
Quoted 1 line
4. Should you catch SVM's RDTSCP vmexit as well as RDTSC?
I thought I remembered seeing code that reported/lied to guests that the rdtscp feature was not present?
Thanks,
Dan
Re: [PATCH] softtsc (was RE: Guest TSC and Xen (Intel and AMD feedback please))
Hmm.. okay I think I will throw away the debug-output pieces of the patch. They are logically separate and distinct anyway.
-- Keir
Quoted 31 lines · Dan Magenheimer
Quoted 2 lines
Yes, I will take it, but have the following comments.
2. Your change incommon/keyhandler.cbreaks ia64. :-)Oops! ;-) What's the best way to handle this? It would be unfortunate to lose valuable debug data just because its arch-dependent but I don't see any other arch-dependent code in
keyhandler.cand I'll bet you don't want to start adding ifdef's nor introducearch/xxx/keyhandler.cjust for this.Quoted 3 lines
1. Why do you define new boolean '
constant_tsc'? Can you just usetest_bit(X86_FEATURE_CONSTANT_TSC)?
3. Your change toarch/x86/time.clooks unnecessary.I was thinking that the tests for these features should probably be abstracted (e.g. static inline in a header file or a global function), but wasn't sure about the best way to deal with the datatypes (e.g. struct
cpuinfo_x86) so defaulted to global variables.Both globals are simply for debug output in
keyhandler.cso depending on the answer to (2) above, those patch-parts could just go away.Quoted 1 line
4. Should you catch SVM's RDTSCP vmexit as well as RDTSC?
I thought I remembered seeing code that reported/lied to guests that the rdtscp feature was not present?
Thanks,
Dan