Skip to contentSkip to conversation

Guest TSC and Xen (Intel and AMD feedback please)

PATCH In review Dan Magenheimer 5 participants 22 messages Jul 1, 2008 – Jul 10, 2008

lore
Share & tools
lore
#0 lore raw

Various versions of Linux under various circumstances select TSC as the primary clocksource for the kernel. This is especially true for uniprocessor kernels, but also in some cases for multiprocessor kernels. In most cases, this is because a processor bit (tsc_invariant? constant_tsc?) is passed through directly from the hardware via Xen and tested by the hvm guest and the result implies that the TSC is "stable".

I'd like to propose that, for a Xen hvm guest, TSC should NEVER be considered stable. For at least these reasons:

1) Often this test is done once at guest boot; if the guest migrates to another machine without the bit set, time will be erratic.
2) I *think* that in some cases even within the same system, TSC values will skew somewhat. Since a uniprocessor guest will often be rescheduled to a different pcpu by Xen, the underlying tsc may appear erratic.

Comments (especially from Intel and AMD)?

If agreed, could Intel and AMD provide patches so that hvm reads of the bits return "false"? Or will this cause other problems?

Another alternative would be to trap all rdtsc's and emulate them but this probably will not be easy and may have significant performance implications. But perhaps it should be an option?

Dan

===================================
Thanks... for the memory
I really could use more / My throughput's on the floor The balloon is flat / My swap disk's fat / I've OOM's in store Overcommitted so much
(with apologies to the late great Bob Hope)

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#1 lore raw
Quoted 4 lines · Dan Magenheimer

Another alternative would be to trap all rdtsc's and emulate them but this probably will not be easy and may have significant performance implications. But perhaps it should be an option?

I think this would be useful, even if just for diagnosing guest time issues.

-- Keir

#2 lore raw
In reply to Dan Magenheimer
Quoted 11 lines

From: Dan Magenheimer
Sent: 2008年7月2日 1:33

Various versions of Linux under various circumstances select TSC as the primary clocksource for the kernel. This is especially true for uniprocessor kernels, but also in some cases for multiprocessor kernels. In most cases, this is because a processor bit (tsc_invariant? constant_tsc?) is passed through directly from the hardware via Xen and tested by the hvm guest and the result implies that the TSC is "stable".

constant_tsc is one of the factors affecting tsc stability, and the keypoint is that one internal variable 'tsc_unstable' is initialized to zero. That means, kernel considers TSC as stable by default, and you need trigger some factors upon which guest kernel'd like to mark tsc as unstable when checking them. However by far none of those factors seem appliable to all circumstances.

For example, even by hidding constant_tsc bit from cpuid, UP guest will still consider tsc as stable, and so does 32bit SMP guest running on Intel processors.

There's one proposal in KVM side, by exposing a faked C2 states into guest. ACPI processor idle path will mark TSC as unstable when trying to enter C2. However that requires guest to enable ACPI processor support, and need some tricky logic to further mark C2 invalid to avoid performance influence after purpose is achieved...

When saying this, I'm not opposing your proposal since it does be a light option for some type of guests. My point is just that no reliabe option to mark tsc as unstable, and whether we have done all what can be done in Xen side to tackle this issue... Sorry that I didn't follow up your long discussion in this area before and thus may lose some context.

Quoted 7 lines

I'd like to propose that, for a Xen hvm guest, TSC should NEVER be considered stable. For at least these reasons:

1) Often this test is done once at guest boot; if the guest migrates to another machine without the bit set, time will be erratic.

Since it's 'if', we should treat it as an option, instead of 'NEVER', right? User may not have migration requirement in some cases, and or the migratable backups are with same bits exactly... :-)

Quoted 4 lines

2) I *think* that in some cases even within the same system, TSC values will skew somewhat. Since a uniprocessor guest will often be rescheduled to a different pcpu by Xen, the underlying tsc may appear erratic.

Can't be the tsc kept monotonic to guest, with some tweak on tsc offset? Also Xen is always trying to sync tsc among cores. As long as you don't run on a box with multiple bus crystals, or a box without constant tsc upon freq change, the tsc drift among cores should be negligible considering the overhead of migration. Then for cases out of 'as long as', we should mark TSC as unstable, still as an option.

Quoted 6 lines

Comments (especially from Intel and AMD)?

If agreed, could Intel and AMD provide patches so that hvm reads of the bits return "false"? Or will this cause other problems?

I vaguely recalled some posts about CPUID virtualization framework. There's no need to a seperate category, and it looks like this option can be aligned with that part?

Thanks,
Kevin

Quoted 6 lines

Another alternative would be to trap all rdtsc's and emulate them but this probably will not be easy and may have significant performance implications. But perhaps it should be an option?

Agree.

Thanks,
Kevin

#3 lore raw

Hi Kevin --

Quoted 11 lines

constant_tsc is one of the factors affecting tsc stability, and the keypoint is that one internal variable 'tsc_unstable' is initialized to zero. That means, kernel considers TSC as stable by default, and you need trigger some factors upon which guest kernel'd like to mark tsc as unstable when checking them. However by far none of those factors seem appliable to all circumstances.

For example, even by hidding constant_tsc bit from cpuid, UP guest will still consider tsc as stable, and so does 32bit SMP guest running on Intel processors.

Yes, looking over some Linux code, I see you are right. In one version of RH, if the processor vendor is Intel, it always assumes tsc is stable!

Quoted 3 lines

Since it's 'if', we should treat it as an option, instead of 'NEVER', right? User may not have migration requirement in some cases, and or the migratable backups are with same bits exactly... :-)

Yes, if it could be done, I agree an option would be better than 'NEVER'. But Keir is not too keen on more time-related options.

Quoted 5 lines
Quoted 4 lines

2) I *think* that in some cases even within the same system, TSC values will skew somewhat. Since a uniprocessor guest will often be rescheduled to a different pcpu by Xen, the underlying tsc may appear erratic.

Quoted 7 lines

Can't be the tsc kept monotonic to guest, with some tweak on tsc offset? Also Xen is always trying to sync tsc among cores. As long as you don't run on a box with multiple bus crystals, or a box without constant tsc upon freq change, the tsc drift among cores should be negligible considering the overhead of migration. Then for cases out of 'as long as', we should mark TSC as unstable, still as an option.

No Xen doesn't actally sync the tsc's, just maintains its own software offset variables. I suppose it could periodically check to make sure the tsc skew is within some reasonable value, but after the guest boots, it is probably too late.

Quoted 3 lines

I vaguely recalled some posts about CPUID virtualization framework. There's no need to a seperate category, and it looks like this option can be aligned with that part?

Yes, good point. I wasn't following that closely either but this could be covered by it.

Quoted 6 lines
Quoted 4 lines

Another alternative would be to trap all rdtsc's and emulate them but this probably will not be easy and may have significant performance implications. But perhaps it should be an option?

Agree.

Is this something that you (or Intel in general) could look at? I would be happy to participate but I don't think I understand VT well enough. Once the trap occurs, I suppose Xen system time could be used as the virtual TSC, possibly scaled up.

Thanks,
Dan

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#4 lore raw
Quoted 11 lines

From: Dan Magenheimer [mailto:[email protected]]
Sent: 2008年7月3日 9:21

Quoted 8 lines

Can't be the tsc kept monotonic to guest, with some tweak on tsc offset? Also Xen is always trying to sync tsc among cores. As long as you don't run on a box with multiple bus crystals, or a box without constant tsc upon freq change, the tsc drift among cores should be negligible considering the overhead of migration. Then for cases out of 'as long as', we should mark TSC as unstable, still as an option.

Quoted 4 lines

No Xen doesn't actally sync the tsc's, just maintains its own software offset variables. I suppose it could periodically check to make sure the tsc skew is within some reasonable value, but after the guest boots, it is probably too late.

You are right. Xen only sync TSCs at boot time and then calibrate TSC pace independently on each cpu. If some factors drag TSC in the middle, then we may observe larger drift. But now I'm not sure whether periodically sync among cpus is light enough, if taking a similar convergence algothrim as boot time. But still, if only TSC drift exists (caused by occasional events instead of a constantly increasing drift from freq difference), this monotonic can be easily exposed to the guest, like derived from xen system time to adjust TSC offset at migration...

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

Another alternative would be to trap all rdtsc's and emulate them but this probably will not be easy and may have significant performance implications. But perhaps it should be an option?

Quoted 7 lines
Quoted 1 line

Agree.

Is this something that you (or Intel in general) could look at? I would be happy to participate but I don't think I understand VT well enough. Once the trap occurs, I suppose Xen system time could be used as the virtual TSC, possibly scaled up.

There should be tiny related to VT, as only turning on some bit to allow RDTSC trapping and then the rest stuff should be common how to handle it. We'll take a look, but can't commit the time due to other scheduled bandwidth. But if you'd like to jump in early we definitely can help with VT side.

Thanks,
Kevin

#5 lore raw
Quoted 11 lines
Quoted 5 lines

Is this something that you (or Intel in general) could look at? I would be happy to participate but I don't think I understand VT well enough. Once the trap occurs, I suppose Xen system time could be used as the virtual TSC, possibly scaled up.

There should be tiny related to VT, as only turning on some bit to allow RDTSC trapping and then the rest stuff should be common how to handle it. We'll take a look, but can't commit the time due to other scheduled bandwidth. But if you'd like to jump in early we definitely can help with VT side.

If you can post a patch with code that:

1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtsc instruction in any hvm domain causes a trap
3) the trap handler just does a physical rdtsc and returns

then I could probably take it from there.

Dan

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#6 lore raw
Quoted 15 lines

From: Dan Magenheimer [mailto:[email protected]]
Sent: 2008年7月5日 1:32

Quoted 11 lines
Quoted 5 lines

Is this something that you (or Intel in general) could look at? I would be happy to participate but I don't think I understand VT well enough. Once the trap occurs, I suppose Xen system time could be used as the virtual TSC, possibly scaled up.

There should be tiny related to VT, as only turning on some bit to allow RDTSC trapping and then the rest stuff should be common how to handle it. We'll take a look, but can't commit the time due to other scheduled bandwidth. But if you'd like to jump in early we definitely can help with VT side.

Quoted 8 lines

If you can post a patch with code that:

1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtsc instruction in any hvm domain causes a trap
3) the trap handler just does a physical rdtsc and returns

then I could probably take it from there.

OK, and Li in CC will take a look and bake a patch for you.

Thanks,
Kevin

#7 lore raw

Hi, Dan

I just add some code to produce vmexit. The function vmx_rdtsc_intercept() is not completed. It is needed to add some code to get the guest tsc.

Thanks
--Li

Quoted 29 lines

-----Original Message-----
From: Tian, Kevin
Sent: Tuesday, July 08, 2008 9:39 AM
To: '[email protected]'; Xen-Devel (E-mail)
Cc: Zhang, Li
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)

Quoted 22 lines

From: Dan Magenheimer [mailto:[email protected]]
Sent: 2008年7月5日 1:32

Quoted 11 lines
Quoted 5 lines

Is this something that you (or Intel in general) could look at? I would be happy to participate but I don't think I understand VT well enough. Once the trap occurs, I suppose Xen system time could be used as the virtual TSC, possibly scaled up.

There should be tiny related to VT, as only turning on some bit to allow RDTSC trapping and then the rest stuff should be common how to handle it. We'll take a look, but can't commit the time due to other scheduled bandwidth. But if you'd like to jump in early we definitely can help with VT side.

If you can post a patch with code that:

1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtsc instruction in any hvm domain causes a trap
3) the trap handler just does a physical rdtsc and returns

Quoted 6 lines
Quoted 1 line

then I could probably take it from there.

OK, and Li in CC will take a look and bake a patch for you.

Thanks,
Kevin

xen/arch/x86/hvm/vmx/vmcs.c +1 -0
diff -r 5eaec8296a22 xen/arch/x86/hvm/vmx/vmcs.c
@@ -89,6 +89,7 @@
 
     min = (CPU_BASED_HLT_EXITING |
            CPU_BASED_INVLPG_EXITING |
+           CPU_BASED_RDTSC_EXITING |
            CPU_BASED_CR3_LOAD_EXITING |
            CPU_BASED_CR3_STORE_EXITING |
            CPU_BASED_MONITOR_EXITING |
xen/arch/x86/hvm/vmx/vmx.c +12 -0
diff -r 5eaec8296a22 xen/arch/x86/hvm/vmx/vmx.c
@@ -1374,6 +1374,13 @@
     HVMTRACE_LONG_2D(INVLPG, curr, /*invlpga=*/ 0, TRC_PAR_LONG(vaddr));
     if ( paging_invlpg(curr, vaddr) )
         vpid_sync_vcpu_gva(curr, vaddr);
+}
+
+static void vmx_rdtsc_intercept(struct vcpu *vcpu)
+{
+    uint64_t tsc; 
+    rdtscll(tsc);
+    return;    
 }
 
 #define CASE_SET_REG(REG, reg)      \
@@ -2193,6 +2200,11 @@
         vmx_invlpg_intercept(exit_qualification);
         break;
     }
+    case EXIT_REASON_RDTSC:
+        inst_len = __get_instruction_length();
+        __update_guest_eip(inst_len);
+        vmx_rdtsc_intercept(v);
+        break;
     case EXIT_REASON_VMCALL:
     {
         int rc;

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#8 lore raw
In reply to Tian, Kevin

Hi, Dan
This seems not to be enough. I will cook another patch.

Quoted 49 lines

-----Original Message-----
From: Zhang, Li
Sent: Tuesday, July 08, 2008 10:28 AM
To: Tian, Kevin; '[email protected]'; 'Xen-Devel (E-mail)'
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)

Hi, Dan

I just add some code to produce vmexit. The function vmx_rdtsc_intercept() is not completed. It is needed to add some code to get the guest tsc.

Thanks
--Li

Quoted 35 lines

-----Original Message-----
From: Tian, Kevin
Sent: Tuesday, July 08, 2008 9:39 AM
To: '[email protected]'; Xen-Devel (E-mail)
Cc: Zhang, Li
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)

Quoted 15 lines

From: Dan Magenheimer [mailto:[email protected]]
Sent: 2008年7月5日 1:32

Quoted 11 lines
Quoted 5 lines

Is this something that you (or Intel in general) could look at? I would be happy to participate but I don't think I understand VT well enough. Once the trap occurs, I suppose Xen system time could be used as the virtual TSC, possibly scaled up.

There should be tiny related to VT, as only turning on some bit to allow RDTSC trapping and then the rest stuff should be common how to handle it. We'll take a look, but can't commit the time due to other scheduled bandwidth. But if you'd like to jump in early we definitely can help with VT side.

Quoted 8 lines

If you can post a patch with code that:

1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtsc instruction in any hvm domain causes a trap
3) the trap handler just does a physical rdtsc and returns

then I could probably take it from there.

OK, and Li in CC will take a look and bake a patch for you.

Thanks,
Kevin

#9 lore raw

Thanks Kevin and Li --

A couple of questions:

1) If the EXITING flag to be set in vmcs.c is to be controlled by a xen boot option, would this work:

static int opt_softtsc = 0;
boolean_param("softtsc", opt_softtsc);
:
min = /* original code */
if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;

2) In vmx_rdtsc_intercept(/* need regs as param*/) if we change the rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and then set regs->edx amd regs->eax from tmptsc, are we almost done?

3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets called to force the trap?

4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is there already code somewhere to emulate WRMSR of tsc?

Sorry I am not very familar with the details of the ia32 instruction set.

Thanks,
Dan

Quoted 49 lines

-----Original Message-----
From: Zhang, Li [mailto:[email protected]]
Sent: Monday, July 07, 2008 9:00 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)

Hi, Dan
This seems not to be enough. I will cook another patch.

Quoted 5 lines

-----Original Message-----
From: Zhang, Li
Sent: Tuesday, July 08, 2008 10:28 AM
To: Tian, Kevin; '[email protected]'; 'Xen-Devel (E-mail)'
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD

feedback please)

Quoted 4 lines

Hi, Dan

I just add some code to produce vmexit. The function

vmx_rdtsc_intercept()

Quoted 1 line

is not completed. It is needed to add some code to get the

guest tsc.

Quoted 10 lines

Thanks
--Li

Quoted 6 lines

-----Original Message-----
From: Tian, Kevin
Sent: Tuesday, July 08, 2008 9:39 AM
To: '[email protected]'; Xen-Devel (E-mail)
Cc: Zhang, Li
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and

AMD feedback please)

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

From: Dan Magenheimer [mailto:[email protected]]
Sent: 2008年7月5日 1:32

Quoted 1 line
Quoted 1 line

Is this something that you (or Intel in general)

could look at?

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I would be happy to participate but I don't think I

understand

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

VT well enough. Once the trap occurs, I suppose

Xen system time

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

could be used as the virtual TSC, possibly scaled up.

Quoted 23 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

There should be tiny related to VT, as only turning

on some bit to

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

allow RDTSC trapping and then the rest stuff should be common how to handle it. We'll take a look, but can't

commit the time due

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

to other scheduled bandwidth. But if you'd like to

jump in early

Quoted 15 lines
Quoted 15 lines
Quoted 10 lines
Quoted 1 line

we definitely can help with VT side.

If you can post a patch with code that:

1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtsc instruction in any hvm domain causes a trap
3) the trap handler just does a physical rdtsc and returns

then I could probably take it from there.

OK, and Li in CC will take a look and bake a patch for you.

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#10 lore raw
Quoted 11 lines

From: Dan Magenheimer [mailto:[email protected]]
Sent: 2008年7月8日 11:34

1) If the EXITING flag to be set in vmcs.c is to be controlled by a xen boot option, would this work:

static int opt_softtsc = 0;
boolean_param("softtsc", opt_softtsc);
:
min = /* original code */
if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;

yes.

Quoted 4 lines

2) In vmx_rdtsc_intercept(/* need regs as param*/) if we change the rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and then set regs->edx amd regs->eax from tmptsc, are we almost done?

Yes, and similar example is under case EXIT_REASON_CPUID.

Quoted 3 lines

3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets called to force the trap?

My feeling is that both Linux and Windows allows user directly read to TSC, and thus CR4.TSD is unlikely to be set. But it needs confirmation. If you really want to trap such setting, hvm_set_cr4 is the point and you may further reference vmx_update_guest_cr if it's vmx specific (but I guess it should be common).

But I doubt the point to support such model, since two levels trapping (Xen trap, inject fault, guest kernel read, xen trap, ...) which adds much delay on Xen maintained guest tsc.

Quoted 3 lines

4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is there already code somewhere to emulate WRMSR of tsc?

I'm not sure about this one, and others may answer.

Thanks,
Kevin

#11 lore raw
In reply to Dan Magenheimer

Hi, Dan

This is the patch which I think it has been completed VT. But there is an instable issue. The reason may be from that this interception is not bound to timer mode yet.

Thanks
--Li

Quoted 27 lines

-----Original Message-----
From: Dan Magenheimer [mailto:[email protected]]
Sent: Tuesday, July 08, 2008 11:34 AM
To: Zhang, Li; Xen-Devel (E-mail)
Cc: Tian, Kevin; [email protected]
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)

Thanks Kevin and Li --

A couple of questions:

1) If the EXITING flag to be set in vmcs.c is to be controlled by a xen boot option, would this work:

static int opt_softtsc = 0;

 boolean_param("softtsc", opt_softtsc);
   :

min = /* original code */
if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;

2) In vmx_rdtsc_intercept(/* need regs as param*/) if we change the rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and then set regs->edx amd regs->eax from tmptsc, are we almost done?

3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets called to force the trap?

Quoted 80 lines

4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is there already code somewhere to emulate WRMSR of tsc?

Sorry I am not very familar with the details of the ia32 instruction set.

Thanks,
Dan

Quoted 72 lines

-----Original Message-----
From: Zhang, Li [mailto:[email protected]]
Sent: Monday, July 07, 2008 9:00 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)

Hi, Dan
This seems not to be enough. I will cook another patch.

Quoted 5 lines

-----Original Message-----
From: Zhang, Li
Sent: Tuesday, July 08, 2008 10:28 AM
To: Tian, Kevin; '[email protected]'; 'Xen-Devel (E-mail)'
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD

feedback please)

Quoted 4 lines

Hi, Dan

I just add some code to produce vmexit. The function

vmx_rdtsc_intercept()

Quoted 1 line

is not completed. It is needed to add some code to get the

guest tsc.

Quoted 10 lines

Thanks
--Li

Quoted 6 lines

-----Original Message-----
From: Tian, Kevin
Sent: Tuesday, July 08, 2008 9:39 AM
To: '[email protected]'; Xen-Devel (E-mail)
Cc: Zhang, Li
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and

AMD feedback please)

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

From: Dan Magenheimer [mailto:[email protected]]
Sent: 2008年7月5日 1:32

Quoted 1 line
Quoted 1 line

Is this something that you (or Intel in general)

could look at?

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I would be happy to participate but I don't think I

understand

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

VT well enough. Once the trap occurs, I suppose

Xen system time

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

could be used as the virtual TSC, possibly scaled up.

There should be tiny related to VT, as only turning

on some bit to

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

allow RDTSC trapping and then the rest stuff should be common how to handle it. We'll take a look, but can't

commit the time due

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

to other scheduled bandwidth. But if you'd like to

jump in early

Quoted 15 lines
Quoted 15 lines
Quoted 10 lines
Quoted 1 line

we definitely can help with VT side.

If you can post a patch with code that:

1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtsc instruction in any hvm domain causes a trap
3) the trap handler just does a physical rdtsc and returns

then I could probably take it from there.

OK, and Li in CC will take a look and bake a patch for you.

Thanks,
Kevin

xen/arch/x86/hvm/vmx/vmcs.c +3 -0
diff -r be52424a543e xen/arch/x86/hvm/vmx/vmcs.c
@@ -40,6 +40,7 @@
 
 static int opt_vpid_enabled = 1;
 boolean_param("vpid", opt_vpid_enabled);
+int softtsc = 1;
 
 /* Dynamic (run-time adjusted) execution control flags. */
 u32 vmx_pin_based_exec_control __read_mostly;
@@ -96,6 +97,8 @@
            CPU_BASED_MOV_DR_EXITING |
            CPU_BASED_ACTIVATE_IO_BITMAP |
            CPU_BASED_USE_TSC_OFFSETING);
+    if (softtsc)
+        min |= CPU_BASED_RDTSC_EXITING;
     opt = (CPU_BASED_ACTIVATE_MSR_BITMAP |
            CPU_BASED_TPR_SHADOW |
            CPU_BASED_ACTIVATE_SECONDARY_CONTROLS);
xen/arch/x86/hvm/vmx/vmx.c +16 -0
diff -r be52424a543e xen/arch/x86/hvm/vmx/vmx.c
@@ -1375,6 +1375,17 @@
     if ( paging_invlpg(curr, vaddr) )
         vpid_sync_vcpu_gva(curr, vaddr);
 }
+
+static void vmx_rdtsc_intercept(struct cpu_user_regs *regs)
+{
+    uint64_t tsc;
+    struct vcpu *v = current; 
+
+    tsc = hvm_get_guest_tsc(v);
+    regs->eax = tsc & 0xffff;
+    regs->edx = tsc >> 32 & 0xffff;
+}
+
 
 #define CASE_SET_REG(REG, reg)      \
     case VMX_CONTROL_REG_ACCESS_GPR_ ## REG: regs->reg = value; break
@@ -2192,6 +2203,11 @@
         vmx_invlpg_intercept(exit_qualification);
         break;
     }
+    case EXIT_REASON_RDTSC:
+        inst_len = __get_instruction_length();
+        __update_guest_eip(inst_len);
+        vmx_rdtsc_intercept(regs);
+        break;
     case EXIT_REASON_VMCALL:
     {
         int rc;

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#12 lore raw

In fact, the issue is from guest. In the guest's terminal, it shows that "your time source seems to be instable..."

Quoted 126 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Zhang, Li
Sent: Tuesday, July 08, 2008 2:49 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)

Hi, Dan

This is the patch which I think it has been completed VT. But there is an instable issue. The reason may be from that this interception is not bound to timer mode yet.

Thanks
--Li

Quoted 7 lines

-----Original Message-----
From: Dan Magenheimer [mailto:[email protected]]
Sent: Tuesday, July 08, 2008 11:34 AM
To: Zhang, Li; Xen-Devel (E-mail)
Cc: Tian, Kevin; [email protected]
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)

Quoted 5 lines

Thanks Kevin and Li --

A couple of questions:

1) If the EXITING flag to be set in vmcs.c is to be controlled by a xen

boot

Quoted 5 lines

option, would this work:

static int opt_softtsc = 0;

 boolean_param("softtsc", opt_softtsc);
   :
Quoted 8 lines

min = /* original code */
if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;

2) In vmx_rdtsc_intercept(/* need regs as param*/) if we change the rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and then set regs->edx amd regs->eax from tmptsc, are we almost done?

3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets

called

Quoted 82 lines

to force the trap?

4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is there already code somewhere to emulate WRMSR of tsc?

Sorry I am not very familar with the details of the ia32 instruction set.

Thanks,
Dan

Quoted 72 lines

-----Original Message-----
From: Zhang, Li [mailto:[email protected]]
Sent: Monday, July 07, 2008 9:00 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)

Hi, Dan
This seems not to be enough. I will cook another patch.

Quoted 5 lines

-----Original Message-----
From: Zhang, Li
Sent: Tuesday, July 08, 2008 10:28 AM
To: Tian, Kevin; '[email protected]'; 'Xen-Devel (E-mail)'
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD

feedback please)

Quoted 4 lines

Hi, Dan

I just add some code to produce vmexit. The function

vmx_rdtsc_intercept()

Quoted 1 line

is not completed. It is needed to add some code to get the

guest tsc.

Quoted 10 lines

Thanks
--Li

Quoted 6 lines

-----Original Message-----
From: Tian, Kevin
Sent: Tuesday, July 08, 2008 9:39 AM
To: '[email protected]'; Xen-Devel (E-mail)
Cc: Zhang, Li
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and

AMD feedback please)

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

From: Dan Magenheimer [mailto:[email protected]]
Sent: 2008年7月5日 1:32

Quoted 1 line
Quoted 1 line

Is this something that you (or Intel in general)

could look at?

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I would be happy to participate but I don't think I

understand

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

VT well enough. Once the trap occurs, I suppose

Xen system time

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

could be used as the virtual TSC, possibly scaled up.

There should be tiny related to VT, as only turning

on some bit to

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

allow RDTSC trapping and then the rest stuff should be common how to handle it. We'll take a look, but can't

commit the time due

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

to other scheduled bandwidth. But if you'd like to

jump in early

Quoted 15 lines
Quoted 15 lines
Quoted 10 lines
Quoted 1 line

we definitely can help with VT side.

If you can post a patch with code that:

1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtsc instruction in any hvm domain causes a trap
3) the trap handler just does a physical rdtsc and returns

then I could probably take it from there.

OK, and Li in CC will take a look and bake a patch for you.

Thanks,
Kevin

#13 lore raw

Hi, Dan
Please ignore previous comments. The above attachment is OK. Sorry for that. :(

Quoted 138 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Zhang, Li
Sent: Tuesday, July 08, 2008 2:58 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)

In fact, the issue is from guest. In the guest's terminal, it shows that "your time source seems to be instable..."

Quoted 8 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Zhang, Li
Sent: Tuesday, July 08, 2008 2:49 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)

Quoted 15 lines

Hi, Dan

This is the patch which I think it has been completed VT. But there is an instable issue. The reason may be from that this interception is not bound to timer mode yet.

Thanks
--Li

Quoted 6 lines

-----Original Message-----
From: Dan Magenheimer [mailto:[email protected]]
Sent: Tuesday, July 08, 2008 11:34 AM
To: Zhang, Li; Xen-Devel (E-mail)
Cc: Tian, Kevin; [email protected]
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback

please)

Quoted 17 lines
Quoted 6 lines

Thanks Kevin and Li --

A couple of questions:

1) If the EXITING flag to be set in vmcs.c is to be controlled by a xen

boot

Quoted 10 lines

option, would this work:

static int opt_softtsc = 0;

 boolean_param("softtsc", opt_softtsc);
   :

min = /* original code */
if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;

2) In vmx_rdtsc_intercept(/* need regs as param*/) if we change the rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and then set regs->edx

amd

Quoted 10 lines
Quoted 3 lines

regs->eax from tmptsc, are we almost done?

3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets

called

Quoted 6 lines

to force the trap?

4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is there already code somewhere to emulate WRMSR of tsc?

Sorry I am not very familar with the details of the ia32 instruction

set.

Quoted 13 lines
Quoted 13 lines

Thanks,
Dan

Quoted 9 lines

-----Original Message-----
From: Zhang, Li [mailto:[email protected]]
Sent: Monday, July 07, 2008 9:00 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Hi, Dan
This seems not to be enough. I will cook another patch.

Quoted 4 lines

-----Original Message-----
From: Zhang, Li
Sent: Tuesday, July 08, 2008 10:28 AM
To: Tian, Kevin; '[email protected]'; 'Xen-Devel

(E-mail)'

Quoted 53 lines
Quoted 53 lines
Quoted 53 lines
Quoted 1 line

Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD

feedback please)

Quoted 4 lines

Hi, Dan

I just add some code to produce vmexit. The function

vmx_rdtsc_intercept()

Quoted 1 line

is not completed. It is needed to add some code to get the

guest tsc.

Quoted 10 lines

Thanks
--Li

Quoted 6 lines

-----Original Message-----
From: Tian, Kevin
Sent: Tuesday, July 08, 2008 9:39 AM
To: '[email protected]'; Xen-Devel (E-mail)
Cc: Zhang, Li
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and

AMD feedback please)

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

From: Dan Magenheimer [mailto:[email protected]]
Sent: 2008年7月5日 1:32

Quoted 1 line
Quoted 1 line

Is this something that you (or Intel in general)

could look at?

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I would be happy to participate but I don't think I

understand

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

VT well enough. Once the trap occurs, I suppose

Xen system time

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

could be used as the virtual TSC, possibly scaled up.

There should be tiny related to VT, as only turning

on some bit to

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

allow RDTSC trapping and then the rest stuff should be common how to handle it. We'll take a look, but can't

commit the time due

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

to other scheduled bandwidth. But if you'd like to

jump in early

Quoted 13 lines
Quoted 13 lines
Quoted 10 lines
Quoted 1 line

we definitely can help with VT side.

If you can post a patch with code that:

1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtsc instruction in any hvm domain causes a trap
3) the trap handler just does a physical rdtsc and returns

then I could probably take it from there.

OK, and Li in CC will take a look and bake a patch for you.

Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

xen/arch/x86/hvm/vmx/vmcs.c +3 -0
diff -r be52424a543e xen/arch/x86/hvm/vmx/vmcs.c
@@ -40,6 +40,7 @@
 
 static int opt_vpid_enabled = 1;
 boolean_param("vpid", opt_vpid_enabled);
+int softtsc = 1;
 
 /* Dynamic (run-time adjusted) execution control flags. */
 u32 vmx_pin_based_exec_control __read_mostly;
@@ -96,6 +97,8 @@
            CPU_BASED_MOV_DR_EXITING |
            CPU_BASED_ACTIVATE_IO_BITMAP |
            CPU_BASED_USE_TSC_OFFSETING);
+    if (softtsc)
+        min |= CPU_BASED_RDTSC_EXITING;
     opt = (CPU_BASED_ACTIVATE_MSR_BITMAP |
            CPU_BASED_TPR_SHADOW |
            CPU_BASED_ACTIVATE_SECONDARY_CONTROLS);
xen/arch/x86/hvm/vmx/vmx.c +16 -0
diff -r be52424a543e xen/arch/x86/hvm/vmx/vmx.c
@@ -1375,6 +1375,17 @@
     if ( paging_invlpg(curr, vaddr) )
         vpid_sync_vcpu_gva(curr, vaddr);
 }
+
+static void vmx_rdtsc_intercept(struct cpu_user_regs *regs)
+{
+    uint64_t tsc;
+    struct vcpu *v = current; 
+
+    tsc = hvm_get_guest_tsc(v);
+    regs->eax = tsc & 0xffffffff;
+    regs->edx = tsc >> 32 & 0xffffffff;
+}
+
 
 #define CASE_SET_REG(REG, reg)      \
     case VMX_CONTROL_REG_ACCESS_GPR_ ## REG: regs->reg = value; break
@@ -2192,6 +2203,11 @@
         vmx_invlpg_intercept(exit_qualification);
         break;
     }
+    case EXIT_REASON_RDTSC:
+        inst_len = __get_instruction_length();
+        __update_guest_eip(inst_len);
+        vmx_rdtsc_intercept(regs);
+        break;
     case EXIT_REASON_VMCALL:
     {
         int rc;

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#14 lore raw

Hi Li,

Shouldn't this

+ regs->eax = tsc & 0xffff;
+ regs->edx = tsc >> 32 & 0xffff;

be

+ regs->eax = tsc & 0xffffffff;
+ regs->edx = tsc >> 32 & 0xffffffff;

Thanks,
Dave

Quoted 161 lines · Li

Hi, Dan
Please ignore previous comments. The above attachment is OK. Sorry for that. :(

Quoted 156 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Zhang, Li
Sent: Tuesday, July 08, 2008 2:58 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)

In fact, the issue is from guest. In the guest's terminal, it shows that "your time source seems to be instable..."

Quoted 8 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Zhang, Li
Sent: Tuesday, July 08, 2008 2:49 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)

Quoted 15 lines

Hi, Dan

This is the patch which I think it has been completed VT. But there is an instable issue. The reason may be from that this interception is not bound to timer mode yet.

Thanks
--Li

Quoted 6 lines

-----Original Message-----
From: Dan Magenheimer [mailto:[email protected]]
Sent: Tuesday, July 08, 2008 11:34 AM
To: Zhang, Li; Xen-Devel (E-mail)
Cc: Tian, Kevin; [email protected]
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback

please)

Quoted 17 lines
Quoted 6 lines

Thanks Kevin and Li --

A couple of questions:

1) If the EXITING flag to be set in vmcs.c is to be controlled by a xen

boot

Quoted 10 lines

option, would this work:

static int opt_softtsc = 0;

 boolean_param("softtsc", opt_softtsc);
   :

min = /* original code */
if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;

2) In vmx_rdtsc_intercept(/* need regs as param*/) if we change the rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and then set regs->edx

amd

Quoted 10 lines
Quoted 3 lines

regs->eax from tmptsc, are we almost done?

3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets

called

Quoted 6 lines

to force the trap?

4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is there already code somewhere to emulate WRMSR of tsc?

Sorry I am not very familar with the details of the ia32 instruction

set.

Quoted 13 lines
Quoted 13 lines

Thanks,
Dan

Quoted 9 lines

-----Original Message-----
From: Zhang, Li [mailto:[email protected]]
Sent: Monday, July 07, 2008 9:00 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Hi, Dan
This seems not to be enough. I will cook another patch.

Quoted 4 lines

-----Original Message-----
From: Zhang, Li
Sent: Tuesday, July 08, 2008 10:28 AM
To: Tian, Kevin; '[email protected]'; 'Xen-Devel

(E-mail)'

Quoted 49 lines
Quoted 49 lines
Quoted 49 lines
Quoted 1 line

Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD

feedback please)

Quoted 4 lines

Hi, Dan

I just add some code to produce vmexit. The function

vmx_rdtsc_intercept()

Quoted 1 line

is not completed. It is needed to add some code to get the

guest tsc.

Quoted 10 lines

Thanks
--Li

Quoted 6 lines

-----Original Message-----
From: Tian, Kevin
Sent: Tuesday, July 08, 2008 9:39 AM
To: '[email protected]'; Xen-Devel (E-mail)
Cc: Zhang, Li
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and

AMD feedback please)

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

From: Dan Magenheimer [mailto:[email protected]]
Sent: 2008年7月5日 1:32

Quoted 1 line
Quoted 1 line

Is this something that you (or Intel in general)

could look at?

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I would be happy to participate but I don't think I

understand

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

VT well enough. Once the trap occurs, I suppose

Xen system time

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

could be used as the virtual TSC, possibly scaled up.

There should be tiny related to VT, as only turning

on some bit to

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

allow RDTSC trapping and then the rest stuff should be common how to handle it. We'll take a look, but can't

commit the time due

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

to other scheduled bandwidth. But if you'd like to

jump in early

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 1 line

we definitely can help with VT side.

If you can post a patch with code that:

1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtsc instruction in any hvm domain causes a trap
3) the trap handler just does a physical rdtsc and returns

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line

then I could probably take it from there.

OK, and Li in CC will take a look and bake a patch for you.

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

------------------------------------------------------------------------

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#15 lore raw

I would go for (uint32_t)tsc and (uint32_t)(tsc >> 32) as being even clearer.

-- Keir

Quoted 187 lines · Dave Winchell

Hi Li,

Shouldn't this

+ regs->eax = tsc & 0xffff;
+ regs->edx = tsc >> 32 & 0xffff;

be

+ regs->eax = tsc & 0xffffffff;
+ regs->edx = tsc >> 32 & 0xffffffff;

Thanks,
Dave

Quoted 163 lines · Li

Hi, Dan
Please ignore previous comments. The above attachment is OK. Sorry for that. :(

Quoted 157 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Zhang, Li
Sent: Tuesday, July 08, 2008 2:58 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)

In fact, the issue is from guest. In the guest's terminal, it shows that "your
time source seems to be instable..."

Quoted 8 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Zhang, Li
Sent: Tuesday, July 08, 2008 2:49 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)

Quoted 15 lines

Hi, Dan

This is the patch which I think it has been completed VT. But there is an instable issue. The reason may be from that this interception is not bound to timer mode yet.

Thanks
--Li

Quoted 6 lines

-----Original Message-----
From: Dan Magenheimer [mailto:[email protected]]
Sent: Tuesday, July 08, 2008 11:34 AM
To: Zhang, Li; Xen-Devel (E-mail)
Cc: Tian, Kevin; [email protected]
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback

please)

Quoted 17 lines
Quoted 6 lines

Thanks Kevin and Li --

A couple of questions:

1) If the EXITING flag to be set in vmcs.c is to be controlled by a xen

boot

Quoted 10 lines

option, would this work:

static int opt_softtsc = 0;

 boolean_param("softtsc", opt_softtsc);
   :

min = /* original code */
if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;

2) In vmx_rdtsc_intercept(/* need regs as param*/) if we change the rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and then set regs->edx

amd

Quoted 10 lines
Quoted 3 lines

regs->eax from tmptsc, are we almost done?

3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets

called

Quoted 6 lines

to force the trap?

4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is there already code somewhere to emulate WRMSR of tsc?

Sorry I am not very familar with the details of the ia32 instruction

set.

Quoted 13 lines
Quoted 13 lines

Thanks,
Dan

Quoted 9 lines

-----Original Message-----
From: Zhang, Li [mailto:[email protected]]
Sent: Monday, July 07, 2008 9:00 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Hi, Dan
This seems not to be enough. I will cook another patch.

Quoted 4 lines

-----Original Message-----
From: Zhang, Li
Sent: Tuesday, July 08, 2008 10:28 AM
To: Tian, Kevin; '[email protected]'; 'Xen-Devel

(E-mail)'

Quoted 49 lines
Quoted 49 lines
Quoted 49 lines
Quoted 1 line

Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD

feedback please)

Quoted 4 lines

Hi, Dan

I just add some code to produce vmexit. The function

vmx_rdtsc_intercept()

Quoted 1 line

is not completed. It is needed to add some code to get the

guest tsc.

Quoted 10 lines

Thanks
--Li

Quoted 6 lines

-----Original Message-----
From: Tian, Kevin
Sent: Tuesday, July 08, 2008 9:39 AM
To: '[email protected]'; Xen-Devel (E-mail)
Cc: Zhang, Li
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and

AMD feedback please)

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

From: Dan Magenheimer [mailto:[email protected]]
Sent: 2008年7月5日 1:32

Quoted 1 line
Quoted 1 line

Is this something that you (or Intel in general)

could look at?

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I would be happy to participate but I don't think I

understand

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

VT well enough. Once the trap occurs, I suppose

Xen system time

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

could be used as the virtual TSC, possibly scaled up.

There should be tiny related to VT, as only turning

on some bit to

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

allow RDTSC trapping and then the rest stuff should be common how to handle it. We'll take a look, but can't

commit the time due

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

to other scheduled bandwidth. But if you'd like to

jump in early

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 1 line

we definitely can help with VT side.

If you can post a patch with code that:

1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtsc instruction in any hvm domain causes a trap
3) the trap handler just does a physical rdtsc and returns

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line

then I could probably take it from there.

OK, and Li in CC will take a look and bake a patch for you.

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

------------------------------------------------------------------------

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#16 lore raw
In reply to Zhang, Li

Looks great! I am continuing to code and test. Some preliminary results:

1) Substituting hvm_get_guest_time() for hvm_get_guest_tsc() seems to just work! The hvm guest believes it is running on a 1000.185 MHz clock.

2) On my test box, a guest rdtsc without softtsc takes about 30ns. With softtsc, rdtsc takes about 1us.

3) On my user-land test program, I am seeing rare "Time goes backwards", even with hvm_get_guest_time(). I'm trying to track this down.

Dan

Quoted 167 lines

-----Original Message-----
From: Zhang, Li [mailto:[email protected]]
Sent: Tuesday, July 08, 2008 3:47 AM
To: Zhang, Li; [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)

Hi, Dan
Please ignore previous comments. The above attachment is OK. Sorry for that. :(

Quoted 3 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf

Of Zhang, Li

Quoted 4 lines

Sent: Tuesday, July 08, 2008 2:58 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD

feedback please)

Quoted 2 lines

In fact, the issue is from guest. In the guest's terminal,

it shows that "your

Quoted 5 lines

time source seems to be instable..."

Quoted 3 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On

Behalf Of Zhang, Li

Quoted 4 lines
Quoted 4 lines

Sent: Tuesday, July 08, 2008 2:49 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and

AMD feedback please)

Quoted 4 lines
Quoted 4 lines

Hi, Dan

This is the patch which I think it has been completed

VT. But there is an

Quoted 1 line
Quoted 1 line

instable issue. The reason may be from that this

interception is not bound

Quoted 11 lines
Quoted 11 lines

to timer mode yet.

Thanks
--Li

Quoted 6 lines

-----Original Message-----
From: Dan Magenheimer [mailto:[email protected]]
Sent: Tuesday, July 08, 2008 11:34 AM
To: Zhang, Li; Xen-Devel (E-mail)
Cc: Tian, Kevin; [email protected]
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel

and AMD feedback

Quoted 7 lines

please)

Quoted 6 lines
Quoted 6 lines

Thanks Kevin and Li --

A couple of questions:

1) If the EXITING flag to be set in vmcs.c is to be

controlled by a xen

Quoted 10 lines
Quoted 10 lines

boot

Quoted 9 lines

option, would this work:

static int opt_softtsc = 0;

 boolean_param("softtsc", opt_softtsc);
   :

min = /* original code */
if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;

2) In vmx_rdtsc_intercept(/* need regs as param*/)

if we change the

Quoted 1 line
Quoted 1 line
Quoted 1 line

rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and

then set regs->edx

Quoted 4 lines

amd

Quoted 3 lines
Quoted 3 lines

regs->eax from tmptsc, are we almost done?

3) Des Linux or Windows use the CR4.TSD flag and, if

so, what code gets

Quoted 4 lines
Quoted 4 lines

called

Quoted 3 lines

to force the trap?

4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is

there already code

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

somewhere to emulate WRMSR of tsc?

Sorry I am not very familar with the details of the

ia32 instruction

Quoted 21 lines

set.

Quoted 20 lines
Quoted 20 lines

Thanks,
Dan

Quoted 16 lines

-----Original Message-----
From: Zhang, Li [mailto:[email protected]]
Sent: Monday, July 07, 2008 9:00 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)

Hi, Dan
This seems not to be enough. I will cook another patch.

Quoted 4 lines

-----Original Message-----
From: Zhang, Li
Sent: Tuesday, July 08, 2008 10:28 AM
To: Tian, Kevin; '[email protected]';

'Xen-Devel

Quoted 2 lines

(E-mail)'

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Subject: RE: [Xen-devel] Guest TSC and Xen

(Intel and AMD

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

feedback please)

Quoted 4 lines

Hi, Dan

I just add some code to produce vmexit. The function

vmx_rdtsc_intercept()

Quoted 1 line

is not completed. It is needed to add some code

to get the

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines

guest tsc.

Quoted 10 lines

Thanks
--Li

Quoted 6 lines

-----Original Message-----
From: Tian, Kevin
Sent: Tuesday, July 08, 2008 9:39 AM
To: '[email protected]'; Xen-Devel (E-mail)
Cc: Zhang, Li
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and

AMD feedback please)

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

From: Dan Magenheimer

[mailto:[email protected]]

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Sent: 2008年7月5日 1:32

Quoted 1 line
Quoted 1 line

Is this something that you (or Intel in general)

could look at?

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I would be happy to participate but I

don't think I

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

understand

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

VT well enough. Once the trap occurs, I suppose

Xen system time

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

could be used as the virtual TSC,

possibly scaled up.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

There should be tiny related to VT, as

only turning

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

on some bit to

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

allow RDTSC trapping and then the rest

stuff should be common

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

how to handle it. We'll take a look, but can't

commit the time due

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

to other scheduled bandwidth. But if you'd like to

jump in early

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line

we definitely can help with VT side.

If you can post a patch with code that:

1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtsc

instruction in any hvm

Quoted 19 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

domain causes a trap
3) the trap handler just does a physical

rdtsc and returns

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

then I could probably take it from there.

OK, and Li in CC will take a look and bake a

patch for you.

Quoted 9 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#17 lore raw

PATCH 1/1 softtsc (was RE: Guest TSC and Xen (Intel and AMD feedback please))

The attached patch adds a new Xen boot option "softtsc" that causes all hvm domains to trap and emulate all rdtsc instructions and fetch Xen system time instead of the hardware tsc. As a result, the processor will appear to be running at 1000.xyz MHz (where xyz indicates some fraction that seems to vary slightly across different boots). I've also added a line of output to "xm debug-key t" to report softsc and tsc-related CPU features.

(Keir, though 3.3 functionality freeze is past, this is more of a diagnostic aid and defaults off, so I don't know if you would consider it for 3.3.)

Dan

P.S. Thanks to Li Zhang for coding the foundation for this patch. Li, do you want to provide a signed-off-by line too?

Quoted 182 lines

-----Original Message-----
From: Dan Magenheimer [mailto:[email protected]]
Sent: Tuesday, July 08, 2008 6:30 PM
To: 'Zhang, Li'; 'Xen-Devel (E-mail)'
Cc: 'Tian, Kevin'
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)

Looks great! I am continuing to code and test. Some preliminary results:

1) Substituting hvm_get_guest_time() for hvm_get_guest_tsc() seems to just work! The hvm
guest believes it is running on a 1000.185 MHz clock.

2) On my test box, a guest rdtsc without softtsc takes about 30ns. With softtsc,
rdtsc takes about 1us.

3) On my user-land test program, I am seeing rare "Time goes backwards", even with
hvm_get_guest_time(). I'm trying to track this down.

Dan

Quoted 115 lines

-----Original Message-----
From: Zhang, Li [mailto:[email protected]]
Sent: Tuesday, July 08, 2008 3:47 AM
To: Zhang, Li; [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)

Hi, Dan
Please ignore previous comments. The above attachment is OK. Sorry for that. :(

Quoted 3 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf

Of Zhang, Li

Quoted 4 lines

Sent: Tuesday, July 08, 2008 2:58 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD

feedback please)

Quoted 2 lines

In fact, the issue is from guest. In the guest's terminal,

it shows that "your

Quoted 5 lines

time source seems to be instable..."

Quoted 3 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On

Behalf Of Zhang, Li

Quoted 4 lines
Quoted 4 lines

Sent: Tuesday, July 08, 2008 2:49 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and

AMD feedback please)

Quoted 4 lines
Quoted 4 lines

Hi, Dan

This is the patch which I think it has been completed

VT. But there is an

Quoted 1 line
Quoted 1 line

instable issue. The reason may be from that this

interception is not bound

Quoted 11 lines
Quoted 11 lines

to timer mode yet.

Thanks
--Li

Quoted 6 lines

-----Original Message-----
From: Dan Magenheimer [mailto:[email protected]]
Sent: Tuesday, July 08, 2008 11:34 AM
To: Zhang, Li; Xen-Devel (E-mail)
Cc: Tian, Kevin; [email protected]
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel

and AMD feedback

Quoted 7 lines

please)

Quoted 6 lines
Quoted 6 lines

Thanks Kevin and Li --

A couple of questions:

1) If the EXITING flag to be set in vmcs.c is to be

controlled by a xen

Quoted 10 lines
Quoted 10 lines

boot

Quoted 9 lines

option, would this work:

static int opt_softtsc = 0;

 boolean_param("softtsc", opt_softtsc);
   :

min = /* original code */
if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;

2) In vmx_rdtsc_intercept(/* need regs as param*/)

if we change the

Quoted 1 line
Quoted 1 line
Quoted 1 line

rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and

then set regs->edx

Quoted 4 lines

amd

Quoted 3 lines
Quoted 3 lines

regs->eax from tmptsc, are we almost done?

3) Des Linux or Windows use the CR4.TSD flag and, if

so, what code gets

Quoted 4 lines
Quoted 4 lines

called

Quoted 3 lines

to force the trap?

4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is

there already code

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

somewhere to emulate WRMSR of tsc?

Sorry I am not very familar with the details of the

ia32 instruction

Quoted 21 lines

set.

Quoted 20 lines
Quoted 20 lines

Thanks,
Dan

Quoted 16 lines

-----Original Message-----
From: Zhang, Li [mailto:[email protected]]
Sent: Monday, July 07, 2008 9:00 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)

Hi, Dan
This seems not to be enough. I will cook another patch.

Quoted 4 lines

-----Original Message-----
From: Zhang, Li
Sent: Tuesday, July 08, 2008 10:28 AM
To: Tian, Kevin; '[email protected]';

'Xen-Devel

Quoted 2 lines

(E-mail)'

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Subject: RE: [Xen-devel] Guest TSC and Xen

(Intel and AMD

Quoted 17 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

feedback please)

Quoted 4 lines

Hi, Dan

I just add some code to produce vmexit. The function

vmx_rdtsc_intercept()

Quoted 1 line

is not completed. It is needed to add some code

to get the

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

guest tsc.

Quoted 8 lines

Thanks
--Li

Quoted 4 lines

-----Original Message-----
From: Tian, Kevin
Sent: Tuesday, July 08, 2008 9:39 AM
To: '[email protected]';

Xen-Devel (E-mail)

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Cc: Zhang, Li
Subject: RE: [Xen-devel] Guest TSC and Xen

(Intel and

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

AMD feedback please)

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

From: Dan Magenheimer

[mailto:[email protected]]

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Sent: 2008年7月5日 1:32

Quoted 1 line
Quoted 1 line

Is this something that you (or Intel

in general)

Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

could look at?

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I would be happy to participate but I

don't think I

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

understand

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

VT well enough. Once the trap occurs,

I suppose

Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Xen system time

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

could be used as the virtual TSC,

possibly scaled up.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

There should be tiny related to VT, as

Quoted 35 lines
Quoted 7 lines

only turning

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

on some bit to

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

allow RDTSC trapping and then the rest

stuff should be common

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

how to handle it. We'll take a look, but can't

commit the time due

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

to other scheduled bandwidth. But if

you'd like to

Quoted 27 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

jump in early

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line

we definitely can help with VT side.

If you can post a patch with code that:

1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtsc

instruction in any hvm

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

domain causes a trap
3) the trap handler just does a physical

rdtsc and returns

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

then I could probably take it from there.

OK, and Li in CC will take a look and bake a

patch for you.

Quoted 9 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

xen/arch/x86/cpu/amd.c +3 -1
diff -r bd97e45e073a xen/arch/x86/cpu/amd.c
@@ -327,8 +327,10 @@ static void __init init_amd(struct cpuin
 
 	if (cpuid_eax(0x80000000) >= 0x80000007) {
 		c->x86_power = cpuid_edx(0x80000007);
-		if (c->x86_power & (1<<8))
+		if (c->x86_power & (1<<8)) {
 			set_bit(X86_FEATURE_CONSTANT_TSC, c->x86_capability);
+			constant_tsc = 1;
+		}
 	}
 
 #ifdef CONFIG_X86_HT
xen/arch/x86/cpu/common.c +2 -0
diff -r bd97e45e073a xen/arch/x86/cpu/common.c
@@ -16,6 +16,8 @@
 
 #define tsc_disable 0
 #define disable_pse 0
+
+int constant_tsc = 0;
 
 static int cachesize_override __cpuinitdata = -1;
 static int disable_x86_fxsr __cpuinitdata;
xen/arch/x86/cpu/intel.c +3 -1
diff -r bd97e45e073a xen/arch/x86/cpu/intel.c
@@ -184,8 +184,10 @@ static void __devinit init_intel(struct 
 	if (c->x86 == 6) 
 		set_bit(X86_FEATURE_P3, c->x86_capability);
 	if ((c->x86 == 0xf && c->x86_model >= 0x03) ||
-		(c->x86 == 0x6 && c->x86_model >= 0x0e))
+		(c->x86 == 0x6 && c->x86_model >= 0x0e)) {
 		set_bit(X86_FEATURE_CONSTANT_TSC, c->x86_capability);
+		constant_tsc = 1;
+	}
 
 	start_vmx();
 }
xen/arch/x86/hvm/hvm.c +16 -1
diff -r bd97e45e073a xen/arch/x86/hvm/hvm.c
@@ -56,6 +56,8 @@ int hvm_enabled __read_mostly;
 
 unsigned int opt_hvm_debug_level __read_mostly;
 integer_param("hvm_debug", opt_hvm_debug_level);
+int opt_softtsc = 0;
+boolean_param("softtsc", opt_softtsc);
 
 struct hvm_function_table hvm_funcs __read_mostly;
 
@@ -148,7 +150,10 @@ u64 hvm_get_guest_tsc(struct vcpu *v)
 {
     u64 host_tsc;
 
-    rdtscll(host_tsc);
+    if ( opt_softtsc )
+        host_tsc = hvm_get_guest_time(v);
+    else
+        rdtscll(host_tsc);
     return host_tsc + v->arch.hvm_vcpu.cache_tsc_offset;
 }
 
@@ -1640,6 +1645,16 @@ void hvm_cpuid(unsigned int input, unsig
     }
 }
 
+void hvm_rdtsc_intercept(struct cpu_user_regs *regs)
+{
+    uint64_t tsc;
+    struct vcpu *v = current; 
+
+    tsc = hvm_get_guest_tsc(v);
+    regs->eax = (uint32_t)tsc;
+    regs->edx = (uint32_t)(tsc >> 32);
+}
+
 int hvm_msr_read_intercept(struct cpu_user_regs *regs)
 {
     uint32_t ecx = regs->ecx;
xen/arch/x86/hvm/svm/svm.c +9 -0
diff -r bd97e45e073a xen/arch/x86/hvm/svm/svm.c
@@ -52,6 +52,8 @@
 #include <asm/hap.h>
 
 u32 svm_feature_flags;
+
+extern int opt_softtsc;
 
 #define set_segment_register(name, value)  \
     asm volatile ( "movw %%ax ,%%" STR(name) "" : : "a" (value) )
@@ -1340,6 +1342,13 @@ asmlinkage void svm_vmexit_handler(struc
         hvm_triple_fault();
         break;
 
+    case VMEXIT_RDTSC:
+        if ( opt_softtsc )
+        {
+            hvm_rdtsc_intercept(regs);
+            break;
+        }
+        /* else fall through */
     case VMEXIT_RDTSCP:
     case VMEXIT_MONITOR:
     case VMEXIT_MWAIT:
xen/arch/x86/hvm/svm/vmcb.c +5 -0
diff -r bd97e45e073a xen/arch/x86/hvm/svm/vmcb.c
@@ -39,6 +39,8 @@
 #include <xen/keyhandler.h>
 
 extern int svm_dbg_on;
+
+extern int opt_softtsc;
 
 #define IOPM_SIZE   (12 * 1024)
 #define MSRPM_SIZE  (8  * 1024)
@@ -165,6 +167,9 @@ static int construct_vmcb(struct vcpu *v
 
     /* TSC. */
     vmcb->tsc_offset = 0;
+    if ( opt_softtsc )
+        vmcb->general1_intercepts |= GENERAL1_INTERCEPT_RDTSC;
+
     
     /* Guest EFER: *must* contain SVME or VMRUN will fail. */
     vmcb->efer = EFER_SVME;
xen/arch/x86/hvm/vmx/vmcs.c +3 -0
diff -r bd97e45e073a xen/arch/x86/hvm/vmx/vmcs.c
@@ -40,6 +40,7 @@
 
 static int opt_vpid_enabled = 1;
 boolean_param("vpid", opt_vpid_enabled);
+extern int opt_softtsc;
 
 /* Dynamic (run-time adjusted) execution control flags. */
 u32 vmx_pin_based_exec_control __read_mostly;
@@ -96,6 +97,8 @@ static void vmx_init_vmcs_config(void)
            CPU_BASED_MOV_DR_EXITING |
            CPU_BASED_ACTIVATE_IO_BITMAP |
            CPU_BASED_USE_TSC_OFFSETING);
+    if (opt_softtsc)
+        min |= CPU_BASED_RDTSC_EXITING;
     opt = (CPU_BASED_ACTIVATE_MSR_BITMAP |
            CPU_BASED_TPR_SHADOW |
            CPU_BASED_ACTIVATE_SECONDARY_CONTROLS);
xen/arch/x86/hvm/vmx/vmx.c +5 -0
diff -r bd97e45e073a xen/arch/x86/hvm/vmx/vmx.c
@@ -2202,6 +2202,11 @@ asmlinkage void vmx_vmexit_handler(struc
         vmx_invlpg_intercept(exit_qualification);
         break;
     }
+    case EXIT_REASON_RDTSC:
+        inst_len = __get_instruction_length();
+        __update_guest_eip(inst_len);
+        hvm_rdtsc_intercept(regs);
+        break;
     case EXIT_REASON_VMCALL:
     {
         int rc;
xen/arch/x86/time.c +1 -1
diff -r bd97e45e073a xen/arch/x86/time.c
@@ -68,7 +68,7 @@ static DEFINE_PER_CPU(struct cpu_time, c
 static DEFINE_PER_CPU(struct cpu_time, cpu_time);
 
 /* TSC is invariant on C state entry? */
-static bool_t tsc_invariant;
+bool_t tsc_invariant;
 
 /*
  * We simulate a 32-bit platform timer from the 16-bit PIT ch2 counter.
xen/common/keyhandler.c +4 -0
diff -r bd97e45e073a xen/common/keyhandler.c
@@ -251,6 +251,8 @@ static void read_clocks(unsigned char ke
     unsigned int cpu = smp_processor_id(), min_cpu, max_cpu;
     u64 min, max, dif, difus;
     static DEFINE_SPINLOCK(lock);
+    extern int opt_softtsc, constant_tsc;
+    extern bool_t tsc_invariant;
 
     spin_lock(&lock);
 
@@ -284,6 +286,8 @@ static void read_clocks(unsigned char ke
     printk("Min = %"PRIu64" ; Max = %"PRIu64" ; Diff = %"PRIu64
            " (%"PRIu64" microseconds)\n",
            min, max, dif, difus);
+    printk("softtsc = %d, constant_tsc = %d, tsc_invariant = %d\n",
+         opt_softtsc, constant_tsc, (int)tsc_invariant);
 }
 
 extern void dump_runq(unsigned char key);
xen/include/asm-x86/cpufeature.h +2 -0
diff -r bd97e45e073a xen/include/asm-x86/cpufeature.h
@@ -74,6 +74,8 @@
 #define X86_FEATURE_P3		(3*32+ 6) /* P3 */
 #define X86_FEATURE_P4		(3*32+ 7) /* P4 */
 #define X86_FEATURE_CONSTANT_TSC (3*32+ 8) /* TSC ticks at a constant rate */
+
+extern int constant_tsc;
 
 /* Intel-defined CPU features, CPUID level 0x00000001 (ecx), word 4 */
 #define X86_FEATURE_XMM3	(4*32+ 0) /* Streaming SIMD Extensions-3 */
xen/include/asm-x86/hvm/support.h +2 -0
diff -r bd97e45e073a xen/include/asm-x86/hvm/support.h
@@ -125,6 +125,8 @@ void hvm_hlt(unsigned long rflags);
 void hvm_hlt(unsigned long rflags);
 void hvm_triple_fault(void);
 
+void hvm_rdtsc_intercept(struct cpu_user_regs *regs);
+
 /* These functions all return X86EMUL return codes. */
 int hvm_set_efer(uint64_t value);
 int hvm_set_cr0(unsigned long value);

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#18 lore raw

RE: [PATCH] softtsc (was RE: Guest TSC and Xen (Intel and AMD feedback please))

OK.

Thanks

Quoted 244 lines

-----Original Message-----
From: Dan Magenheimer [mailto:[email protected]]
Sent: Thursday, July 10, 2008 5:33 AM
To: [email protected]; Zhang, Li; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: [PATCH] softtsc (was RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please))

The attached patch adds a new Xen boot option "softtsc" that causes all hvm domains to trap and emulate all rdtsc instructions and fetch Xen system time instead of the hardware tsc. As a result, the processor will appear to be running at 1000.xyz MHz (where xyz indicates some fraction that seems to vary slightly across different boots). I've also added a line of output to "xm debug-key t" to report softsc and tsc-related CPU features.

(Keir, though 3.3 functionality freeze is past, this is more of a diagnostic aid and defaults off, so I don't know if you would consider it for 3.3.)

Dan

P.S. Thanks to Li Zhang for coding the foundation for this patch. Li, do you want to provide a signed-off-by line too?

Quoted 217 lines

-----Original Message-----
From: Dan Magenheimer [mailto:[email protected]]
Sent: Tuesday, July 08, 2008 6:30 PM
To: 'Zhang, Li'; 'Xen-Devel (E-mail)'
Cc: 'Tian, Kevin'
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)

Looks great! I am continuing to code and test. Some preliminary results:

1) Substituting hvm_get_guest_time() for hvm_get_guest_tsc() seems to just work! The hvm
guest believes it is running on a 1000.185 MHz clock.

2) On my test box, a guest rdtsc without softtsc takes about 30ns. With softtsc,
rdtsc takes about 1us.

3) On my user-land test program, I am seeing rare "Time goes backwards", even with
hvm_get_guest_time(). I'm trying to track this down.

Dan

Quoted 115 lines

-----Original Message-----
From: Zhang, Li [mailto:[email protected]]
Sent: Tuesday, July 08, 2008 3:47 AM
To: Zhang, Li; [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)

Hi, Dan
Please ignore previous comments. The above attachment is OK. Sorry for that. :(

Quoted 3 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf

Of Zhang, Li

Quoted 4 lines

Sent: Tuesday, July 08, 2008 2:58 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD

feedback please)

Quoted 2 lines

In fact, the issue is from guest. In the guest's terminal,

it shows that "your

Quoted 5 lines

time source seems to be instable..."

Quoted 3 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On

Behalf Of Zhang, Li

Quoted 4 lines
Quoted 4 lines

Sent: Tuesday, July 08, 2008 2:49 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and

AMD feedback please)

Quoted 4 lines
Quoted 4 lines

Hi, Dan

This is the patch which I think it has been completed

VT. But there is an

Quoted 1 line
Quoted 1 line

instable issue. The reason may be from that this

interception is not bound

Quoted 11 lines
Quoted 11 lines

to timer mode yet.

Thanks
--Li

Quoted 6 lines

-----Original Message-----
From: Dan Magenheimer [mailto:[email protected]]
Sent: Tuesday, July 08, 2008 11:34 AM
To: Zhang, Li; Xen-Devel (E-mail)
Cc: Tian, Kevin; [email protected]
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel

and AMD feedback

Quoted 7 lines

please)

Quoted 6 lines
Quoted 6 lines

Thanks Kevin and Li --

A couple of questions:

1) If the EXITING flag to be set in vmcs.c is to be

controlled by a xen

Quoted 10 lines
Quoted 10 lines

boot

Quoted 9 lines

option, would this work:

static int opt_softtsc = 0;

 boolean_param("softtsc", opt_softtsc);
   :

min = /* original code */
if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;

2) In vmx_rdtsc_intercept(/* need regs as param*/)

if we change the

Quoted 1 line
Quoted 1 line
Quoted 1 line

rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and

then set regs->edx

Quoted 4 lines

amd

Quoted 3 lines
Quoted 3 lines

regs->eax from tmptsc, are we almost done?

3) Des Linux or Windows use the CR4.TSD flag and, if

so, what code gets

Quoted 4 lines
Quoted 4 lines

called

Quoted 3 lines

to force the trap?

4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is

there already code

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

somewhere to emulate WRMSR of tsc?

Sorry I am not very familar with the details of the

ia32 instruction

Quoted 21 lines

set.

Quoted 20 lines
Quoted 20 lines

Thanks,
Dan

Quoted 16 lines

-----Original Message-----
From: Zhang, Li [mailto:[email protected]]
Sent: Monday, July 07, 2008 9:00 PM
To: [email protected]; Xen-Devel (E-mail)
Cc: Tian, Kevin
Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)

Hi, Dan
This seems not to be enough. I will cook another patch.

Quoted 4 lines

-----Original Message-----
From: Zhang, Li
Sent: Tuesday, July 08, 2008 10:28 AM
To: Tian, Kevin; '[email protected]';

'Xen-Devel

Quoted 2 lines

(E-mail)'

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Subject: RE: [Xen-devel] Guest TSC and Xen

(Intel and AMD

Quoted 17 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

feedback please)

Quoted 4 lines

Hi, Dan

I just add some code to produce vmexit. The function

vmx_rdtsc_intercept()

Quoted 1 line

is not completed. It is needed to add some code

to get the

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

guest tsc.

Quoted 8 lines

Thanks
--Li

Quoted 4 lines

-----Original Message-----
From: Tian, Kevin
Sent: Tuesday, July 08, 2008 9:39 AM
To: '[email protected]';

Xen-Devel (E-mail)

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Cc: Zhang, Li
Subject: RE: [Xen-devel] Guest TSC and Xen

(Intel and

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

AMD feedback please)

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

From: Dan Magenheimer

[mailto:[email protected]]

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Sent: 2008年7月5日 1:32

Quoted 1 line
Quoted 1 line

Is this something that you (or Intel

in general)

Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

could look at?

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I would be happy to participate but I

don't think I

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

understand

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

VT well enough. Once the trap occurs,

I suppose

Quoted 13 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Xen system time

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

could be used as the virtual TSC,

possibly scaled up.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

There should be tiny related to VT, as

only turning

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

on some bit to

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

allow RDTSC trapping and then the rest

stuff should be common

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

how to handle it. We'll take a look, but can't

commit the time due

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

to other scheduled bandwidth. But if

you'd like to

Quoted 27 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

jump in early

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line

we definitely can help with VT side.

If you can post a patch with code that:

1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtsc

instruction in any hvm

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

domain causes a trap
3) the trap handler just does a physical

rdtsc and returns

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

then I could probably take it from there.

OK, and Li in CC will take a look and bake a

patch for you.

Quoted 9 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#19 lore raw

Re: [PATCH] softtsc (was RE: Guest TSC and Xen (Intel and AMD feedback please))

In reply to Dan Magenheimer
Quoted 11 lines · Dan Magenheimer

The attached patch adds a new Xen boot option "softtsc" that causes all hvm domains to trap and emulate all rdtsc instructions and fetch Xen system time instead of the hardware tsc. As a result, the processor will appear to be running at 1000.xyz MHz (where xyz indicates some fraction that seems to vary slightly across different boots). I've also added a line of output to "xm debug-key t" to report softsc and tsc-related CPU features.

(Keir, though 3.3 functionality freeze is past, this is more of a diagnostic aid and defaults off, so I don't know if you would consider it for 3.3.)

Yes, I will take it, but have the following comments.
1. Why do you define new boolean 'constant_tsc'? Can you just use
test_bit(X86_FEATURE_CONSTANT_TSC)?
2. Your change in common/keyhandler.c breaks ia64. :-)
3. Your change to arch/x86/time.c looks unnecessary.
4. Should you catch SVM's RDTSCP vmexit as well as RDTSC?

-- Keir

#20 lore raw

RE: [PATCH] softtsc (was RE: Guest TSC and Xen (Intel and AMD feedback please))

Quoted 2 lines

Yes, I will take it, but have the following comments.
2. Your change in common/keyhandler.c breaks ia64. :-)

Oops! ;-) What's the best way to handle this? It would be unfortunate to lose valuable debug data just because its arch-dependent but I don't see any other arch-dependent code in keyhandler.c and I'll bet you don't want to start adding ifdef's nor introduce arch/xxx/keyhandler.c just for this.

Quoted 3 lines

1. Why do you define new boolean 'constant_tsc'? Can you just use
test_bit(X86_FEATURE_CONSTANT_TSC)?
3. Your change to arch/x86/time.c looks unnecessary.

I was thinking that the tests for these features should probably be abstracted (e.g. static inline in a header file or a global function), but wasn't sure about the best way to deal with the datatypes (e.g. struct cpuinfo_x86) so defaulted to global variables.

Both globals are simply for debug output in keyhandler.c so depending on the answer to (2) above, those patch-parts could just go away.

Quoted 1 line

4. Should you catch SVM's RDTSCP vmexit as well as RDTSC?

I thought I remembered seeing code that reported/lied to guests that the rdtscp feature was not present?

Thanks,
Dan

#21 lore raw

Re: [PATCH] softtsc (was RE: Guest TSC and Xen (Intel and AMD feedback please))

Hmm.. okay I think I will throw away the debug-output pieces of the patch. They are logically separate and distinct anyway.

-- Keir

Quoted 31 lines · Dan Magenheimer
Quoted 2 lines

Yes, I will take it, but have the following comments.
2. Your change in common/keyhandler.c breaks ia64. :-)

Oops! ;-) What's the best way to handle this? It would be unfortunate to lose valuable debug data just because its arch-dependent but I don't see any other arch-dependent code in keyhandler.c and I'll bet you don't want to start adding ifdef's nor introduce arch/xxx/keyhandler.c just for this.

Quoted 3 lines

1. Why do you define new boolean 'constant_tsc'? Can you just use
test_bit(X86_FEATURE_CONSTANT_TSC)?
3. Your change to arch/x86/time.c looks unnecessary.

I was thinking that the tests for these features should probably be abstracted (e.g. static inline in a header file or a global function), but wasn't sure about the best way to deal with the datatypes (e.g. struct cpuinfo_x86) so defaulted to global variables.

Both globals are simply for debug output in keyhandler.c so depending on the answer to (2) above, those patch-parts could just go away.

Quoted 1 line

4. Should you catch SVM's RDTSCP vmexit as well as RDTSC?

I thought I remembered seeing code that reported/lied to guests that the rdtscp feature was not present?

Thanks,
Dan