lwp: adding support for AMD lightweight profiling
RFC PATCH 4/5 lwp: adding support for AMD lightweight profiling
Add xsave/xrstor support for LWP
Because LWP is not tracked by TS, which is used in current xsave/xrstor implementation for FPU/AVX. As a result, we have to save and restore LWP in extended area whenever vcpu is re-scheduled. To avoid unnecessary lwp save/restore, this patch checks the LWP_CBADDR to determine whether LWP has been turned on.
exporting patch:
# HG changeset patch
# User Wei Huang <[email protected]>
# Date 1297378527 21600
# Node ID 1a1d593c16f36e9cf0153b2ca90dddae649a52bd # Parent 705de1dacabeb899fae6e8121483b2d88d120e57 Add xsave/xrstor support for LWP
Because LWP is not tracked by TS, which is used in current xsave/xrstor implementation for FPU/AVX. As a result, we have to save and restore LWP in extended area whenever vcpu is re-scheduled. To avoid unnecessary lwp save/restore, this patch checks the LWP_CBADDR to determine whether LWP has been turned on.
xen/arch/x86/domain.c +2 -0
@@ -1385,6 +1385,7 @@ stack_regs, CTXT_SWITCH_STACK_BYTES);save_init_fpu (p); +xsave_lwp (p); p->arch.ctxt_switch_from (p); }
@@ -1404,6 +1405,7 @@ CTXT_SWITCH_STACK_BYTES);if ( cpu_has_xsave && n->arch.xcr0 !=get_xcr0 () )set_xcr0 (n->arch.xcr0); +xrstor_lwp (n); n->arch.ctxt_switch_to (n); }
xen/arch/x86/i387.c +49 -0
55 lines not on this page · raw message
xen/include/asm-x86/i387.h +3 -1
@@ -20,6 +20,8 @@void xsave_init (void );int xsave_alloc_save_area (struct vcpu *v);void xsave_free_save_area (struct vcpu *v); +void xsave_lwp (struct vcpu *v); +void xrstor_lwp (struct vcpu *v);#define XSAVE_AREA_MIN_SIZE (512 + 64)/* FP/SSE + XSAVE.HEADER */ #define XSTATE_FP (1ULL << 0)
@@ -33,7 +35,7 @@#define XSAVEOPT (1 << 0)/* The features that the OS saves/restores by default. */ -#define XCNTXT_DEFAULT (-1) +#define XCNTXT_DEFAULT (~XSTATE_LWP)struct xsave_struct {
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Quoted 8 lines
Quoted 1 line
Quoted 1 line · Wei Huang
xen/arch/x86/i387.c +1
-0@@ -65,6 +65,55 @@static void init_fpu (void );static void restore_fpu (struct vcpu *v); +/* Save AMD LWP */
xen/arch/x86/i387.c line 78
Quoted 10 lines
xen/arch/x86/i387.c +10
-0+void xsave_lwp (struct vcpu *v) +{ +uint64_t lwpcb; +bool_t ts; +struct xsave_struct *xsave_area = v->arch.xsave_area; + +if ( cpu_has_lwp ) + { +/* Has LWP been used? */ +rdmsrl (MSR_AMD_LWP_CBADDR , lwpcb);
There's no way to track LWP-using state for a vCPU, is there? rdmsr seems pretty expensive for being used in the context switch unconditionally (on CPUs supporting LWP)...
xen/arch/x86/i387.c line 90
Quoted 12 lines
xen/arch/x86/i387.c +12
-0+if ( !lwpcb ) { +/* Guest might have turned off LWP. So clean the bit here. */ + xsave_area->xsave_hdr.xstate_bv &= ~XSTATE_LWP ; +return ; + } + +/* Disable TS temporarily to avoid recursion. */ + ts =read_cr0 () &X86_CR0_TS ; +clts (); +xsave (v,XSTATE_LWP ); +if ( ts ) +stts ();
Together with the xrstor_lwp() ones, quite a few manipulations of CR0, and hence making context switch between two LWP-using vcpus pretty expensive. I'm sure some of this redundancy can be eliminated.
Jan
Hi Jan,
No there is no quick way (like TS bit) to keep track LWP state. Here is an excerpt from lwp spec:
"LWP does not support the "lazy" state save and restore that is possible for floating point and SSE state. It does not interact with the CR0.TS bit. Operating systems that support LWP must always do an XSAVE to preserve the old thread's LWP context and an XRSTOR to set up the new LWP context. The OS can continue to do a lazy switch of the FP and SSE state by ensuring that the corresponding bits in EDX:EAX are clear when it executes the XSAVE and XRSTOR to handle the LWP context."
Thanks,
-Wei
-----Original Message-----
From: Jan Beulich [mailto:[email protected]]
Sent: Monday, February 14, 2011 3:12 AM
To: Huang2, Wei
Cc: Gang Wei; [email protected]; KeirFraser
Subject: Re: [Xen-devel] [PATCH 4/5][RFC] lwp: adding support for AMD lightweight profiling
Quoted 8 lines
Quoted 1 line
Quoted 1 line · Wei Huang
xen/arch/x86/i387.c +1
-0@@ -65,6 +65,55 @@static void init_fpu (void );static void restore_fpu (struct vcpu *v); +/* Save AMD LWP */
xen/arch/x86/i387.c line 78
Quoted 10 lines
xen/arch/x86/i387.c +10
-0+void xsave_lwp (struct vcpu *v) +{ +uint64_t lwpcb; +bool_t ts; +struct xsave_struct *xsave_area = v->arch.xsave_area; + +if ( cpu_has_lwp ) + { +/* Has LWP been used? */ +rdmsrl (MSR_AMD_LWP_CBADDR , lwpcb);
There's no way to track LWP-using state for a vCPU, is there? rdmsr seems pretty expensive for being used in the context switch unconditionally (on CPUs supporting LWP)...
xen/arch/x86/i387.c line 90
Quoted 12 lines
xen/arch/x86/i387.c +12
-0+if ( !lwpcb ) { +/* Guest might have turned off LWP. So clean the bit here. */ + xsave_area->xsave_hdr.xstate_bv &= ~XSTATE_LWP ; +return ; + } + +/* Disable TS temporarily to avoid recursion. */ + ts =read_cr0 () &X86_CR0_TS ; +clts (); +xsave (v,XSTATE_LWP ); +if ( ts ) +stts ();
Together with the xrstor_lwp() ones, quite a few manipulations of CR0, and hence making context switch between two LWP-using vcpus pretty expensive. I'm sure some of this redundancy can be eliminated.
Jan
Quoted 11 lines
Quoted 1 line
Quoted 1 line · Wei
No there is no quick way (like TS bit) to keep track LWP state. Here is an excerpt from lwp spec:
"LWP does not support the "lazy" state save and restore that is possible for floating point and SSE state. It does not interact with the CR0.TS bit. Operating systems that support LWP must always do an XSAVE to preserve the old thread's LWP context and an XRSTOR to set up the new LWP context. The OS can continue to do a lazy switch of the FP and SSE state by ensuring that the corresponding bits in EDX:EAX are clear when it executes the XSAVE and XRSTOR to handle the LWP context."
I certainly wasn't thinking of CR0.TS, but after reading patch 5 my question should have been re-phrased into whether the intercept of the control MSR can't be used for tracking purposes.
Jan
This might be possible. The frequency of MSR read/write inside guest VM will be less compared with vcpu context switch. I will try it in next spin.
-Wei
-----Original Message-----
From: Jan Beulich [mailto:[email protected]]
Sent: Monday, February 14, 2011 10:18 AM
To: Huang2, Wei
Cc: Gang Wei; [email protected]; KeirFraser
Subject: RE: [Xen-devel] [PATCH 4/5][RFC] lwp: adding support for AMD lightweight profiling
Quoted 11 lines
Quoted 1 line
Quoted 1 line · Wei
No there is no quick way (like TS bit) to keep track LWP state. Here is an excerpt from lwp spec:
"LWP does not support the "lazy" state save and restore that is possible for floating point and SSE state. It does not interact with the CR0.TS bit. Operating systems that support LWP must always do an XSAVE to preserve the old thread's LWP context and an XRSTOR to set up the new LWP context. The OS can continue to do a lazy switch of the FP and SSE state by ensuring that the corresponding bits in EDX:EAX are clear when it executes the XSAVE and XRSTOR to handle the LWP context."
I certainly wasn't thinking of CR0.TS, but after reading patch 5 my question should have been re-phrased into whether the intercept of the control MSR can't be used for tracking purposes.
Jan