Intel Processor Trace virtulization enabling
cover Intel Processor Trace virtulization enabling
Hi All,
Here is a patch-series which adding Processor Trace enabling in XEN guest. You can get It's software developer manuals from:
https://software.intel.com/sites/default/files/managed/c5/15/architecture-instruction-set-extensions-programming-reference.pdf
In Chapter 5 INTEL PROCESSOR TRACE: VMX IMPROVEMENTS.
Introduction:
Intel Processor Trace (Intel PT) is an extension of Intel Architecture that captures information about software execution using dedicated hardware facilities that cause only minimal performance perturbation to the software being traced. Details on the Intel PT infrastructure and trace capabilities can be found in the Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 3C.
The suite of architecture changes serve to simplify the process of virtualizing Intel PT for use by a guest software. There are two primary elements to this new architecture support for VMX support improvements made for Intel PT.
1. Addition of a new guest IA32_RTIT_CTL value field to the VMCS.
— This serves to speed and simplify the process of disabling trace on VM exit, and restoring it on VM entry.
2. Enabling use of EPT to redirect PT output.
— This enables the VMM to elect to virtualize the PT output buffer using EPT. In this mode, the CPU will treat PT output addresses as Guest Physical Addresses (GPAs) and translate them using EPT. This means that Intel PT output reads (of the ToPA table) and writes (of trace output) can cause EPT violations, and other output events.
Luwei Kang (6):
x86: add a flag to enable Intel processor trace
x86: configure vmcs for Intel processor trace virtualization
x86: add intel proecessor trace support for cpuid
x86: add intel processor trace context
x86: implement intel processor trace context switch
x86: Pass through intel processor trace MSRs
docs/misc/xen-command-line.markdown | 7 ++ tools/libxc/xc_cpuid_x86.c | 12 ++- xen/arch/x86/cpu/Makefile | 1 + xen/arch/x86/cpu/intel_pt.c | 114 ++++++++++++++++++++++++++++ xen/arch/x86/cpuid.c | 22 ++++++ xen/arch/x86/domctl.c | 4 + xen/arch/x86/hvm/vmx/vmcs.c | 36 +++++++-- xen/arch/x86/hvm/vmx/vmx.c | 4 + xen/include/asm-x86/cpufeature.h | 1 + xen/include/asm-x86/cpuid.h | 12 ++- xen/include/asm-x86/hvm/vmx/vmcs.h | 12 +++ xen/include/asm-x86/intel_pt.h | 47 ++++++++++++ xen/include/asm-x86/msr-index.h | 16 ++++ xen/include/public/arch-x86/cpufeatureset.h | 1 + 14 files changed, 281 insertions(+), 8 deletions(-) create mode 100644 xen/arch/x86/cpu/intel_pt.c create mode 100644 xen/include/asm-x86/intel_pt.h
Signature · 7 lines
1.8.3.1 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
PATCH 1/6 x86: add a flag to enable Intel processor trace
This patch add a flag to enable Intel PT (Intel processor trace). Default value is 1 (enabled).
--- docs/misc/xen-command-line.markdown | 7 +++++++ xen/arch/x86/cpu/Makefile | 1 + xen/arch/x86/cpu/intel_pt.c | 27 +++++++++++++++++++++++++++ xen/include/asm-x86/intel_pt.h | 26 ++++++++++++++++++++++++++ 4 files changed, 61 insertions(+) create mode 100644 xen/arch/x86/cpu/intel_pt.c create mode 100644 xen/include/asm-x86/intel_pt.h
docs/misc/xen-command-line.markdown +7 -0
@@ -1009,6 +1009,13 @@ debug hypervisor only). ### idle\_latency\_factor > `= <integer>` +### intel\_pt +> `= <boolean>` + +> Default: `true` + +Flag to enable Intel Processor Trace. + ### ioapic\_ack > `= old | new`
xen/arch/x86/cpu/Makefile +1 -0
@@ -8,3 +8,4 @@ obj-y += intel.o obj-y += intel_cacheinfo.o obj-y += mwait-idle.o obj-y += vpmu.o vpmu_amd.o vpmu_intel.o +obj-y += intel_pt.o
xen/arch/x86/cpu/intel_pt.c new file +27 -0
@@ -0,0 +1,27 @@ +/* +* intel_pt.c: Support Intel Processor Trace Virtualization. +* +* Copyright (c) 2017, Intel Corporation. +* +* This program is free software; you can redistribute it and/or modify it +* under the terms and conditions of the GNU General Public License, +* version 2, as published by the Free Software Foundation. +* +* This program is distributed in the hope it will be useful, but WITHOUT +* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +* more details. +* +* You should have received a copy of the GNU General Public License along with +* this program; If not, see <http://www.gnu.org/licenses/>. +* +* Author: Luwei Kang <[email protected]> +*/ + +#include <xen/types.h> +#include <xen/cache.h> +#include <xen/init.h> + +/* intel_pt: Flag to enable Intel Processor Trace (default on). */ +bool_t __read_mostly opt_intel_pt =1 ; +boolean_param ("intel_pt" , opt_intel_pt);
xen/include/asm-x86/intel_pt.h new file +26 -0
@@ -0,0 +1,26 @@ +/* +* intel_pt.h: Intel Processor Trace virtualization for HVM domain. +* +* Copyright (c) 2017, Intel Corporation. +* +* This program is free software; you can redistribute it and/or modify it +* under the terms and conditions of the GNU General Public License, +* version 2, as published by the Free Software Foundation. +* +* This program is distributed in the hope it will be useful, but WITHOUT +* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +* more details. +* +* You should have received a copy of the GNU General Public License along with +* this program; If not, see <http://www.gnu.org/licenses/>. +* +* Author: Luwei Kang <[email protected]> +*/ + +#ifndef __ASM_X86_HVM_INTEL_PT_H_ +#define __ASM_X86_HVM_INTEL_PT_H_ + +extern bool_t opt_intel_pt; + +#endif /* __ASM_X86_HVM_INTEL_PT_H_ */
Signature · 7 lines
1.8.3.1 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
PATCH 2/6 x86: configure vmcs for Intel processor trace virtualization
In reply to Luwei KangThis patch configure VMCS to make Intel PT output address can be treat as guest physical address and translated by EPT when intel_pt option is true.
There have some constraint condition on VMCS configuration, otherwise will cause VM entry failed.
1. If the “Guest PT uses Guest Physical Addresses” execution control is 1, the “Clear IA32_RTIT_CTL on exit” exit control and the “Load IA32_RTIT_CTL on entry” entry control must also be 1.
2. If the “Guest PT uses Guest Physical Addresses” execution control is 1, the "enable EPT" execution control must also be 1.
--- xen/arch/x86/hvm/vmx/vmcs.c | 36 +++++++++++++++++++++++++++++++----- xen/include/asm-x86/hvm/vmx/vmcs.h | 7 +++++++ 2 files changed, 38 insertions(+), 5 deletions(-)
xen/arch/x86/hvm/vmx/vmcs.c +31 -5
@@ -40,6 +40,7 @@#include <asm/shadow.h> #include <asm/tboot.h> #include <asm/apic.h> +#include <asm/intel_pt.h> static bool_t __read_mostly opt_vpid_enabled =1 ;boolean_param ("vpid" , opt_vpid_enabled);
@@ -242,6 +243,9 @@ static int vmx_init_vmcs_config(void)rdmsrl (MSR_IA32_VMX_MISC , _vmx_misc_cap);if ( _vmx_misc_cap &VMX_MISC_VMWRITE_ALL ) opt |=SECONDARY_EXEC_ENABLE_VMCS_SHADOWING ; +if ( _vmx_misc_cap &VMX_MISC_PT_ENABLE ) + opt |=SECONDARY_EXEC_PT_USE_GPA | +SECONDARY_EXEC_CONCEAL_PT_PIP ;if ( opt_vpid_enabled ) opt |=SECONDARY_EXEC_ENABLE_VPID ;if ( opt_unrestricted_guest_enabled )
@@ -343,7 +347,8 @@ static int vmx_init_vmcs_config(void) min =VM_EXIT_ACK_INTR_ON_EXIT ; opt =VM_EXIT_SAVE_GUEST_PAT |VM_EXIT_LOAD_HOST_PAT | -VM_EXIT_CLEAR_BNDCFGS ; +VM_EXIT_CLEAR_BNDCFGS |VM_EXIT_CONCEAL_PT_PIP | +VM_EXIT_CLEAR_IA32_RTIT_CTL ; min |=VM_EXIT_IA32E_MODE ; _vmx_vmexit_control =adjust_vmx_controls ("VMExit Control" , min, opt,MSR_IA32_VMX_EXIT_CTLS , &mismatch);
@@ -383,13 +388,28 @@ static int vmx_init_vmcs_config(void) _vmx_secondary_exec_control &= ~SECONDARY_EXEC_ENABLE_VIRT_EXCEPTIONS ; min =0 ; - opt =VM_ENTRY_LOAD_GUEST_PAT |VM_ENTRY_LOAD_BNDCFGS ; + opt =VM_ENTRY_LOAD_GUEST_PAT |VM_ENTRY_LOAD_BNDCFGS | +VM_ENTRY_CONCEAL_PT_PIP |VM_ENTRY_LOAD_IA32_RTIT_CTL ; _vmx_vmentry_control =adjust_vmx_controls ("VMEntry Control" , min, opt,MSR_IA32_VMX_ENTRY_CTLS , &mismatch);if ( mismatch )return -EINVAL ; +if ( !(_vmx_secondary_exec_control &SECONDARY_EXEC_ENABLE_EPT ) || + !(_vmx_secondary_exec_control &SECONDARY_EXEC_PT_USE_GPA ) || + !(_vmx_vmexit_control &VM_EXIT_CLEAR_IA32_RTIT_CTL ) || + !(_vmx_vmentry_control &VM_ENTRY_LOAD_IA32_RTIT_CTL ) ) + { + _vmx_secondary_exec_control &= ~(SECONDARY_EXEC_PT_USE_GPA | +SECONDARY_EXEC_CONCEAL_PT_PIP ); + _vmx_vmexit_control &= ~(VM_EXIT_CONCEAL_PT_PIP | +VM_EXIT_CLEAR_IA32_RTIT_CTL ); + _vmx_vmentry_control &= ~(VM_ENTRY_CONCEAL_PT_PIP | +VM_ENTRY_LOAD_IA32_RTIT_CTL ); + opt_intel_pt =0 ; + } +if ( !vmx_pin_based_exec_control ) {/* First time through. */
@@ -1032,10 +1052,16 @@ static int construct_vmcs(struct vcpu *v) v->arch.hvm_vmx.secondary_exec_control &= ~(SECONDARY_EXEC_ENABLE_EPT |SECONDARY_EXEC_UNRESTRICTED_GUEST | -SECONDARY_EXEC_ENABLE_INVPCID ); +SECONDARY_EXEC_ENABLE_INVPCID | +SECONDARY_EXEC_PT_USE_GPA | +SECONDARY_EXEC_CONCEAL_PT_PIP ); vmexit_ctl &= ~(VM_EXIT_SAVE_GUEST_PAT | -VM_EXIT_LOAD_HOST_PAT ); - vmentry_ctl &= ~VM_ENTRY_LOAD_GUEST_PAT ; +VM_EXIT_LOAD_HOST_PAT | +VM_EXIT_CONCEAL_PT_PIP | +VM_EXIT_CLEAR_IA32_RTIT_CTL ); + vmentry_ctl &= ~(VM_ENTRY_LOAD_GUEST_PAT | +VM_ENTRY_CONCEAL_PT_PIP | +VM_ENTRY_LOAD_IA32_RTIT_CTL ); }/* Disable Virtualize x2APIC mode by default. */
xen/include/asm-x86/hvm/vmx/vmcs.h +7 -0
@@ -220,6 +220,8 @@ extern u32 vmx_pin_based_exec_control;#define VM_EXIT_LOAD_HOST_EFER 0x00200000#define VM_EXIT_SAVE_PREEMPT_TIMER 0x00400000#define VM_EXIT_CLEAR_BNDCFGS 0x00800000 +#define VM_EXIT_CONCEAL_PT_PIP 0x01000000 +#define VM_EXIT_CLEAR_IA32_RTIT_CTL 0x02000000extern u32 vmx_vmexit_control;#define VM_ENTRY_IA32E_MODE 0x00000200
@@ -229,6 +231,8 @@ extern u32 vmx_vmexit_control;#define VM_ENTRY_LOAD_GUEST_PAT 0x00004000#define VM_ENTRY_LOAD_GUEST_EFER 0x00008000#define VM_ENTRY_LOAD_BNDCFGS 0x00010000 +#define VM_ENTRY_CONCEAL_PT_PIP 0x00020000 +#define VM_ENTRY_LOAD_IA32_RTIT_CTL 0x00040000extern u32 vmx_vmentry_control;#define SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES 0x00000001
@@ -247,7 +251,9 @@ extern u32 vmx_vmentry_control;#define SECONDARY_EXEC_ENABLE_VMCS_SHADOWING 0x00004000#define SECONDARY_EXEC_ENABLE_PML 0x00020000#define SECONDARY_EXEC_ENABLE_VIRT_EXCEPTIONS 0x00040000 +#define SECONDARY_EXEC_CONCEAL_PT_PIP 0x00080000#define SECONDARY_EXEC_XSAVES 0x00100000 +#define SECONDARY_EXEC_PT_USE_GPA 0x01000000#define SECONDARY_EXEC_TSC_SCALING 0x02000000extern u32 vmx_secondary_exec_control;
@@ -268,6 +274,7 @@ extern u32 vmx_secondary_exec_control;#define VMX_VPID_INVVPID_SINGLE_CONTEXT_RETAINING_GLOBAL 0x80000000000ULLextern u64 vmx_ept_vpid_cap; +#define VMX_MISC_PT_ENABLE 0x00004000#define VMX_MISC_CR3_TARGET 0x01ff0000#define VMX_MISC_VMWRITE_ALL 0x20000000
Signature · 7 lines
1.8.3.1 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
PATCH 3/6 x86: add intel proecessor trace support for cpuid
In reply to Luwei KangThis patch add Intel processor trace support
for cpuid handling.
--- tools/libxc/xc_cpuid_x86.c | 12 ++++++++++-- xen/arch/x86/cpuid.c | 22 ++++++++++++++++++++++ xen/arch/x86/domctl.c | 4 ++++ xen/include/asm-x86/cpufeature.h | 1 + xen/include/asm-x86/cpuid.h | 12 +++++++++++- xen/include/public/arch-x86/cpufeatureset.h | 1 + 6 files changed, 49 insertions(+), 3 deletions(-)
tools/libxc/xc_cpuid_x86.c +10 -2
@@ -38,7 +38,7 @@ enum {#define clear_feature (idx, dst) ((dst) &= ~bitmaskof(idx))#define set_feature (idx, dst) ((dst) |= bitmaskof(idx)) -#define DEF_MAX_BASE 0x0000000du +#define DEF_MAX_BASE 0x00000014u#define DEF_MAX_INTELEXT 0x80000008u#define DEF_MAX_AMDEXT 0x8000001cu
@@ -471,6 +471,7 @@ static void xc_cpuid_hvm_policy(xc_interface *xch,case 0x00000002 :/* Intel cache info (dumped by AMD policy) */ case 0x00000004 :/* Intel cache info (dumped by AMD policy) */ case 0x0000000a :/* Architectural Performance Monitor Features */ +case 0x00000014 :/* Intel Processor Trace Features */ case 0x80000002 :/* Processor name string */ case 0x80000003 :/* ... continued */ case 0x80000004 :/* ... continued */
@@ -757,12 +758,19 @@ int xc_cpuid_apply_policy(xc_interface *xch, domid_t domid,continue ; } +if ( input[0 ] ==0x14 ) + { + input[1 ]++; +if ( input[1 ] ==1 ) +continue ; + } + input[0 ]++;if ( !(input[0 ] &0x80000000u ) && (input[0 ] > base_max ) ) input[0 ] =0x80000000u ; input[1 ] =XEN_CPUID_INPUT_UNUSED ; -if ( (input[0 ] ==4 ) || (input[0 ] ==7 ) ) +if ( (input[0 ] ==4 ) || (input[0 ] ==7 ) || (input[0 ] ==0x14 ) ) input[1 ] =0 ;else if ( input[0 ] ==0xd ) input[1 ] =1 ;/* Xen automatically calculates almost everything. */
xen/arch/x86/cpuid.c +22 -0
@@ -9,6 +9,7 @@#include <asm/paging.h> #include <asm/processor.h> #include <asm/xstate.h> +#include <asm/intel_pt.h> const uint32_t known_features[] =INIT_KNOWN_FEATURES ;const uint32_t special_features[] =INIT_SPECIAL_FEATURES ;
@@ -487,7 +488,19 @@ void recalculate_cpuid_policy(struct domain *d)__clear_bit (X86_FEATURE_VMX , max_fs);__clear_bit (X86_FEATURE_SVM , max_fs); } + +/* +* Hide Intel Processor trace feature when hardware not support +* PT-VMX or intel_pt option is disabled. +*/ +if ( !opt_intel_pt ) + { +__clear_bit (X86_FEATURE_INTEL_PT , max_fs); +zero_leaves (p->intel_pt.raw,0 ,ARRAY_SIZE (p->intel_pt.raw) -1 ); + } } +else +zero_leaves (p->intel_pt.raw,0 ,ARRAY_SIZE (p->intel_pt.raw) -1 );/* * Allow the toolstack to set HTT, X2APIC and CMP_LEGACY. These bits
@@ -634,6 +647,15 @@ void guest_cpuid(const struct vcpu *v, uint32_t leaf, *res = p->feat.raw[subleaf];break ; +case 0x14 : +ASSERT (p->intel_pt.max_subleaf <ARRAY_SIZE (p->intel_pt.raw)); +if ( subleaf >min_t (uint32_t, p->intel_pt.max_subleaf, +ARRAY_SIZE (p->intel_pt.raw) -1 ) ) +return ; + + *res = p->intel_pt.raw[subleaf]; +break ; +case XSTATE_CPUID :if ( !p->basic.xsave || subleaf >=ARRAY_SIZE (p->xstate.raw) )return ;
xen/arch/x86/domctl.c +4 -0
@@ -100,6 +100,10 @@ static int update_domain_cpuid_info(struct domain *d, p->feat.raw[ctl->input[1 ]] = leaf;break ; +case 0x14 : + p->intel_pt.raw[ctl->input[1 ]] = leaf; +break ; +case XSTATE_CPUID : p->xstate.raw[ctl->input[1 ]] = leaf;break ;
xen/include/asm-x86/cpufeature.h +1 -0
@@ -95,6 +95,7 @@#define cpu_has_mpx boot_cpu_has(X86_FEATURE_MPX)#define cpu_has_rdseed boot_cpu_has(X86_FEATURE_RDSEED)#define cpu_has_smap boot_cpu_has(X86_FEATURE_SMAP) +#define cpu_has_intel_pt boot_cpu_has(X86_FEATURE_INTEL_PT)#define cpu_has_sha boot_cpu_has(X86_FEATURE_SHA)/* CPUID level 0x80000007.edx */
xen/include/asm-x86/cpuid.h +11 -1
@@ -61,10 +61,11 @@ extern struct cpuidmasks cpuidmask_defaults;/* Whether or not cpuid faulting is available for the current domain. */ DECLARE_PER_CPU (bool, cpuid_faulting_enabled); -#define CPUID_GUEST_NR_BASIC (0xdu + 1) +#define CPUID_GUEST_NR_BASIC (0x14u + 1)#define CPUID_GUEST_NR_FEAT (0u + 1)#define CPUID_GUEST_NR_CACHE (5u + 1)#define CPUID_GUEST_NR_XSTATE (62u + 1) +#define CPUID_GUEST_NR_INTEL_PT (1u + 1)#define CPUID_GUEST_NR_EXTD_INTEL (0x8u + 1)#define CPUID_GUEST_NR_EXTD_AMD (0x1cu + 1)#define CPUID_GUEST_NR_EXTD MAX(CPUID_GUEST_NR_EXTD_INTEL, \
@@ -169,6 +170,15 @@ struct cpuid_policy } comp[CPUID_GUEST_NR_XSTATE ]; } xstate; +/* Structured feature leaf: 0x00000014[xx] */ +union { +struct cpuid_leaf raw[CPUID_GUEST_NR_INTEL_PT ]; +struct { +/* Subleaf 0. */ +uint32_t max_subleaf; + }; + } intel_pt; +/* Extended leaves: 0x800000xx */ union {struct cpuid_leaf raw[CPUID_GUEST_NR_EXTD ];
xen/include/public/arch-x86/cpufeatureset.h +1 -0
@@ -215,6 +215,7 @@ XEN_CPUFEATURE(SMAP, 5*32+20) /*S Supervisor Mode Access Prevention */ XEN_CPUFEATURE(AVX512IFMA, 5*32+21) /*A AVX-512 Integer Fused Multiply Add */ XEN_CPUFEATURE(CLFLUSHOPT, 5*32+23) /*A CLFLUSHOPT instruction */ XEN_CPUFEATURE(CLWB, 5*32+24) /*A CLWB instruction */ +XEN_CPUFEATURE(INTEL_PT, 5*32+25) /*H Intel Processor Trace */ XEN_CPUFEATURE(AVX512PF, 5*32+26) /*A AVX-512 Prefetch Instructions */ XEN_CPUFEATURE(AVX512ER, 5*32+27) /*A AVX-512 Exponent & Reciprocal Instrs */XEN_CPUFEATURE (AVX512CD ,5 *32 +28 )/*A AVX-512 Conflict Detection Instrs */
Signature · 7 lines
1.8.3.1 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
PATCH 4/6 x86: add intel processor trace context
In reply to Luwei KangThis patch add Intel processor trace context
date structure for guest.
--- xen/include/asm-x86/hvm/vmx/vmcs.h | 3 +++ xen/include/asm-x86/intel_pt.h | 17 +++++++++++++++++ xen/include/asm-x86/msr-index.h | 16 ++++++++++++++++ 3 files changed, 36 insertions(+)
xen/include/asm-x86/hvm/vmx/vmcs.h +3 -0
@@ -20,6 +20,7 @@#include <asm/hvm/io.h> #include <irq_vectors.h> +#include <asm/intel_pt.h> extern void vmcs_dump_vcpu (struct vcpu *v);extern void setup_vmcs_dump (void );
@@ -171,6 +172,8 @@ struct arch_vmx_struct {* pCPU and wakeup the related vCPU. */ struct pi_blocking_vcpu pi_blocking; + +struct pt_desc pt_desc; };int vmx_create_vmcs (struct vcpu *v);
xen/include/asm-x86/intel_pt.h +17 -0
@@ -21,6 +21,23 @@#ifndef __ASM_X86_HVM_INTEL_PT_H_#define __ASM_X86_HVM_INTEL_PT_H_ +#include <asm/msr-index.h> + +struct pt_ctx { +u64 ctl; +u64 status; +u64 output_base; +u64 output_mask_ptrs; +u64 cr3_match; +u64 addr[NUM_MSR_IA32_RTIT_ADDR ]; +}; + +struct pt_desc { +bool intel_pt_enabled; +unsigned int addr_num; +struct pt_ctx guest_pt_ctx; +}; +extern bool_t opt_intel_pt;#endif /* __ASM_X86_HVM_INTEL_PT_H_ */
xen/include/asm-x86/msr-index.h +16 -0
@@ -528,4 +528,20 @@#define MSR_PKGC9_IRTL 0x00000634#define MSR_PKGC10_IRTL 0x00000635 +/* Intel PT MSRs */ +#define MSR_IA32_RTIT_CTL 0x00000570 +#define MSR_IA32_RTIT_STATUS 0x00000571 +#define MSR_IA32_RTIT_CR3_MATCH 0x00000572 +#define MSR_IA32_RTIT_OUTPUT_BASE 0x00000560 +#define MSR_IA32_RTIT_OUTPUT_MASK_PTRS 0x00000561 +#define MSR_IA32_RTIT_ADDR0_A 0x00000580 +#define MSR_IA32_RTIT_ADDR0_B 0x00000581 +#define MSR_IA32_RTIT_ADDR1_A 0x00000582 +#define MSR_IA32_RTIT_ADDR1_B 0x00000583 +#define MSR_IA32_RTIT_ADDR2_A 0x00000584 +#define MSR_IA32_RTIT_ADDR2_B 0x00000585 +#define MSR_IA32_RTIT_ADDR3_A 0x00000586 +#define MSR_IA32_RTIT_ADDR3_B 0x00000587 +#define NUM_MSR_IA32_RTIT_ADDR 8 +#endif /* __ASM_MSR_INDEX_H */
Signature · 7 lines
1.8.3.1 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
PATCH 5/6 x86: implement intel processor trace context switch
In reply to Luwei KangThis patch implement Intel proecessor trace context switch.
--- xen/arch/x86/cpu/intel_pt.c | 79 ++++++++++++++++++++++++++++++++++++++ xen/arch/x86/hvm/vmx/vmx.c | 4 ++ xen/include/asm-x86/hvm/vmx/vmcs.h | 2 + xen/include/asm-x86/intel_pt.h | 4 ++ 4 files changed, 89 insertions(+)
xen/arch/x86/cpu/intel_pt.c +79 -0
@@ -21,7 +21,86 @@#include <xen/types.h> #include <xen/cache.h> #include <xen/init.h> +#include <asm/hvm/vmx/vmx.h> +#include <asm/intel_pt.h> /* intel_pt: Flag to enable Intel Processor Trace (default on). */ bool_t __read_mostly opt_intel_pt =1 ;boolean_param ("intel_pt" , opt_intel_pt); + +static inline void pt_load_msr (struct pt_ctx *ctx,u32 addr_num) +{ +u32 i; +wrmsrl (MSR_IA32_RTIT_STATUS , ctx->status); +wrmsrl (MSR_IA32_RTIT_OUTPUT_BASE , ctx->output_base); +wrmsrl (MSR_IA32_RTIT_OUTPUT_MASK_PTRS , ctx->output_mask_ptrs); +wrmsrl (MSR_IA32_RTIT_CR3_MATCH , ctx->cr3_match); +for ( i =0 ; i < addr_num; i++ ) +wrmsrl (MSR_IA32_RTIT_ADDR0_A + i, ctx->addr[i]); +} + +static inline void pt_save_msr (struct pt_ctx *ctx,u32 addr_num) +{ +u32 i; +rdmsrl (MSR_IA32_RTIT_STATUS , ctx->status); +rdmsrl (MSR_IA32_RTIT_OUTPUT_BASE , ctx->output_base); +rdmsrl (MSR_IA32_RTIT_OUTPUT_MASK_PTRS , ctx->output_mask_ptrs); +rdmsrl (MSR_IA32_RTIT_CR3_MATCH , ctx->cr3_match); +for ( i =0 ; i < addr_num; i++ ) +rdmsrl (MSR_IA32_RTIT_ADDR0_A + i, ctx->addr[i]); +} + +void pt_guest_enter (struct vcpu *v) +{ +struct pt_desc *pt = &v->arch.hvm_vmx.pt_desc; + +if ( pt->intel_pt_enabled ) + { +vmx_vmcs_enter (v); +__vmwrite (GUEST_IA32_RTIT_CTL , pt->guest_pt_ctx.ctl); +vmx_vmcs_exit (v); + +pt_load_msr (&pt->guest_pt_ctx, pt->addr_num); + } +} + +void pt_guest_exit (struct vcpu *v) +{ +struct pt_desc *pt = &v->arch.hvm_vmx.pt_desc; + +if ( pt->intel_pt_enabled ) + { +vmx_vmcs_enter (v); +__vmread (GUEST_IA32_RTIT_CTL , &pt->guest_pt_ctx.ctl); +vmx_vmcs_exit (v); + +pt_save_msr (&pt->guest_pt_ctx, pt->addr_num); + } +} + +void pt_vcpu_init (struct vcpu *v) +{ +struct pt_desc *pt = &v->arch.hvm_vmx.pt_desc; +unsigned int eax, ebx, ecx, edx; + +memset (pt,0 ,sizeof (struct pt_desc )); + pt->intel_pt_enabled = false; + +if ( !cpu_has_intel_pt || !opt_intel_pt || + !(v->arch.hvm_vmx.secondary_exec_control &SECONDARY_EXEC_PT_USE_GPA ) ) +return ; + +/* get the number of address ranges */ +if (cpuid_eax (0x14 ) ==1 ) +cpuid_count (0x14 ,1 , &eax, &ebx, &ecx, &edx); +else +return ; + + pt->addr_num = eax &0x7 ; + pt->guest_pt_ctx.output_mask_ptrs =0x7F ; + pt->intel_pt_enabled = true; + +vmx_vmcs_enter (v); +__vmwrite (GUEST_IA32_RTIT_CTL ,0 ); +vmx_vmcs_exit (v); +}
xen/arch/x86/hvm/vmx/vmx.c +4 -0
@@ -468,6 +468,8 @@ static int vmx_vcpu_initialise(struct vcpu *v)if ( v->vcpu_id ==0 ) v->arch.user_regs.rax =1 ; +pt_vcpu_init (v); +return 0 ; }
@@ -3492,6 +3494,7 @@ void vmx_vmexit_handler(struct cpu_user_regs *regs)__vmread (GUEST_RSP , ®s->rsp);__vmread (GUEST_RFLAGS , ®s->rflags); +pt_guest_exit (v);hvm_invalidate_regs_fields (regs);if (paging_mode_hap (v->domain) )
@@ -4260,6 +4263,7 @@ bool vmx_vmenter_helper(const struct cpu_user_regs *regs) } } +pt_guest_enter (curr); out:if (unlikely (curr->arch.hvm_vmx.lbr_fixup_enabled) )lbr_fixup ();
xen/include/asm-x86/hvm/vmx/vmcs.h +2 -0
@@ -421,6 +421,8 @@ enum vmcs_field {GUEST_PDPTE0 =0x0000280a , #define GUEST_PDPTE(n) (GUEST_PDPTE0 + (n) * 2) /* n = 0...3 */GUEST_BNDCFGS =0x00002812 , +GUEST_IA32_RTIT_CTL =0x00002814 , +GUEST_IA32_RTIT_CTL_HIGH =0x00002815 ,HOST_PAT =0x00002c00 ,HOST_EFER =0x00002c02 , HOST_PERF_GLOBAL_CTRL = 0x00002c04,
xen/include/asm-x86/intel_pt.h +4 -0
@@ -40,4 +40,8 @@ struct pt_desc {extern bool_t opt_intel_pt; +void pt_vcpu_init (struct vcpu *v); +void pt_guest_enter (struct vcpu *v); +void pt_guest_exit (struct vcpu *v); +#endif /* __ASM_X86_HVM_INTEL_PT_H_ */
Signature · 7 lines
1.8.3.1 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
PATCH 6/6 x86: Pass through intel processor trace MSRs
In reply to Luwei KangThis patch pass through Intel processor trace MSRs to guest.
--- xen/arch/x86/cpu/intel_pt.c | 8 ++++++++ 1 file changed, 8 insertions(+)
xen/arch/x86/cpu/intel_pt.c +8 -0
@@ -82,6 +82,7 @@ void pt_vcpu_init(struct vcpu *v) {struct pt_desc *pt = &v->arch.hvm_vmx.pt_desc;unsigned int eax, ebx, ecx, edx; +int i;memset (pt,0 ,sizeof (struct pt_desc )); pt->intel_pt_enabled = false;
@@ -102,5 +103,12 @@ void pt_vcpu_init(struct vcpu *v)vmx_vmcs_enter (v);__vmwrite (GUEST_IA32_RTIT_CTL ,0 ); +vmx_clear_msr_intercept (v,MSR_IA32_RTIT_CTL ,VMX_MSR_RW ); +vmx_clear_msr_intercept (v,MSR_IA32_RTIT_STATUS ,VMX_MSR_RW ); +vmx_clear_msr_intercept (v,MSR_IA32_RTIT_OUTPUT_BASE ,VMX_MSR_RW ); +vmx_clear_msr_intercept (v,MSR_IA32_RTIT_OUTPUT_MASK_PTRS ,VMX_MSR_RW ); +vmx_clear_msr_intercept (v,MSR_IA32_RTIT_CR3_MATCH ,VMX_MSR_RW ); +for ( i =0 ; i < pt->addr_num; i++ ) +vmx_clear_msr_intercept (v,MSR_IA32_RTIT_ADDR0_A + i,VMX_MSR_RW );vmx_vmcs_exit (v); }
Signature · 7 lines
1.8.3.1 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
Quoted 6 lines · Luwei Kang
Hi All,
Here is a patch-series which adding Processor Trace enabling in XEN guest. You can get It's software developer manuals from:
https://software.intel.com/sites/default/files/managed/c5/15/architecture-instruction-set-extensions-programming-reference.pdf
In Chapter 5 INTEL PROCESSOR TRACE: VMX IMPROVEMENTS.
Quoted 8 lines
Introduction:
Intel Processor Trace (Intel PT) is an extension of Intel Architecture that captures information about software execution using dedicated hardware facilities that cause only minimal performance perturbation to the software being traced. Details on the Intel PT infrastructure and trace capabilities can be found in the Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 3C.The suite of architecture changes serve to simplify the process of virtualizing Intel PT for use by a guest software. There are two primary elements to this new architecture support for VMX support improvements made for Intel PT.
1. Addition of a new guestIA32_RTIT_CTLvalue field to the VMCS.
— This serves to speed and simplify the process of disabling trace on VM exit, and restoring it on VM entry.
2. Enabling use of EPT to redirect PT output.
— This enables the VMM to elect to virtualize the PT output buffer using EPT. In this mode, the CPU will treat PT output addresses as Guest Physical Addresses (GPAs) and translate them using EPT. This means that Intel PT output reads (of the ToPA table) and writes (of trace output) can cause EPT violations, and other output events.
Hello,
Having read the new proposed extensions, I've got some architecture questions before diving into the patches themselves.
First of all, is this technology expected to end up in Icelake, or something later?
In Vol 3, the existing VMX support describes a number of scenarios (system wide, VMM-only, VM-only, guest aware), which require the use of MSR load lists to atomically alter the IA32_RTIT_* msrs.
Obviously, system wide mode is incompatible with also allowing the guest to use PT itself, but what about Xen wanting to use PT for itself, and for the guest to use PT as well?
Previously, this appears to be possible using the MSR load lists (albeit with Xen needing to shadow the ToPA records to cause the packet stream to end up in the right place).
However, the new VM consistency checks require that using EPT redirection requires clear/load CTL on exit/entry be set, and having load on entry set requires the host TraceEn to be clear.
Therefore, as far as I can see, allowing a guest to use PT via EPT now prohibits Xen also using PT for its own purposes outside of non-root mode.
Is this intentional and/or expected, or have I misunderstood something in the manuals?
Thanks,
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Quoted 18 lines
Quoted 9 lines
Hi All,
Here is a patch-series which adding Processor Trace enabling in XEN guest. You can get It's software developer manuals from:
https://software.intel.com/sites/default/files/managed/c5/15/architect
ure-instruction-set-extensions-programming-reference.pdf
In Chapter 5 INTEL PROCESSOR TRACE: VMX IMPROVEMENTS.Introduction:
Intel Processor Trace (Intel PT) is an extension of Intel Architecture that captures information about software execution usingdedicated hardware facilities that cause only minimal performance perturbation to the software being traced. Details on the Intel PT infrastructure and trace capabilities can be found in the Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 3C.
Quoted 2 lines
The suite of architecture changes serve to simplify the process of virtualizing Intel PT for use by a guest software. There are two
primary elements to this new architecture support for VMX support improvements made for Intel PT.
Quoted 4 lines
1. Addition of a new guest
IA32_RTIT_CTLvalue field to the VMCS.
— This serves to speed and simplify the process of disabling trace on VM exit, and restoring it on VM entry.
2. Enabling use of EPT to redirect PT output.
— This enables the VMM to elect to virtualize the PT output buffer using EPT. In this mode, the CPU will treat PT output
Quoted 8 lines
addresses as Guest Physical Addresses (GPAs) and translate them using EPT. This means that Intel PT output reads (of the ToPA table) and writes (of trace output) can cause EPT violations, and other output events.
Hello,
Having read the new proposed extensions, I've got some architecture questions before diving into the patches themselves.
First of all, is this technology expected to end up in Icelake, or something later?
Yes, this would be enabled on Icelake.
Quoted 3 lines
In Vol 3, the existing VMX support describes a number of scenarios (system wide, VMM-only, VM-only, guest aware), which require the use of MSR load lists to atomically alter the IA32_RTIT_* msrs.
Currently, I just enabling the guest only mode(VM-only) in my patches.
Quoted 2 lines
Obviously, system wide mode is incompatible with also allowing the guest to use PT itself,
Yes, system mode(collect trace packets of the entire system) can't work with guest only mode at the same time.
Quoted 1 line
but what about Xen wanting to use PT for itself, and for the guest to use PT as well?
I think this can be named by host-guest mode. This may need add new command or interface to enable PT in Xen hypervisor. Trace vmm-only and guest simultaneous and output to their respective buffer.
Quoted 3 lines
Previously, this appears to be possible using the MSR load lists (albeit with Xen needing to shadow the ToPA records to cause the packet stream to end up in the right place).
Yes.
Quoted 3 lines
However, the new VM consistency checks require that using EPT redirection requires clear/load CTL on exit/entry be set, and having load on entry set requires the host TraceEn to be clear.
Yes. New bits added in VMCS can make sure PT be disabled before VM exit and IA32_RTIT_CTL MSR will be written with the value of the associated Guest State field of the VMCS on VM entry.
Thanks for your response.
Luwei Kang
Quoted 9 lines
Therefore, as far as I can see, allowing a guest to use PT via EPT now prohibits Xen also using PT for its own purposes outside of non- root mode.
Is this intentional and/or expected, or have I misunderstood something in the manuals?
Thanks,
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Quoted 20 lines · Luwei
Quoted 20 lines
Quoted 9 lines
Hi All,
Here is a patch-series which adding Processor Trace enabling in XEN guest. You can get It's software developer manuals from:
https://software.intel.com/sites/default/files/managed/c5/15/architect
ure-instruction-set-extensions-programming-reference.pdf
In Chapter 5 INTEL PROCESSOR TRACE: VMX IMPROVEMENTS.Introduction:
Intel Processor Trace (Intel PT) is an extension of Intel Architecture that captures information about software execution usingdedicated hardware facilities that cause only minimal performance perturbation to the software being traced. Details on the Intel PT infrastructure and trace capabilities can be found in the Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 3C.
Quoted 1 line
The suite of architecture changes serve to simplify the process of virtualizing Intel PT for use by a guest software. There are two
primary elements to this new architecture support for VMX support improvements made for Intel PT.
Quoted 4 lines
1. Addition of a new guest
IA32_RTIT_CTLvalue field to the VMCS.
— This serves to speed and simplify the process of disabling trace on VM exit, and restoring it on VM entry.
2. Enabling use of EPT to redirect PT output.
— This enables the VMM to elect to virtualize the PT output buffer using EPT. In this mode, the CPU will treat PT outputaddresses as Guest Physical Addresses (GPAs) and translate them using EPT. This means that Intel PT output reads (of the ToPA table) and writes (of trace output) can cause EPT violations, and other output events.
Quoted 6 lines
Quoted 5 lines
Hello,
Having read the new proposed extensions, I've got some architecture questions before diving into the patches themselves.
First of all, is this technology expected to end up in Icelake, or something later?
Yes, this would be enabled on Icelake.
Thanks.
Quoted 4 lines
Quoted 3 lines
In Vol 3, the existing VMX support describes a number of scenarios (system wide, VMM-only, VM-only, guest aware), which require the use of MSR load lists to atomically alter the IA32_RTIT_* msrs.
Currently, I just enabling the guest only mode(VM-only) in my patches.
That is fine. I'm not asking you to implement these modes; I am just trying to work out how they would interact.
Quoted 7 lines
Quoted 2 lines
Obviously, system wide mode is incompatible with also allowing the guest to use PT itself,
Yes, system mode(collect trace packets of the entire system) can't work with guest only mode at the same time.
Quoted 1 line
but what about Xen wanting to use PT for itself, and for the guest to use PT as well?
I think this can be named by host-guest mode. This may need add new command or interface to enable PT in Xen hypervisor. Trace vmm-only and guest simultaneous and output to their respective buffer.
Quoted 7 lines
Quoted 2 lines
Previously, this appears to be possible using the MSR load lists (albeit with Xen needing to shadow the ToPA records to cause the packet stream to end up in the right place).
Yes.
Quoted 2 lines
However, the new VM consistency checks require that using EPT redirection requires clear/load CTL on exit/entry be set, and having load on entry set requires the host TraceEn to be clear.
Yes. New bits added in VMCS can make sure PT be disabled before VM exit and
IA32_RTIT_CTLMSR will be written with the value of the associated Guest State field of the VMCS on VM entry.
I am not sure I explained myself clearly.
It appears that it is possible to implement host-guest mode using MSR load lists, because all the host configuration gets replaced by guest configuration on vmentry, and host configuration gets restored at vmexit.
However with these PT extension, a new restriction is that a vmentry failure will occur if we try to load the guest RTIT_CTL value while the current RTIT_CTL.TraceEn is set.
As far as I can tell, this prohibits host-guest mode from working, unless we tolerate having Xen clear RTIT_CTL before restoring guest GPR state.
Is this correct, or have I missed something?
Thanks,
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Quoted 36 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Hi All,
Here is a patch-series which adding Processor Trace enabling in XEN guest. You can get It's software developer manuals from:
https://software.intel.com/sites/default/files/managed/c5/15/archite
ct ure-instruction-set-extensions-programming-reference.pdf
In Chapter 5 INTEL PROCESSOR TRACE: VMX IMPROVEMENTS.Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Introduction:
Intel Processor Trace (Intel PT) is an extension of Intel Architecture that captures information about software execution usingdedicated hardware facilities that cause only minimal performance
perturbation to the software being traced. Details on the Intel PT infrastructure and trace capabilities can be found in the Intel 64and IA-32 Architectures Software Developer’s Manual, Volume 3C.
Quoted 12 lines
Quoted 12 lines
Quoted 2 lines
The suite of architecture changes serve to simplify the process of virtualizing Intel PT for use by a guest software. There are two
primary elements to this new architecture support for VMX support improvements made for Intel PT.
Quoted 5 lines
1. Addition of a new guest
IA32_RTIT_CTLvalue field to the VMCS.
— This serves to speed and simplify the process of disabling trace on VM exit, and restoring it on VM entry.
2. Enabling use of EPT to redirect PT output.
— This enables the VMM to elect to virtualize the PT output buffer
using EPT. In this mode, the CPU will treat PT outputaddresses as Guest Physical Addresses (GPAs) and translate them using EPT. This means that Intel PT output reads (of the ToPA
table) and writes (of trace output) can cause EPT violations, and other output events.Quoted 6 lines
Quoted 5 lines
Hello,
Having read the new proposed extensions, I've got some architecture questions before diving into the patches themselves.
First of all, is this technology expected to end up in Icelake, or something later?
Yes, this would be enabled on Icelake.
Thanks.
Quoted 1 line
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
In Vol 3, the existing VMX support describes a number of scenarios
(system wide, VMM-only, VM-only, guest aware), which require the use of MSR load lists to atomically alter the IA32_RTIT_*msrs.
Quoted 1 line
Currently, I just enabling the guest only mode(VM-only) in my patches.
That is fine. I'm not asking you to implement these modes; I am just trying to work out how they would interact.
System-Wide: trace Xen hypervisor and guest and output to host buffer;
VMM-only: trace Xen hypervisor only and output to host buffer.;
VM-only: trace guest only and output to guest buffer;
Quoted 16 lines
Quoted 8 lines
Quoted 4 lines
Obviously, system wide mode is incompatible with also allowing the guest to use PT itself,
Yes, system mode(collect trace packets of the entire system) can't work with guest only mode at the same time.
Quoted 1 line
but what about Xen wanting to use PT for itself, and for the guest to use PT as well?
I think this can be named by host-guest mode. This may need add new command or interface to enable PT in Xen hypervisor.
Trace vmm-only and guest simultaneous and output to their respective buffer.
Quoted 7 lines
Quoted 3 lines
Previously, this appears to be possible using the MSR load lists
(albeit with Xen needing to shadow the ToPA records to cause the packet stream to end up in the right place).Yes.
Quoted 2 lines
However, the new VM consistency checks require that using EPT
redirection requires clear/load CTL on exit/entry be set, and having load on entry set requires the host TraceEn to be clear.
Quoted 7 lines
Quoted 1 line
Yes. New bits added in VMCS can make sure PT be disabled before VM exit and
IA32_RTIT_CTLMSR will be written with the valueof the associated Guest State field of the VMCS on VM entry.
I am not sure I explained myself clearly.
It appears that it is possible to implement host-guest mode using MSR load lists, because all the host configuration gets replaced by guest configuration on vmentry, and host configuration gets restored at vmexit.
Yes, correct.
Quoted 3 lines
However with these PT extension, a new restriction is that a vmentry failure will occur if we try to load the guest
RTIT_CTLvalue while the current RTIT_CTL.TraceEn is set.
Yes, as description in "Intel® architecture instruction set extensions programming reference" 5.2.3.
If the “Load IA32_RTIT_CTL on entry” is 1, IA32_RTIT_CTL.TraceEn must be zero. Otherwise will VM entry fails by loading processor state from the guest-state area of the VMCS.
Quoted 3 lines
As far as I can tell, this prohibits host-guest mode from working, unless we tolerate having Xen clear
RTIT_CTLbefore restoring guest GPR state.
Yes. If implement host-guest mode we should clear RTIT_CTL first and restore the guest state before VM-entry .
1. Clear RTIT_CTL before configuration MSRs. A WRMSR to any of these configuration MSRs that begins and ends with IA32_RTIT_CTL.TraceEn set will #GP fault. Packet generation must be disabled before the configuration MSRs can be changed. (SDM 36.2.7.1)
2. Restore other Intel PT MSRs.
3. Restore guest RTIT_CTL state to VMCS guest IA32_RTIT_CTL.
Thanks,
Luwei Kang
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel